Home Forums Discussions Problems HELP Catastrophic loss of imported items

Friday afternoon AC froze, was restarted, but then noted that on some patients imported items were gone. Called AC today and they 'repaired' Imported Items but told us about 1000 patient records were lost/corrupted!! We were backing up nightly both locally and remotely but according to Angel Support this does NOT include imported items. We are now missing EKG's, labs, consult notes, PFT's, diagnostic testing, etc for about 1000 active patients. I am beside myself, what do I say to my patients, what do I do when someone needs a repeat EKG and I have NOTHING to compare it with, or we are following trending data (PSA, RFT's, HBA1C's, etc)? How could we be paying for backup, using AC backup module to also backup to a local drive as well and not be informed that we were unprotected for Imported Items??

I do not know if there is any way to get back this data, I am uncertain how to proceed in the future, I believe AC has to inform ALL their clients that they are NOT backing up any IMPORTED items and that somehow this needs to be done by the individual clients!

Yuch.......this is pretty rotten nightmare for my providers and patients and I really feel like AC let me down in a HUGE way!!

James P. Clayton, M.D.
US Virgin Islands

UPDATE: According to our IT guy, we were told to turn 'off' backup of 'associated files' as our database was too large causing remote backup failures. My IT guy apparently did not know the consequences, nor did he ask, when AC Angel support told him to do this :-((((

That is VERY unfortunate. I back up the II with my backup file, and it is about 4 GB. I know Bert and some others may back them up separately. I certainly recommend NOT using the cloud if it cannot back up II successfully. Bert has many ideas about using RAID drives. I use a G-Safe RAID unit with 2 drives, and have a 3rd drive for swapping. I swap in the drive and take another home: it then rebuilds the backup on the new drive, giving me a backup, backup backup, and a third backup backup backup that is off-site. I would not trust the cloud for this!

James,

Are they saying that there *were* another 1000 items, but AC no longer sees those files on the disk and they are missing?

There is a critical issue here of whether there is a disk error or a corruption of the file system.

As a precaution - you should IMMEDIATELY run a manual backup of AC that INCLUDES Imported Items, Images - EVERYTHING.

You are going to want to write that to another disk *NOT* the one where AC currently resides. Turn ON all of the error checking and file verification steps, but don't upload it because it won't go.

You are going to also want to grab a copy of your backup from Thursday (or last one before Friday), because you are going to need it to determine the BEFORE list of imported Items, if it comes to that.

If you have another machine that can serve as your temp AC server, that would be your best bet to do next, because what you have to do next is determine if there is a physical disk problem, or just OS/file corruption. Testing that isn't going to work on a production machine.

There are tools we have used on the SMB/Enterprise side to recover files from corrupted disks - those work best if you take the machine offline until you are ready to recover so that minimal files are over-written, or further corruption takes place.

If the disk is bad/going bad, that has to be dealt with separately from data recovery.

Some questions:
What machine can you make the AC server for the next few days?
Are the server disks RAIDed?
Does your IT help have experience in doing data recovery?
Do you have a spare disk on-hand that matches your existing disk(s)?

I'm not a Doc; combat first aid is as far as I got, but we are trying to minimize the damage, prevent further damage while finding a path for you to continue to operate, then see what we can do about reconstruction.

I have also see where folks are ditzing around with the AC/Imported Items directory and inadvertently drag them somewhere else on the machine - it could be something that simple, just in case it is something that simple.

You've mentioned you have local IT folks, but if you need to reach out to a crusty sailor, feel free.

Indy,

Your suggestions are very good! We can't tell why some files and not others are gone.

We don't have another system or disks to easily move the production DB off to.

We have a Raid environment. I will check to see if the Raid disks failed or got corrupted. In the heat of the battle I didn't think of that, I am only thinking of triage and surgery. (I am a tech guy but I thought I would throw in some medical terms I remember from 'Mash'). Thanks for this!!

I have backups, just not full ones that include the Import Items. We were able to identify and find 1204 files. I am running 'recuva' in low priority mode right now to find some more of the corrupted files.

I have added backing up the ImportItems to the future backup but that doesn't help us now.

Tonight we are going to backup the AC database to another disk. Then we will restore the backup made before the crash. That will give us a report of the still missing files, if any.

Your suggestions have been spot on! Thank you for responding to James. I find your message very helpful.

John

First, some tough love. You will hate me for this first paragraph, but I just can't get away without saying it. This is not AC's fault. You are using their backup, but like any backup, you have to look at the settings and see what it does. The reason you have the choice is because at some point you will have 10GBs of II and it won't go to the cloud. Under the admin section, you can turn the II on and off.

When I give talks on backups, I always talk about general to specific. The AC backup is ONLY important because it is quick. I wouldn't trust my entire practice to one backup. I hope this doesn't come across wrong, but we have backups to 10 drives over a year back. Many different types of software. So always back up your entire server or computer at least five days back on different drives and test them.

Now, to try to help. First, they say they are corrupt. Nowhere did I read that an EKG was deleted. If you go to the patient whose EKG was deleted and then you go to the folder in II of that patient, is the file there? If yes on all, you are fine.

You can try Indy's method. If that works great.
Make sure you look physically for the file in the individual folders in II in AC.
Under Administrative Options and Tools, there is an Imported Items: Repair missing files and links.

From what you have said, it sounds more like links from AC to II.
This could also be a SQL database issue, which could be fixed with someone like Raja.

I apologize if I am out of line. I just feel bad for you. But, the absolute first thing I would do is to look for the file that says it is missing.

Thanks for the tough love Bert.

I do the back up 10 ways from Sunday for everything else. I even backup AC's backup in three places. I honestly didn't know to backup the import items folder.

Moving forward I will be backing up the Import Items folder to the normal backup drive and the external backup drive that gets swapped out each week.

I will put in a script in the reboot for the server that copies the who Amazing Charts folder to a different drive. We reboot once a week.

It is sort of like closing the barn after the cows are out but it will protect the practice moving forward.

I was working with Craig and Carlos during the day today. They will be handing off the job to Anthony this evening. We have most of the files back but we are missing some actual files. I am looking for them in the old Import Items folder and using a utility to find / un-delete them.

John

I would be EXTREMELY interested in knowing what caused this.

JamesNT

Good to hear that John. The topic of where or to change where the II folder has always been an issue. Only because of the offsite backup, which at some point forces one to back them up separately.

A special warning would be nice from AC that as their cloud service is excluding II, another method must be in place. Apparently not everyone knows this.

You just gave it Chris.

No, that's a good idea.

Why couldn't AC offer incremental or deduplication for the IIs?

This is just to keep everyone informed of the status of this restore project.

Every step is very long and time consuming due to the II having over 30,000 files.

I found some of the missing missing files from the installation on an older server we were using.

I am recovering files from the current server disk that were deleted from the II. It looks very promising.

Tomorrow night we will backup the current DB, restore and old one and get a good report of the missing files, then restore the current DB for the morning.

We will then run a relink utility that AC has for the previously missing files that are now found.

The final step is to compare the missing links report to the files found. That will tell us what is still missing.

During the day I am working with Craig. Tomorrow during the day we will try to ascertain what could have caused this.

I will keep everyone posted. Many thanks to your assistance so far!

How are you going to merge the II from the backup disc? (Since some are missing, and others are probably new compared to the backup)

I always warned about this. We developed a backup service that lets you backup the II (incrementally) and database completely. So it even works for people with slow DSL connections. We also do an added bonus of providing a cloud failover for you local server in case something happens to the server in your office.

Boondoc,

I will use xcopy to ensure file edited (or created) last is the one left standing. I was told from Andrew that there is a utility that will build new references for new files it finds in the folder associated with the patient.

Can we go back to the beginning for a moment? What computer is the database on? Is it on a server? If so, which machine? If a server, what OS?

I am totally lost about the set-up here, as well as exactly what happened. I assume that James and John in VI are in pretty deep, trying to sort this out. I would add to the chorus of voices asking that when you are done (if not before), please report back about your set-up and any information about exactly what occurred. It likely will be helpful to others.

I second that. It would be helpful. Not only that, it is disconcerting (one backup or a hundred), that someone could potentially lose some or all of their imported items.

In fact when all is said and done, I think it would be helpful if a support tech (Craig would be good I suppose given his work on it) try to explain what bug or non bug made this happen.

And, while I took a little different stance earlier, it would also be helpful if in the next version that updates the backup, that it state clearly on it that it backs up the AC data by default, but the user must choose to include the imported items or not.

OK, I am so curious as to what the OS that AC was on, I am turning on my notification on my email on smart phone.

Glad to share!

Server 2003 R2 64 bit, member server
Dual XEON quad with hyper
4 gb RAM
4 Raid 5 with 7200 rpm industrial class drives with 1 spare
2 Raid 05 with 7200 rpm industrial class drives for OS
1 one gb hard drive in a USB hard drive dock for backup
AC 6.3.2
Full SQL 2005 std

I was logged in at 2:23 pm
Logs show external backup drive turned on and off several times
I was ensuring the hard drive dock was online and available
I was editing text file scripts for login and logoff
run any yet.
Received a call that AC had crashed.
Logs show errors started in both the application log at 3:45
No error messages until 3:45 pm Friday 8/16

3:45 pm rebooted system - didn't fix AC
3:52 pm that AC automatic backup was started - tried optimize
4:12 pm amazing backup began uploading backup file
AC still didn't work
4:37 pm Serious errors for amazingChartService that system
exceptions begain. I believe these were folks trying
to log in
tested, looked around and found the XML file was missing
4:50 pm or so I ran amazing utilities repair connections
5:09 pm rebooted system
5:10 pm people were able to us AC again - no more errors

no more error messages

Monday found out that people were missing some imported items for some patients started Saturday and Monday

I've had this happen to a practice before. Except that practice was much worse off. They didn't have the II or the DB backed up. Here's how I did it:

If you've already restored the previous backup, that may be okay.

If you have any previous backup of the server from that time. E.g. were you running Windows Server Backup or something similar. Maybe even system restore, but if you have a backup, that's better. Avoid system restore for now. Or if you do run it, clone your hard drives for sure.

You can restore the server to its state before Monday in a virtual machine. Copy the whole AC folder (esp. the II) to a disk. Reinstall AC on another computer and make sure AC works and all the II are present.

From here you can either copy the II back to the main computer or use the AmazingSync. Then have the docs verify that everything is present. I know Dr. Clayton has a relatively large practice so this will require some weekend time to make sure everything is up and running Monday.

Also just a side note, judging from the hardware, it's time to upgrade.

You need to have several backup systems in place for a practice that size.
3 Levels
1. Onsite Local (This should have been your go to before getting the one from AC GA)
2. Offsite Local (Either one of the docs homes or your homes. Maybe a branch office)
3. Offsite Remote (AC Online Backup, CrashPlan, etc)

I also have a backup service that includes II (incrementally) and the DB as mentioned above.

See PM as soon as possible.

The server shut down this morning:

The previous system shutdown at 7:57:24 AM on 8/21/2013 was unexpected.

I am attempting to find out if someone rebooted it or if it happened on its own. I don't see anything in the error logs to show that it shut down due to a recordable error. I also don't see the normal commanded shutdown evidence.

Don't forget to ask if the power went out.

JamesNT

OK, I posted this somewhere else, but anyway. Windows 2003 should have (I know SBS 2003 did) VSS. Which means if Volume Shadow Copies were enabled on that drive you may have several pristine copies of AC.

Shadow copies are not enabled.

Why?

There are two reasons shadow copies are typically not enabled:

1. Lot of people, even so-called IT Pros, don't know about it.

2. Not enough disk spaced was purchased with the server in question.

JamesNT

Good question.

The rest of the servers did not reboot. This server uses two new APC 1500 battery backups and are hooked up via a USB cable to the software. If the power would have been down long enough for a power outage the server should have been on a controlled shutdown.

I sent a question off to find out for sure.

True, but one of the one things Microsoft got right was VSS. The byproduct is allowing you to make images at the times you wish. You can also control how many. They have saved me many times. They are a backup you don't even need to think about.

Here is my recommendation: All in good faith. Been pulling for you. I know the feeling. The all nighters.

Keep trying. But, at some point cut your losses. If you have the money send the disks to professional extractors. $$$$

I agree with Sandeep. Get to 2012. Maybe Hyper-V. Get 32GBs of RAM. etc. But, I know you were likely there. Probably like I am.

HAVE YOU CHECKED THE PM. IT IS CRUCIAL. I can change the wording above.

@James

Thought I was going to be a hero on VSS, lol.

This server is at about 80% capacity on drive C: and 50% on drive D:

Honestly though my server team had a issues with drive shadowing in the past which made me un-trusting. When I started managing servers a while ago I made a server run out of drive space and lock up at night using it. My boss was pissed. Ever since then I haven't thought about it to give it another try.

Yeah. Everyone has their own experience. Easy to enlarge drive D, though. Plus, 2008 and 2011/2012 VSS copies much better. I would give it more thought next time Maybe now you will, lol

PS Everything I say is to help or in jest. Not trying to be a smartass

At the risk of sounding stupid, what is meant by PM?

Oh, sorry, I sent you a private message. You should see a flashing mailbox at the top of your screen.

Support knows we are using full SQL. We had to go to full SQL a while ago when the practice's database got too large. AC did a good job working with us converting to using full SQL.

Yeah I have helped a few. With a SQL expert. But, I guess they can't hold that against anyone. Now with Express you get 10GBs. Still not enough. But, the issue with Express that you don't have is the memory. We (I) and 99% on here get 1GB of memory which sucks.

But, there is no way AC can make people purchase an expensive product that won't install with the program. But, there are many that make you do or purchase the ultimate expensive Oracle.

Whoa. Good, though.