Home Forums Discussions General Discussion Our recovery from a "catastrophic" crash

I am posting this in case it is of interest and may help someone else down the road...

The docs in our office were on staycation this week, but checking in daily. Everything was fine with AC when the staff left at 4:30 Thursday. Friday morning, no one could connect to the AC database and the "server" on our peer-to-peer network looked like this....

Unfortunately the computer kept cycling through restarts, never advancing beyond that screen.

My staff called me and then left an urgent message for AC at 8:30AM.  We got a call back from Clare from Guardian Angels in about 15 minutes.  We quickly decided the best approach was to move the SQL/database version of AC to one of our client computers, and reinstall/restore from there.  My understanding was that we needed AC to "provision" the client which is one reason we needed to get in touch with support urgently. Claire quickly got to work doing the uninstall/reinstall for us.

For backups I use IDrive which is super cheap, and allows both local and online incremental backups of multiple machines. The bad news is that at some point my online IDrive backups of the server stopped working and I didn't pay attention to that failure.  Perhaps this was when we upgraded last month to V11. Fortunately,  we had local backups. In 2011 I purchased this IoSafe drive. It has been chugging along silently for 10 years, storing (and auto-deleting) automated daily back-ups.  I use it for a daily copy of my enc file of AC, and I back-up my Imported Items daily as well (separately) to this local drive (connected directly to the "server"). The backups to the external drive were fine, so we could simply plug it into the new "server" and restore the enc file.  Imported items took longer because a). Our Imported Items are gigantic (over 135gb) and b). they are encrypted by IDrive for the incremental backups. As a result, the "restore" process took about 4 hours.

When all was said and done, we were up and running by 4PM.  Luckily this happened on a vacation day for the docs, so the impact on patient care was minimal.  We did lose Thursday's data... the backups were not scheduled until 9PM and the crash presumably occurred between 430 and 9, so the day's work was not included.


Take home messages for us from the event...

1. I always felt that in case of a crash like this, an advantage of a P2P set-up was the ability to make a relatively easy switch like this without an immediate need for new hardware (and the inherent delay and cost that would entail).  This proved to be true.

2.  AC provided superb support.  It was VERY prompt (despite our concerns that there might be ongoing support "overload" due to the recent mandatory upgrade).  Clare is knowledgeable, super competent, friendly and calm.  She was a tremendous help.

3.  Local and online backups, with redundancy, make a huge difference. Even though I was remiss regarding the failed online II backups, having a local one saved the day.

4. Backup software does not have to be expensive.  IDrive more than paid for a decade worth of annual fees yesterday.

5. We need to look at the timing of our backups.  I tried to do them at night so they don't run while we are still using AC, but perhaps that should not be the main concern.

6. It helps to have crashes occur when you are on vacation, but in town.  Intelligent, reliable staff is important, too!

Well you knew I would be the first to respond. As backups are one of my favorite topics.

First, thank you Jon for sharing this with the rest of us. It is very helpful...very much to me as well. Everyone used to make fun of me because I have this backup program which backs up the entire server (now the VMs) nightly to multiple places. It does send an email daily with the result of the backup. But, I never scheduled the .enc backup, nor our billing software or our 3rd party programs which work with AC. I always do those manually and watch them finish. Obviously, people with imports within their application could have difficulty. SQL Express 2012 also has its own .bak file.

One of the things that kept the old board hopping were the battles that went back and forth between P2P and client/server based setups. The overall benefits and downsides to both notwithstanding, this is related solely to the risk/benefits of backups. I am not quite sure why a P2P setup would be beneficial other than providing another computer to run from. But, that could be done easily enough with a server. But, I can see where if someone were not using a server with another computer or server to failover to (which essentially is what Jon did minus the advantages of an automatic failover).

I have learned a lot from Jon's sharing this and how well AC support was able to help very quickly. And, the steps he took to fix it. I am not writing this to critique his setup. It does sound like there was a lot of thought put into the backup plan. And, I am sure there are many out there with no backup plan. I have often wondered why there are no others sharing experiences like this. There must be many others. I had a serious crash of my server once and fortunately I had backups with Veeam, which did require a phone call to them for help.

Finally, I think there are three groups. Those with no or a poorly thought out backup program. Those like Jon and me who have good backup setups. There is a third group whose backup plan would be more costly but value their overall data over their inability to understand and set up a good backup plan. There are third party IT companies who don't need to take over your overall system but are dedicated to the backup of your system including on premise, locally to their servers. They would monitor backups were done daily and test them weekly. Of course, you would need a BAA, if that is the correct term. Finally, the only good backup is one that is tested to see if it actually works.

But, as stated, this may sound strange, but I am proud of Jon for getting his system back up so quickly and sharing his story with the rest of us.

Just my three cents worth. I don't think I have ever given two cents worth.

At Jon, I hope I am not offending you -- just adding some things that Indy and Sandeep would have added as well as James.