Home Forums It's time to get real f&*(g serious about backups

We use RHV and KVM, but functionally I expect it will be the same from a virtualization perspective.

We clone/snapshot before we do an upgrade, and that gives us an ability to revert seamlessly when an upgrade goes sideways, but it also allows us to run an upgrade on the clone for a practice to experiment with prior to deciding if they want to do the upgrade, or to test a restore.

Virtualization [and containers] is where technology is headed, and I know certain Docs on the board and elsewhere who have been using Linux servers and virtualization for years. Did I mention free and high performing?

This is what I'm talking about (see screenshot below). Count the number of virtual machines in this backup.

And one of them is an Exchange Server for about 20 people and another is a SQL Server database server that aggregates data from three dozen other databases every night into one SQL Server database using SQL Server Integration Services. In other words, data changes.

Yet this backup took 12 MINUTES 25 SECONDS. This is how it's DONE, people. And this is fully automated both onsite and OFFSITE backup (published to the cloud for a monthly fee). This backup runs every two hours.

Not only was the backup completed, but if you'll look at the print out database transaction logs were truncated.

My recommendation is to virtualize all servers using Windows Server 2016 Hyper-V and get Veeam to handle backups.

With Windows Server 2012 Standard through Windows Server 2016 Standard, you get virtual licensing for two guests.

JamesNT

So if we pay for backup with AC and we get a message everyday on AC that "Backup Successful" we are all set or this is not enough?

It sounds like you are using the offsite backup with incremental backups. I don't use that, so I can't say. I would guess that with the cloud backup, they would be sending verification either through AC or email. That would be enough given you can't test them. You would have to assume they are good backups.

I was referring to the backups I do locally with AC. When they finish they leave a successful backup notification in my inbox. That just tells me that a backup was made and is stored. I pretty much trust it. But, I do test them every once in a while. Except for the changes made in v9 and above, I have found the backups to be bulletproof for over 11 years.

Thanks Bert. I did mean that the doc gets the successful notification message in his inbox. We have never done a double check to make sure this is working. I'm assuming the backup is being stored on the server?

If you are referring to the local backup app that you find in the AmazingCharts folder:



then a successful notification will confirm that your backup ran and ended up everywhere you wanted it to go. You can see here that other than the default backup, there are three more paths it can go, which is probably a bit of overkill. There are settings here for scheduling and deciding if you are going to backup your imported items offsite. It's a good idea to not do that as they can get 5, 10, even 20GBs and take forever to back up.

FYI: If you are talking about the new offsite backup program, it has an incremental option so you can backup everything including the Imported Items. But, you still want a local backup, because it can take a day or so to download the offsite backup.

As far as where it defaults to, you are correct. It defaults to the server in the BACKUPS folder in the AmazingCharts folder.



It is good because those not familiar with backups may not know how to find or browse to the backup, so there is always one location that support can tell them where to find it. On the other hand, it is the worst location as far as a backup strategy as your backup is in the root folder of AC.

I believe in version 9.4, you will have the option of turning that one off. Reason being if you forget they are there, you could end up with a very large folder on your server.

Two points:

1. Bert makes an excellent post on AC's backup. However, keep in mind that AC's backup does not cover the rest of the server. If you have other files on that server you need to keep, you need to reconsider your backup strategy.

2. If you've never tried restoring a backup to make certain it works and you understand the process, then you don't have a backup strategy.

JamesNT

I keep my server (on my peer to peer) clean... other than some shared files (documents) and AC, there ain't much there. So imaging the server is not an issue for me.

I totally agree that simply getting a message that you have a back up is not sufficient. I would be interested in hearing more specifics about how people test their back ups. I think that a restore to a sandbox machine is the only reliable way. Do the experts agree?

Either a sandbox or VM. Virtual machines are easier, because they are on the same machine. For the purpose of this discussion, we will stick with sandbox as not everyone is knowledgeable about VMs.

Installing to the same version on a test mashine is the best way. Of course, you have to get around all the v9 roadblocks.

One big advantage to imaging the server is that you don't need AC to restore. Meaning it's doable at anytime of the day. Also, it's way less time consuming than resetting up all the folder shares, IP addresses, etc.

Virtualization is great. You get to have more servers without buying more hardware. It does add a level of complexity however. Not sure if this applies to 8, but Windows 10 can be moved between machines rather easily. Even if the hardware is different.

With older versions of Windows, you would get a bluescreen but it appears Windows 10 is able to detect when it's moved to new hardware and triggers sysprep at bootup automatically.

I have a test server that I lug around to some of my client offices and I restore their backups onto that server. If their backups are to the cloud, I just download the backup myself onto the test server in my office. Other times the client sends me their backup via Fed-Ex. I then have the client inspect the server to make certain the data is as it should be. For example, QuickBooks has all the transactions it should have, last few new patients entered into Amazing Charts, etc.

JamesNT

Latest Crypto virus now has the ability to delete backups from known backup softwares such as Veeam, Windows Backup, Altaro, and BackupExec.

Make certain your backups are offsite and air-gapped.

JamesNT

The easiest way to do this is use SFTP transfer so there are no Windows mount points to cross.

James and Indy,

These are very helpful. But, I have no idea what air-gapped and SFTP transfer and windows mounts are. I am guessing many others don't either.

Now I have preached probably three times in this thread alone that I backup two backups incrementally to an external USB drive and disconnect them. Maybe that is air-gapped. This is over and above my regular backups.

You can also disable the driver. I hope they can't get around that.

Bert,

Air-Gapped is the highly intuitive and technical term meaning there is absolutely positively without a doubt no way for "device A" to connect to "device B". Let's say Device A is your computer and Device B is your USB drive. Disconnect your USB drive, take it into another room and place it in a lock box labeled "DO NOT USE WITHOUT SIGNED AUTHORIZATION FROM BERT." There. It's air-gapped. The fact that you backup to external USB drives (have to have more than one) and AT LEAST ONE OF THEM IS DISCONNECTED AT ALL TIMES means you're pretty darn close to having an air-gapped backup. Real close.

Windows Mounts are when Windows mounts a device and assigns it a drive letter or other means to communicating with it. When you insert a USB drive and in File Explorer is shows up as Drive G, that is a mount point.

SFTP means Secure File Transfer Protocol. FTP by itself is insecure. SFTP is the secure version using Secure Shell (SSH) or SSL. Having a backup software send the backup over via SFTP would count as air-gapped (for the most part, GE would not let this fly) as credentials must be passed over for the connection to the remote site.

JamesNT

Thanks James for your information about the crypto viruses and the scary consequences. It really has me concerned.

Sometime ago one of the AC tech folks told me that I don't need to test their backups, so I haven't. I suspect that you believe that to not be a good idea. I would appreciate your comments and recommendations about my current situation, as I suspect that there are a lot of us who are flying solo in their 60s doing something similar to me. I think that your comments would have great relevance to many people.

I am on AC 9.1. My "main computer" (not an actual "server") is a 5 1/2 year old Dell XPS Windows 7 Pro 64 bit i7 machine with at least 16G of ram, 2 1T (RAID 1?)internal hard drives, backed up to a new Western Digital 4T My Book external hard drive. I use Acronis True Image 2016 (full version)for local backup, and AC Cloud. Acronis configures a drive image incrementally. I went with Acronis because the free version came with my first WD external drive, so I figured if WD recommended it, it might be okay. Billing software is EZ Claim, which my biller backs up to her computer off site. I have 5 local computers, both i5 and i7 which I personally hard wired to a Net Gear 1G switch.

My "mostly retired" former IT guy is traveling the Globe for months at a time. He recommends that I consider Dropbox or Amazon's cloud service which now supposedly are HIPAA compliant.

Thoughts, recommendations? Or should I just quit and go fishing? However, I do love taking care of patients, and solving their mysteries.

Thanks,

Norm Numerof
Internal Medicine
Vail, Colorado

Just a few ideas while you are waiting. If you are in a RAID config, it is either RAID0 or RAID1. RAID1 would be the choice so you have redundancy. Acronis is a great backup software. I stopped using them when they got too difficult for me to understand. But, can't go wrong. For me, the only thing is I like at least one full backup a night. Not sure what is on your main computer. Only AC? If not, then you are backing up to only one place and that is probably right next to the server. But, it sounds like you do have one good backup.

My biggest question is where is your EZ Claim software? Is it on the main computer? So, it is getting backed up by your WD drive. If it is, then you have two backups of the EZ Claim. If not, I don't think I would be comfortable with just having your biller backing up to her offsite computer. How does it get there? I just think 1. It may not be a good idea for two people to be involved in the backup, 2) Is it encrypted from the office to her computer and who has access to her computer, and 3) I only trust myself or a commercial IT person to do backups. I know nothing of your biller, but the only people who quit or I fire are billers. And, most of those didn't go very well. So, I wouldn't want a single employer to have the backups to what may be the most important data you have.

Do you have the name of that tech. Because I don't think I want to get support from that one. I do have to say that AC's backups are pretty good or were at least when I could restore them.

Norm,

I'll give you three reasons to test backups:

1. So you'll actually have working knowledge on how to actually restore your backups. Without this working knowledge, you are at the mercy of tech-support availability and you'll be sitting there praying every step of the way that you understood the instructions you were being giving correctly.

2. To have confidence in your backups. See reason 1.

3. TO HAVE AT LEAST SOME PROOF THEY WORK. Again, I say with conviction, if you haven't restored your backup to another computer to ensure they work then by definition YOU DO NOT HAVE BACKUPS. It's bad enough that even with all the testing in the world, come shat day your backups may not work. But at least with some testing and by going through the process you have at least some assurances and a real fighting chance. Right now, you have nothing.

I haven't used Acronis in 12 years so I cannot speak for it.

You should have your drives in a RAID 1. Never use RAID 0.

Dropbox is not HIPAA compliant. Please execute the person who told you it was.

I recommend two things: Make sure your drives on your server are in a RAID 1 config. And then test your backups by doing an actual restore.

For those of you who have not noticed, I'm no longer in a playful mood when it comes to backups. Starting tomorrow, I'm on my next Crypto recovery and, yet again, the owner of the company is already upset because he doesn't know the status of backups when I asked on the phone today.

And the bad guys want $15,000 this time.

JamesNT

James, you may want to be clear he should test the backup on a different database.

Well, I did say to another computer so by definition....

JamesNT

Going to be hard to do it on another computer if he is on v9+. It's not even worth it anymore. Activation codes and getting the backups tweaked by AC.

Good point. I keep forgetting about that. This is yet another reason why everyone should just virtualize.

JamesNT

Note to self: brush up on virtualization.

Can I get a link to learn about this virtualization along with an AMEN

First, there are tons. Google. Yes, some know better links.

But, I think I will try wasting my time for the 50th time, lol.

Just go to www.trainsignal.com aka www.pluralsight.com

Type virtualization in the search bar. There are 196 courses on the subject. NOT articles. Courses. Some are great. Some could be better. Depends on the lecturer. But, they are definitely qualified. I think most people never check it out. Most are turned off by the cost. Yep, doctors. It's a monthly subscription. Used to be you bought it on DVD.

There are TRIALS. And, there are all levels and every possible virtualization you could do SQL, Linux, Windows.

Here you go.

Less that an hour to install the graphical OS and the graphical KVM tools. Commercial support available if you want/need it.

Provide dinner when I'm in town and it will be done before desert.

Why James and I recommend virtualization. Do it once during the work week to get the instance certified, then never have that issue again. It will fit on a thumb drive, and runs fine on completely different host machines.

Also, once virtualized, you can make a copy of the instance on the spur of the moment, try a new version or some other software, and never have to break a sweat about what *MIGHT* happen.

Another reason Indy and I recommend virtualization:

Restoring to bare metal SUCKS [censored]. Just re-install the host OS and copy over a few files and DONE. You have restored. No more screwing around with drivers and praying.

JamesNT

Amen brother. We work from a set of baseline images, that we update every so often. I haven't done a Windows full OS install in a loooong time.

Matters not what the underlying hardware is, completely isolated.

Where's the link?

Fair question, what I get for multi-tasking

https://www.youtube.com/watch?v=G95cPAR5TEs&spfreload=5

another take:

https://www.youtube.com/watch?v=7hqX2eIMatM

So tell me how insecure/dumb my backup plan is, and how I can improve it. (I learned from hard experience that the AC Backup can not be relied upon to make accurate backups).

I run a Windows 7 Pro machine as my AC server. The only business software on this machine is AC and my PM software, serving 7 clients, and the server does not get used as a client. The server has some shared directories mapped to the clients, but not the AC or PM software directories.

The ImportItems and BACKUP directories (for AC) are backed up to the cloud nightly, and in encrypted form (using CrashPlan), as are the daily backups of my PM database. I have also tested the reliability of the daily, incremental, AC backups and believe them to be reliable. I also, at least twice a week, shutdown the SQL service and copy the AC database files (*.mdf and *.ldf) from my server hard drive to USB thumb drives (rotating), which then are locked in my safe.

I have a disc image of the server from when it was first purchased. So worst case restore scenario is a bare metal restore of the server to the state it was in when received from Dell, followed by re-installation of AC and my PM software, and then restoration of the databases for each (and the PM software installation/restoration would wait until after business hours, as I don't need it to see clinic, and billing could be delayed by a day or two).

How many versions are on CrashPlan? What is backing up the shared directories? I am assuming they have important files given they are shared out. Ransomeware can usually get to mapped drives?

The biggest deficiency I see is the lack of an image of the whole server nightly. But, this may be personal preference. It's kind of like what my anatomy teacher used to say, with General to Specific. My feeling is that a whole backup of the server nightly is the most significant backup. Then doing the specific backups would be great. Maybe use an external drive or two. You say worst case scenario do a bare metal restore....then restore the specific stuff. Why not have a backup from the night before. Restore to bare metal and be done. You have thought through the most important data, maybe even all of it, but with a full backup, you KNOW you have everything, and you are getting redundancy, because you are backing up the PM and the AC folder.

As it is, you have CrashPlan's software, but there is no mention of any other backup software. Win 7 Pro has its own backup software, and it isn't terrible, but it's not BackupAssist or Macrium or OO, etc.

There is also no reason you couldn't start on a Friday and back up the server to Crashplan, then do incrementals. You wouldn't be able to restore the entire backup quickly, but you would have access to every file. Hope this helps.

There are really 2 forms of crashplan: the cloud backup and the "free" backup. Free backup can be remotely to another computer at another location or to another computer at the same office. The cloud backup is where they make the money. The program is the same, you have to pay if you use their cloud backup.

You can backup to multiple sources. I have not used the cloud backup because I backup to multiple external sources (alternate office and home). I backup import items and the backup file. I do have the server with it's own separate backup, (I never set it up as a VM, although I am reminded through this thread that I should and that was my original intent) which I do not backup by crashplan, but probably should.

Let's see if I can answer these questions.
I am not sure what you mean by number of versions of CrashPlan. I use the CrashPlan cloud service, with all files being encrypted. As I understand it, the files are encrypted by CrashPlan before being copied to the cloud, so that even if CrashPlan cloud servers were hacked, I am the only person with the key to decrypt the files. AC Backup keeps at least 30d of backups, and so those are all in the cloud. The shared directories primarily have scanned copies of our old charts. Those have already been saved to DVDs, with the master copy in my safe deposit box, and a copy in our safe in the office. The only other shared directory is scanned financial documents (invoices, bills, payroll reports, etc.) and that too is backed up to CrashPlan.

I've thought about this and have decided it just isn't worth the effort. I can restore the server to new and would have to reinstall only three other software programs - CrashPlan, AC, and our PM software. That way I know I am getting a clean system, and there is not a file lurking around that may have been hibernating for a few days or weeks before screwing up my system. If I had an infected image, I'd have to go back to a previous image, which means I'd be restoring from other backups anyway, to have the most up to date data in AC and my PM.

I'm not sure why I need another back up program. Everything is safely encrypted in the cloud, and I can restore the system fairly quickly to a state where I can see patients, and then finish restoring the system after office hours.

I will answer your questions in a few. To be fair, you asked us to look and point out any deficiencies. There are two main ones.

The biggest deficiency is not doing a full backup of the server every night. I can't stress it enough. It is not a matter of you have a way of restoring everything. A good backup system has one, whole, intact backup. This isn't the only reason, but your method would take X amount of time. Restoring last night's imagine would take a lot less. And, there will be things you installed like updates and drivers and this and that, you will need to find. And, there will ultimately be a file you need that you thought you had backed up. What if the entire computer crashes. You are hoping that a backup a year ago will be compatible with a new computer with different hardware. Macrium and other backup programs can make backups compatible with different hardware. (If those work like that on regular computers). Most do on servers.

I could be wrong, but ransomeware can encrypt an encrypted file.

Versioning is the ability of a backup program or other programs to make multiple versions of something. So, if you back up ransomeware from last night, it is no good. But, there version is safe. It is different than backing up 100 copies, although I forget why.

To each his own on the number of backups with AC. But, for me, over 10, and I may as well close my practice.

Hope this helps.

PS I think good backup software would help. Just my opinion and recommendation. Not a big fan of copy and paste. Mainly for the overnight backup. Especially an image made for bare metal backup. On servers, they will back up SQL properly every 15 minutes or less. The backups throughout the day take on average < 1 second. And, you can set for as many days.

Here are my two famous quotes:

The best backup you will ever make is the full backup from the night before.
The next best backup you will ever make is the full backup just prior to making a change.

Like installing a new version of AC, for instance.

Short answer: I think what you are doing is fine. Maybe that is because my system is fairly similar (though I don't use Crashplan, I use iDrive). I don't think you have explained your system for backing up Imported Items, though.

Here are some principles that I have learned from conversations over the years with AC users:
1. You should have periodic backups of your AC files (enc) and Imported Items files. You should have them in the cloud, in your office, and ideally, locally on a machine but outside of your office (one that is readily accessible in an emergency).

2. The backups should be automatic; they should occur without any need for you to "do something" or "remember something" on a regular basis.

3. You should know where those backups are and how to access them in an emergency.

4. You should check the backups periodically to be sure they "work", and to know HOW to restore in the event of an emergency, as well as how long it will take to do so.

In my opinion, if you are doing these things, you are in good shape. My guess... and this is just a guess, is that if you are taking these basic steps, you are doing more than the majority (maybe the great majority) of AC users.

I have also learned this: No matter how you back up; no matter how many redundancies you have in software, hardware, locations and numbers of backups, etc... you will always find someone who tells you that you should do something more, because "What if....". Just keep in mind that everyone has their own level of comfort about this. It's not that anyone is right or wrong, just that opinions vary. Widely.

Keep your prime goal in mind regarding backups: your patient data is precious. You need to have it in the event of an untoward event, ranging from the common (a blown power supply or hard drive on your computer) to the unusual (a fire, flood or other catastrophe).
Another principle is to remember that if one of these events occurs, you will be unable to practice normally (or perhaps at all) for a period of time. That time may be measured in hours, minutes, or days. A secondary goal is to minimize that downtime. You have to measure the amount you choose to invest now in hardware, software, time (and your time IS money), and energy, and then balance it against the downtime cost of a disaster. It is not an obvious calculation, and the answer will vary widely based on everything from the number of providers in your practice to your own level of anxiety.

By the way, when you copy and past the .mdf files, there is no need to copy the .ldf files. SQL will regenerate those automatically.