Home Forums It's time to get real f&*(g serious about backups

First, my apologies to Indy for not having time to speak to him when he called me yesterday.

People, I come to you today begging you to take a new look at your current disaster recovery plans.

The reason I'm begging you to do this is simple: These crypto-viruses are ripping businesses apart limb from limb with horrific efficiency.

For those of you not in the know, crypto-viruses encrypt all of your data and the bad guys demand THOUSANDS OF DOLLARS IN RANSOM to give you the decryption key.

THOUSANDS OF DOLLARS.

Let that sink in.

And that's on top of the ENTIRE DAY(S) if not LONGER that your IT person will spend decrypting files and cleaning off computers from infection.

I've personally been involved in recovering a few networks from infection. The situations thus far have gone like this:

1. People call complaining about not being able to access files, files are missing, and all these *.mp3 or *.kpe files are all over the place. No one knows what they are.

2. I arrive on scene and determine that they have the crypto-virus all over the entire network because one dumbass opened a bad email.

3. Person responsible spends all day denying they opened an email from a personal account on a work computer. This person will most likely get away with their crime because they have some authority in the company (e.g. the boss himself, his wife, his kid, etc.).

4. I ask the owner of the company for their backups and license keys to things like Windows to begin the restore process.

NO ONE KNOWS WHERE ANY OF THE LICENSE KEYS ARE. NOT A SINGLE PERSON IN THE WHOLE FRIGGIN OFFICE KNOWS WHERE ANY OF THAT IS. HOLY EXCREMENT.

Backups have never been monitored. They've been failing for the past several weeks.

THE MOST RELIABLE BACKUP IS DATED 2/2/2017. EXPLETITIVE. EXPLETIVITE. EXPLETITIVE.

I explain all this to the owner who proceeds to freak out and BLAME ME FOR EVERYTHING despite the fact that we just met and I had nothing to do with the sh!tty backup process he has in place. I suddenly find myself very thankful I took that psychology class in college that explained the 7 stages of grief. Step 1: blame the nearest IT guy before taking any responsibility yourself.

The bad guys want $5,000.00. The owner DEMANDS that I "geek person" all this and decrypt the files using "my geek powers" or "what I was taught playing Warcraft" to fix all this without his paying the money.

Seriously, James, why can't you just hack these files open? Didn't you see that part in Jurassic Park where the teenage girl hacked into a "unix system" to get the stuff they needed? Why can't you do that? Or what about all those episodes in the show 24 where Chloe always got around encryption? Sheet doesn't look that dam hard. Why do I have to pay these Russians money?

Another day goes by. The owner finally pays the ransom.

I spend another two days setting up computers, cleaning them off, fixing Active Directory after the unholy damage crypto did to SYSVOL, and all that.

By this time, the owner is out around $30k after paying the ransom, my labor, and all that downtime.

If only I could have restored that backup. If the business had a recent backup that I could have restored in a few hours, this would have been an inconvenience rather than an unmitigated disaster.

Check your backups. Restore them to another computer. MAKE CERTAIN YOUR BACKUPS WORK. The business you save could be your own.

All the bad stuff you've heard about Crypto is REAL.

JamesNT

Hi James,

Thanks for looking out for us!

To be clear- you are talking about OFF SITE backups because anything on site (external hard drives other PCs etc) could be involved?

Thanks

Gene

Gene,

Yes, thank you. Offsite backups are needed. Local drives can be encrypted by crypto as well.

And, make certain you test your backups to make certain they actually work.

JamesNT

With the V9.x series, it becomes more difficult to verify backups with the "phone home" technology included.

I agree multiple forms of backup are the key

James, thanks for bringing this to our attention. No matter how hard we try to keep current with this, it remains a challenge. Now is clearly the time to stop procrastinating and get back on schedule.

Wendell, this has always been an issue and I agree it has become more of a problem as AC gets more and more tied to the internet. I have not done this in V9 yet, and now is the time. So let's talk about the specifics of how to handle it, and see if you guys have a suggestion for me...

The way I have always verified back-ups is by using the enc file on a computer that is not connected to our regular network. In my case, I use one at my home. What I will do now is get a "key" from AC, install the program on the home computer, then do a restore with the enc file from work. One problem is that the home computer must be connected to the internet to see if the back-up functions appropriately. On the other hand, I have found that if I leave it connected, things get screwed up. For example, certain labs arriving via interface show up on the home computer. So my questions are: 1. am I still describing the best way to check a back-up, and 2. what is the easiest way to leave AC on the home computer, leave the computer connected to the internet, but "turn-off" the connection from AC to the internet until I want to check a back-up again?

Two points on Amazing Charts Backup:

1. You most likely have other information on your server you need to keep besides the AC database.

2. AC backup does not cover that other information.

I would look like virtualizing your important servers and using Veeam for your local and offsite backups needs. I wouldn't even bother with AC backup.

www.veeam.com

JamesNT

James - I have AC stick a .enc AC charts back up file into my dropbox pro account nightly, which goes to a number of other computers not at work as well as the cloud, and another on a connected hard drive on the computer ( small office , 2 computers on network, one desktop computer is server ) . Is that bullet proof enough ? small potatoes here .

Have you tested restoring one of those backups to a test computer to make sure it will actually restore?

If you haven't then you don't know what you have.

JamesNT

James, Could you take a look at my question above, please?

Dr. Jon,

1. Yes.

2. The AC server assumes it is the only AC server you have running, hence the problem. One way may be to block certain IP addresses that AC reaches out to via your firewall at home.

JamesNT

Is there a service I can turn off easily that will stop -only AC- from connecting?

Remember with Ver 9 the test computer has to be certified by AC as a legit server which leads to the question does anyone have the experience of having 1 identified AC database being loaded onto 2 separate verified by AC servers?

If I can get this done, koby, I will let you know!
You realize that what you are basically asking is the embarrassing question, "has ANYONE on V9.1 or above tested their back-ups"?

guilty as charged sad to say

Good advice James. One major thing I would like to add is that you should notify your IT Provider immediately when you start noticing that you're unable to open things like II. I've had a few offices call me weeks after Crypto had spread through their entire system.

Sometimes they are introduced by bringing an infected machine on the network.

Some other pieces of advice:
1) It's almost always introduced via email. Get a business class email with threat protection like Office 365.
2) Secure your file shares. Do not allow Everyone to see shared files. This is how machines that are added to the network attack.
3) If you use a NAS for backup, make that login separate from your domain and only accessible via the server. I know it seems like a good idea to join it to the domain but don't. At least not since Crypto became a major player.
4) Do not use your main computer/server as a client. Cryptolocker does not have the ability to access locally stored files like a backup drive attached to a server.

Comment on Number 4: New Crypto does. It encrypts local documents and one version goes as far as to encrypt the entire C drive except for the Windows and Program Files directories. Also will go after USB drives.

In fact, the last infection I cleaned I found that Crypto had created a whole new Group Policy object such that as soon as a machine joined the domain then the group policy object would create a scheduled task that would run crypto.

JamesNT

What I meant on number 4 is that it can't encrypt local files on other machines. Only network shares. That's why you shouldn't use your server as a client machine.

Only way Crypto could've made a GP is if it had domain admin permissions. I assume in that case someone was using the server as a client? Or used a domain admin account?

Domain Admin permissions are indeed required to set up a GPO.

I don't know what the hell happened at that particular client's office. And I don't want to know. At this point I'm just praying there is enough of the client left to pay my invoice.

JamesNT

yes have used them unfortunately many times :-(

James and Sandeep,
A question based on the above discussion...I have an Iosafe drive connected to my server. As one of my back-ups, I have a copy of my enc files and imported items sent to that drive. Is that back-up vulnerable to Crypto?

JBS,

If that drive registers as a drive letter such as E, D, or G or other letter in Windows, then the latest crypto will encrypt it and your backups will be available only after you've paid some slime bag in Russia some serious coinage.

JamesNT

JBS,


Not that I'm aware of. The service that phones home also does other stuff you need so you can't just turn it off.

Firewall time.

JamesNT

To be clear... I don't need AC to run while that service is off. I want to install the program, restore a back-up (with the service on) to test it, and once that is done, turn that service off until the next time I want to test the back up.
Please don't make me mess with my firewall.

Then you're all set.

And believe me, all of you, once you successfully restore a backup and you KNOW your backups works, you'll have the same feeling you would have if you paid off your car, your mortgage, and all of your student loans all at once.

It. Feels. That. Good.

JamesNT

Been a long week, and I'm still working, but let me weigh in about some things that have been asked and not answered.

Once the server is authorized by AC (whole subject all it's own), and you restore the DB, then after you log in and validate the last few notes you signed are gtg, then you need to put that install to sleep.

You do that by first DISABLING interfaces. You will need to check the inbound directories for results that are pulled in by the interfaces before you shut them down.

Every time you restore, you will have to do this again.

You can use the same methodology we use for securely moving files from your test to production server, I'll cover that separately.

Next step is to go into services and stop and disable all the AC services. Depending on your version, there are 4 now.

That will effectively make that server inert until you want to bring it to life again.

When it comes to moving files, sync'ing files we recommend good olde SFTP.

Even if the storage is on your network, in the cloud, or around the world, you can securely move the files, sync the files, all without giving the Windows based malware an attack path.

For Windows, there is even WinSCP, which allows you to script the transfer and sync'ing process.

Describing this process is a subject in itself, but the point is accomplishes the desired effect without using Windows mount points.

Yes, we have regularly does this as we move clients into our environment from ACInTheCloud, or their local server.

We bring up an instance with the practice data, get it "authorized", and then test the restoration.

When it is time to do the production conversion, all of the preparation is done.

As long as you are meticulous about being clear about shutting things down after a restore, and moving files that are inadvertently pulled down, you can have a backup "server" ready to go.

I would love to see a comment from someone at AC, about why I have to do this to protect my data, since moving to ver. 9 (which I was encouraged to do by AC, without disclosing that it would render backups impossible to restore without their assistance).

thanks James !

all those are paid off already, but I still don't feel that good!

and yes that was a backup for V 9.1 I think, so the backups are still working.

Lynn

AC will have to answer themselves.

I will say that I have told them repeatedly that we are fine with taking night/weekend calls, but they need to make the process of requesting a new server certification self-service. Make it something that runs from the client portal that is automated.

@ryanjo

I had a conversation with someone at AC about v9 and above. About the backups, about the activation key, about the install issues. I said everyone just wants it back to v8 style again. Sounded like I was getting somewhere.

Then we had a CAB meeting. What came out of that is that everything is super.

_______________________

As for backups, I can't echo James' comment enough about entire backups. It seems people only worry about AC backups. I just restore AC with my main backup. Given many people take a drive home anyway, it's not a bad idea to disconnect the cable to a backup drive. Even if you keep it at the office.

Virtualization for servers is going to be key. Bare metal restore just SUCKS [censored]. I mean, seriously. Restoring bare metal then screwing with drivers and all that is for the birds.

I am well aware of the fact that some practices are just starting out and sometimes cash flow may be tight. But when some Russian jackass demands $5,000 to give your own data back to you and you're into your IT person for three days of labor, and you've been down for half a week, having a server with Windows Server 2016 running a couple of virtual machines and Veeam doing your backups will seem mighty cheap in comparison.

Again, The horror stories you've heard about Crypto are REAL.

JamesNT

Threads like this are why this is the best forum for any EHR bar none. Kudos to AC for hosting it.

I find for small business IT in general, the advice and discussion here is better than anywhere on the webs.

For backups how long do of a backup history should we keep? Since crypto slowly corrupts files before we recognize it, how long is enough that we could go back and find uncorrupted copies of specific imported times. In other worse what's the longest one should plan for between infection and detection a Crypto virus?

With VMs is it any easier or harder to "authenticate" an AC restoration on v9.x?

I run 1 2012 R2 essentials VM for domain and AC. Using Windows built-in backup on the host "Server 2012 R2 essentials" to 2 alternating usb hard drives. The windows backup is a bit obtuse guess i should try veeam? advantages?

Other general security/backup/crypto prevention recommendations? I'm considering using "Sandboxie" for employee web browsing. Or buy them all Chromebooks and tell them to use those on break?

The only thing better is if you have a SPECIFIC need or troubleshooting and you need world class experts to follow you in a thread, then you need Experts-Exchange. Just as an idea, I have 600 questions in EE (there is a $9.99 subscription), but there is tons of information on it. And, if your problem is fixable, they will fix it. And, start in about 20 minutes or less.

How long of a backup history should you keep?

My backups go back three years, but that is due to a weekly backup to a 3TB drive using SIS. I just take the drive home after a year and rotate them.

My normal backups of the server go back six months. Sure, you really want the file from yesterday, and you wouldn't restore from six months ago, but it is nice to be able to go back and get a file from a long time ago. Say you made a spreadsheet which took you hours to do. Then, you stopped using it. One day, you just delete it. Three months later you need it. You have it in your normal, daily backup. And, you definitely have it from two years ago. Storage is CHEAP. But, this doesn't answer your question.

Since ransomeware slowly corrupts files before we know it is there, how long is enough to go back and get unencrypted files? How long between infection and detection? Imported Items?

First, let's look at database backups and your OS backup, etc. While it's nice to be able to do a bare-metal backup (just go with it for now -- I know bare metal isn't perfect, but it will save your [censored]), and have your entire server back to square one, you really don't need the OS backup as it can be installed again with the data restored separately. This is why it is nice to partition your server or have actual different drives. This is why it is nice to back up everything, but also back up just the data. But, if you used a back from two weeks ago, you could handle the hit to the OS, but you would lose hundreds of patient encounters and files and labs and II in that time. Not to mention your billing. Information like an AC backup or a Medware backup is meant to be used in one chunk. Like I don't keep many more than five AC backups, because that is too much data to lose. I depend on my full backup.

So, how far to go back? You want to go back pretty far for your full backups, but only two or three days is important for billing, and AC, etc. II is a little different, because if you have a backup two weeks old, and it has pristine II, that will be about 98% of your II.

So, the way I look at it is:

1. I want to get back up and running quickly: Full image backups with bare metal CDs to boot from. These go way back.
2. Crucial database information that would cause me to go out of business:
-- AC -- II -- Billing
I want good backups from one to two days before.
While AC could be backed up online, if you back it up locally, you want to back it up every evening to an external drive. A drive letter is bad. But, just make sure you disconnect it. Ransomeware hasn't evolved enough to jump through the air and attack it. If you have two USB drives, unplug one. If you get home and forget, remote in and disable the driver. But, I would guess if you had all of AC from that day and all of your billing and scheduling, you would survive.

Now on the infection/detection. Ransomeware can encrypt your files in under a minute. Sometimes longer, but usually never longer than a day. Once it starts, you likely can't stop it. But, the actual infection happens sooner. It generally gets on your machine, makes some registry keys and his somewhere like a user profile. But, the trigger is generally something like a restart or logon. It doesn't take long. It won't be days to weeks. The hacker wants your money and doesn't want your A/V to pick up the file. Eventually, the A/V companies make definitions for these.

Two things you can do. Educate like crazy. Also, they are starting to make ransomeware detection software with the A/V suites. They are known as zero-day detection. They aren't looking for DNA or signatures, because the malware may have been made today. They are looking for a small program that is doing things that don't appear right. Such as making random registry keys. Or even if they start randomly encrypting five Word documents in a row. You can Google these. I have one from MBAM and one from HitmanPro.Alert.

But, mainly back up AC and your billing and anything else mission critical to an external drive and disconnect it. You can also back these up online. I can't say for sure that even online backups are safe. The AC backup is done via incremental. Really nice ones would be ones that back up Backup1, then Backup2, etc. so that Backup1 isn't exposed to the public.

Authenticating VMs on v9.1

Good question. We have been told some of what the new install looks for. I am going to say it is the same. A virtual machine is basically a completely new machine, although it does use the hardware of the computer. However, I believe it is more registry based or something about the VM. If the computer never had SQL, then it likely won't install without authentication. VM would just as easily. That is my understanding.

Veeam or other:

I am all about spending money. You would have to spend at least $1,450 to backup SQL. Around $2400 for the whole deal. It is a good way to go. But, there are other backup programs like Macrium and BackupAssist that are much cheaper. But, think about it.

Sandboxie:

Never heard of it. Googled it. It looks like a VM that just fits on the drive. There is also software called Currentware that allows your staff to use only the sites you choose. Inexpensive, bulletproof and the best support ever.

Veeam Backup Essentials is less than $1,000 and may work for many here.

https://www.cdw.com/shop/products/Veeam-Backup-Essentials-Standard-for-Hyper-V-license/3085689.aspx

Veeam Backup and Replication for one CPU socket is also less than $1,000.

https://www.cdw.com/shop/products/V...amp;programidentifier=3&RecoType=RSR

I would think backing up the Hyper-V Virtual Machine in its entirety then restoring it would preserve Amazing Charts' certification of the server. However, I have yet to test that.

If using Veeam I recommend keeping at least 1 week's worth of backups AND BACKING UP EVERY TWO HOURS. Veeam makes this easy because of its "forever incremental" approach and it's great ability to deduplicate files and so forth. It's also a very fast backup.

Ransomware does NOT occur slowly. It goes through your network as fast as it can encrypting as much as it can so the bad guys can maximize the ransom demand.

JamesNT

I think Larry wants it to go even faster than that.

He can go faster than that if he has the server and storage for it.

JamesNT

No faster. Daily backup is pretty good for me. Maybe will change to 2/day.

I had contemplated server replication but never got around to it.

For employee surfing: Chromebooks, sandboxie, or outright personal email ban?

There are better options than two a day. Especially if you are going to disconnect the drive.

When I said faster I meant you wanted it faster than it occurring three weeks prior. You would want it like right in front of you. So, the backup from the day before may still be good. But, all the files would be encrypted anyway.

Chromebooks would be an excellent choice. You can now get over 16" displays for less than $300, and they can be used in the practice if you are running a terminal services solution for rooming patients, vitals, etc.

They are extremely hard to jack-up, by default store no patient data, and are not Windows based, which is where the predominance of attacks are focused.

Increasingly, practices are using a variety of platforms (e.g tablets, Macs, Chromebooks) to interact with our platforms and AC, and generally happier to use the tools they want.

Each Chromebook can have several user profiles, but their usage is segregated.

A Chromebook wakes in about 2 seconds, and can default to a password to unlock each time it is opened.