Home Forums Discussions Problems Ransomware alert - it wasn't updox

Friday at 9 am my office was attacked by ramsomware. All the files in my server / emr were encrypted. Thankfully I had a recent backup. It took all weekend to restore our files by IT. We lost 1 day's work. We finally (we think)figured out out where the attack came from: updox. See the attached screenshot of files / fax received via updox on 3/16/18 at 9 am. We believe this is were
it started. That is the time that everything went to hell. Beware. I sent an email to updox just now.

Gerardo



edited to erase patient stuff.

Did someone open a file from updox or download a file to the server from updox? Is that the attachment seen on the Updox page?

Guess I am wondering why the To: and From: are from the same person.

Thank you for the heads up. Scary!

Good question Koby. I have no idea, but I suspect so. I was not in the office when the attack happened. I have 10 office staff and I guess one could have opened it. They are not computer savvy. Like I said, we believe the source to be "possibly" from updox. I frankly do not know and likely will never know, but I suspect so. I did not contact the ransom website noted. I did not ask them how I got it. My IT persons (hero) decided to follow a "scorched earth" protocol. Unplug all usb devices, Delete, format drive several times (actually considered buying new one), reinstall os, upload backup. We get so much incoming faxes via updox, however, which makes it a good vector for infection. I have seen file attachments that look like this in updox before but they are usually pictures uploaded in bmp format. This one said java. First time I see that. Could it have been uploaded from my sever to updox? maybe. Also, I did not notice this file in updox until server was up. I sent the office staff home Friday morning once it became clear there was nothing we could do until server up so I was the first to look in updox (today).

Yes, Gerardo, thanks very much for posting this.
You are experiencing what many of us see as a nightmare scenario. Hopefully you have navigated it successfully.
Of course we know not to click on email attachments from unknown or unexpected sources, but I was not aware that an attachment could come with an electronic fax.

A couple of questions:
1. Did someone click to open one or both of the attachments you show?
2. Did having a back-up enable you to completely restore your data with the process you describe?
3. Any follow-up from Updox about this? (I suspect the issue was not Updox, per se, but any electronic fax process would have done the same, but I am interested in their opinion).

Please continue to provide follow-up!

Jon:

1. I don't know. I expect someone did. I marked both attachments as spam in updox and went into archive folder and deleted both. I use professional gmail with my own domain for my office mail. They say they keep an eye out for this sort of stuff. I have not gone through the employee's mail to see if there is a suspicious file. Should I? I do not know where to begin. I have 12 users (including 2 physicians) Suggestions?

2. Yes. Took long. Lot of data. My imported files alone are over 100 gig. I have a 1 gig switch. I have two NAT, one encrypted and one not encrypted as a "soft backup?". Non-encrypted NAT bit the dust, was infected as well. Encrypted NAT was safe. Don't ask me what it means, my IT guy (cousin) knows. He also works for IT at UCSF. He said getting a 10 gig switch would have made the process faster, but it is very expensive. So, considering this is the first time ever having to restore my entire system completely, not just emr or billing system... even domain controlled had to be rebuilt... It was not a big deal. Lucky it happened Friday when I do not have clinic, my associate was out, and I was in the OR. David said it was mostly a lot of waiting for unencrypting, formatting, and waiting for backup to upload, etc. My son helped, he is interested in computers. Since all computers had to be rebuilt, we had to start with a "brand new" computer. This was actually an old server that David told me to keep around "just in case". Actual work was only a few hours. This does not include the anguish and anxiety experienced by yours truly...

3. No. I just notified them at 3 pm today. I got the usual canned response. I am not sure updox is the source, but if, somehow, the file was uploaded to UPDOX from my server, I want them to be aware so they can take any steps necessary to prevent its spread.

On a side note, my network is running super fast now. There is always a silver lining...

David tells me I have to relinquish my admin privileges for my regular user account. I can have one user account with admin privileges but I am not to use it regularly.

He thinks this started with a user with admin privileges as the other user accounts do not have privileges to install programs.

My office manager, my associate, and I are the only ones with admin privileges.

I have been demoted

Also told me to get malwarebytes professional. I have the free version but it does not have real time protection. Also, he says office staff often stops the program from running because it slows down their computer. Some computers had not been scanned for weeks, apparently. More educational opportunities for the staff.

We use the free microsoft antivirus. It was up to date.

I need to correct one of my statements. Earlier I said: "We get so much incoming faxes via updox, however, which makes it a good vector for infection."

Currently incoming faxes go to my land fax line and are saved as a pdf file in a folder which is automatically uploaded to updox. (I am cheap, don't want to pay for receiving faxes) I only use updox to send faxes out. Which is why it is so interesting to see this "fax" file with that format.

I need some more information. Was it Updox itself, or just the fact that Updox was the point of entry for the pdfs. Generally, the only way to get malware via a pdf is from an embedded Word document where the user would have to enable editing.

As far as Malwarebytes free, not sure if it is supposed to be run commercially. You can use it at a 3rd party. Malwarebytes Endpoint Protection is excellent. These days, it is crucial to run some type of malware prevention that protects against zero-day exploits such as ransomware. May have blocked the one that came in. Good job on the backups. Were they disconnected from the network.

That is also bizarre that the staff turned off the program. I don't think we would have one employee who would figure that out, unless it was just ridiculously slow. I doubt I would have. Time to get rid of local admin rights. And, take the icon off the system tray and keep them from finding it. Plenty of 3rd party apps which will block tampering with an app. Yeah, but going to a corporate edition of something is crucial. You definitely get what you pay for. And, most malware companies are getting away from DNA definitions and going with behavior-oriented.

I do not know if updox was the source. I thought so when I made the first post, but the more I think about it, the more doubtful I am. Frankly, I have no idea. Maybe it was my server that somehow uploaded the file to updox... Well, I hope making other ac / updox users aware will help them avoid the anxiety and grief I just went through.

Backups were disconnected from network and encrypted.

I have been using malwarebytes free for years. I do not recall ever reading that it was not to be used for business. You pay for real time protection, I think. Never saw the need for real-time protection... until now...

David says it would almost certainly have caught it had it been active real-time.

I do not know much about ramsonware. I am getting educated fast. Agree with getting rid of local admin rights. David says he will be making a lot of changes in the near future, but for now my server is back and running. Monday we will have to educate staff and reconstruct the half day of work. Thankfully, no notes were made in AC. Two new patient charts were created which I already recreated. Missing some imported stuff and paperwork. Only lost 1/2 day, maybe only 2 hours of work. Busy clinic tomorrow morning.

This really looks like Updox just uploaded the temp files that the ransomware used while encrypting the files in your Updox Central auto-upload folders. One file for the fax upload folder and one for the general upload folder.

So a result of the infection, not likely the source.

Not to say Updox couldn't be an avenue for an infected file, since it seems to accept any file type...

Good points. I don't know much about Updox. Thinking only PDFs. Word documents would be another thing. But, they would still have to be changed to Edit.

Having your backups "air gapped" should give you 5 stars! Encrypted is good, but it wouldn't stop ransomware if it attacked the backups.

Gerardo, I have the same set up as you.
I was thinking the same as mczdsm: it is not that Updox caused the problem, just that it imported the fax that contained it. (You wouldn't say Gmail was the source of the problem if it came in an email attachment).
The urgent message for now, already transmitted to our staff, is NEVER click an attachment in Updox.
Just like you don't click it on an attachment in a suspicious email... but in Updox NEVER do it.

We get over 100 faxes a day. None of these have malware. Word files can be embedded in PDFs, but again, they would have to be changed to Edit from read only.

As to A/V. Going back to Win 95, I have not seen an A/V that doesn't protect in real time. Any time a drive-by site attacks or a virus is activated or just gets on the machine, if the behavior or if a definition spots it, the A/V goes off. It doesn't wait for a scan.

Most of the high-end A/V, anti-malware programs continually go through each file and check for malware.

https://www.malwarebytes.com/premium/

The other thing is a lot are adding cloud protection, meaning the agent connects to the cloud and gets all of its information from there, updates and the like. They also have central monitoring and installs, etc.

OK, maybe 50.

I bet if you asked Gerardo a week ago if he ever got malware in a fax, he would have said never had either.
I am not saying that is where he got it, just that we should take appropriate precautions.

I didn't say he didn't get it in a fax. I said he didn't get it in a PDF. He would have to get it as a Word file EMBEDDED IN a PDF. This would still need to be changed to Edit from read only. Technically, you could call that a fax, but it's an important distinction. I don't want to ask Gerardo anything.

If his system is like mine, the way I understand it, fax files are received as TIFF files and then converted and imported to Updox as pdf's. It looks like a pdf in Updox but it originated as a fax (TIFF).

Hey everyone, Tobin (at Updox) here. It's been a while since I've been out here, but I wanted to jump in today to provide a little more information. Hopefully, I can fill in a few gaps, although I can't provide the 100% complete picture.

Odd from/to labelling
Dr Carcamo mentioned he is using Updox Central to upload PDF files that his fax server has generated. Updox Central can watch 2 separate folders for auto-uploading files into Updox. They both work exactly the same way, except files in 1 folder will show in Updox as "file" and the other will show in Updox as "fax". This is purely for convenience, so users can easily tell which items came in on their fax server (Dr Carcamo's use case is exactly the reason we built this feature). But, this is why his faxes are showing up from "South Texas Surgeons" to "South Texas Surgeons", Updox Central always uploads items from the practice to the practice, because it doesn't have any more information. It doesn't know the actual fax number it came from (or was sent to), it just knows what practice it is connected to.

PDF files
While PDF files may be received via email, or uploaded by Updox Central, or manually dragged-and-dropped into Updox, Updox never actually presents these PDF files to the user. On our server, we process and break the PDF into its individual pages and store those pages as PNG files. Those PNG files are the images you see in Updox, both in the desktop client and the web app. The actual PDF is never transmitted/displayed anywhere. If user imports something into Amazing Charts, we actually take those PNG images and build a new PDF out of them. So, even if an dangerous PDF makes its way into Updox, it is immediately neutralized and 1 of 2 things happens: either the PDF is so badly malformed that we can't even turn it into images and you would see nothing in Updox, or we would turn the PDF into one or more PNG images and you could view them in Updox, but there would be no risk as PNG files are not a possible vector for a ransomware attack.

Weird Java files
I believe the 2 weird Java files are indeed, as mczdsm above postulated, the result of the ransomeware encrypting files on the server and Updox Central uploading the resulting encrypted files. We see 1 of these files labeled as an "upload" and 1 of them as a "fax" (mirroring the Updox Central auto-upload capability) and while Updox Central knows it is not supposed to upload the Thumbs.db files that Windows creates, when the ransomware encrypted the 2 Thumbs.db files in the 2 upload folders and changed their names, Central happily uploaded them. I looked at the files and they are clearly encrypted files, I don't think that they are anything that represent a risk. Even if someone tried to download them and double-clicked on them, your computers would have no idea what to do with them.

Virus scanning
Updox does do virus scanning on inbound email attachments.

And finally...
All that being said, it's wise to treat Updox like you'd treat Gmail, especially if you're receiving email in Updox. You probably don't need to download and run something that somebody emailed you in Updox and told you to double-click on. Treat external links in email carefully. You know the drill, be smart out there.

I hope this helps answer some of the questions. I'm not usually out here on the forums these days, but I'll keep on eye on this thread for a day or two.

Tobin

Tobin,

Thank you for your thorough and very detailed information.

Thanks Tobin, and everyone elses' input. I have been very busy and emotionally drained from this whole ordeal.

It is now obvious that UPDOX was NOT the source. In the "heat" of it all I thought it was. We do not know where the attack came from, likely never will.

Now have malware bites real time scanning in all computers. $3k for 20 stations for 3 years, but far cheaper than paying $70k (?) ransom for their data as UCLA recently did.

Java is now up to date. Some computers had old java as updox desktop app was not working with newer java way back then. This issue has been corrected per updox; it now works on newest java. It would have been nice for UPDOX to send a note about this, maybe they did and I did not notice. As we well know, inertia takes over and it is easy to let things be the way they are. It takes effort to change. "if it ain't broken don't fix it" Well, it did break, big time. Well at least the data was not compromised. IT tells me no data was ever transferred out, so you will not be reading about me in the NYTimes anytime soon. Maybe just as an epitaph.

All admin rights have been revoked. No-one can stop antivirus / malware bites scans anymore. No more installing programs on your own. A pain to have to ask for anything to be done, but such is life.

We were back to working order Monday, as if nothing ever happened. Minus 2 hours of work. And a lot more gray hair.

Part of the problem is that my associate and I usually access out office via rdp (used to be logmein but rdp is free) from different computers in different hospitals to do our charting. We also access other hospitals via rdp / cisco (methodist, baptist, CSR) to view patient info on their EMR / PACS). We also access from home, but at least we have control over that. We have no way of knowing if one of those hospital computers had an infection and can't control for their antivirus / malware status.

Any advise on this regard would be appreciated. Obviously best solution would be to have a completely isolated network, but that is not practical. We need real time access to our data to best care for our patients, chart/bill, and keep our overhead low.

Correct.

Yep.
Maybe I haven't yet.
Most likely fax was not the source of the attack and the infected files / faxes were uploaded to updox from my server

I get pdf faxes, lots of them. 2,860,200 faxes since September 2011 to be exact. I do not recall ever seeing a pdf with "fillable fields" if that is what you mean. I use pdf forms with fields I can type on internally. I thought about putting them in the website / updox, but currently I just created a form in updox that patients can fill. Not very helpful, though, as it does not go straight into AC. Staff does not usually check to see if the patient already sent a form so patients are asked to fill out another one when they get it. I have to fix this one day.

You can ask me questions, Bert. I don't bite... Hard...

Corrected number of faxes. Made an error in calculation.

I do not know how to change the tittle of this tread to reflect the consensus that UPDOX is not the source. If someone know how, please do, or tell me how to.

Thanks Bert.

Actually David says what really saved the day was shutting down the server so quickly, which was my son and David's decision (I was in the OR and found out after the fact). If given time, it probably would have infected the entire system, regardless of encryption or not.

This is the timeline of events:
I was charting via rdp from hospital computer on a procedure I was just finished a little after 8 am
I was kicked out of rdp suddenly.
I reconnected and tried to log on to AC; window pops up saying unable to find database, please find path...
Called to OR as next case ready.
Did quick case, 1 hour
Meanwhile Staff started to be kicked out as well and import pdf files were unable to be opened in AC. Soon spread to all user files.
Staff called my son who is my local IT man; he was in college taking class.
get call from son around 10 am, "I shut down the server we are having ramsonware attack"

I changed the title for you. That is something that you can't do. Not to brag too much, but generally PM me with those types of questions.

Good choice with Malwarebytes. They are rated in the top three for anti-malware and anti-ransomware. Still make sure IF your users use email at all, that you have a meeting monthly to go over the best way to be safe and not fall for spammers and hackers, etc. The spear phishing is getting really good.

I am assuming you are using something like Malwarebyte Endpoint Protection maybe using their cloud product. Keep in mind that MBAM is not a substitue for an antivirus program. It does not include antivirus. Their support will generally tell you that you don't need a separate antivirus, although I run the above and MSE. Have to keep in mind that you may have to exclude AC from MSE. Or you can run other free antivirus programs, which work with definitions.

Finally, while anything can happen, I don't think you are at too much risk when you remote to your hospitals. Citrix and Cerner plus Juniper and other software used provides you with a connection to a secure https:// web portal. You are only passing mouse clicks and keyboard strokes and accessing the information on their end. The information is not passing back and forth. I would suppose that if you have the ability to download a file, say from PACS, you may be slightly at risk, but I would very much doubt that you would be subject to malware.

Bert, maybe you can start a thread on the best antivirus anti malware programs and why. My tech people recommended Vipre and I used them for years. Then there is the Windows Defender that comes with Windows 10

I will try to work on that this weekend. Pretty busy.

breaking it(needs for AV/malware) down by server/client would be nice

Got it. NP.

Just having had the chance to get to this. I will try. Mainly, it is OK to run a full anti-malware program such as MBAM EPP or others. They should be a version which is OK to run on a business network. There are many good anti-virus programs working with definitions that you can layer with it. Personally, I think MSE is good enough for up to Win7 and Windows Defender for Windows 10. It is debatable as to whether you should run an A/V, anti-malware program on a server, and it is only used as a server, you should't be on it anyway. But, malware can move across the network. I feel more confident just running Windows Defender on it.

Not a very good answer. Most antivirus and anti-malware programs these days are very good. You just want to make sure the anti-malware program has anti-ransomware as well. It's not perfect, but certainly better than nothing. Some have tools to fix the problem and there are 3rd party programs as well, but I have never tried to use them.