Home Forums Discussions Problems AC Totally Down

Is anyone else's AC in the clouds TOTALLY NOT ACCESSIBLE??? We are seeing patients and have NO ACCESS TO CHARTS. AC not responding to our requests. HELP!!!

David Lee, MD
IM
Dallas, TX

Apparently the hosting company had a power outage. Don't they have backup power??? Still down.

David Lee, MD

My nurse initially couldn't sign on, I just did, seems to be working now.

Yup me too.
Awaiting the long queue at support.
Patients lining up.
Dead in the water .
Bill

I am number 3 in the chat. I will let you know how it goes.

http://amazingcharts.com/support/help-me-now/

Then push CHAT and you will be put in the queue

30 minute wait then 5 minute fix

Ours is back up. When I did the online chat it was 117 minute wait! I'm still concerned the hosting company doesn't have backup power???

David Lee, MD

I think each service needs to be restarted manually. The power is ok now.
They should be monitoring this and should restart everyone with a cloud connection without being asked to do so.

If you never got through to support they must have done exactly what I suggested above. I have been in the cloud for 5 years and this is the first time this has happened.

Don't know where they co-locate, but that is the reason we pay the extra to co-locate in Tier4 (Telco-grade) facilities. multiple separate fiber connections, those semi-size diesel generators, and that gas-turbine engine that spins up in seconds to keep the bus clean while the diesels auto-start. Full equipment redundancy gets expensive as well.

Interesting point... so AC's hosted version went down.
Indy, when was the last time your hosted version died for clients?

Just a followup from my earlier message. We had no issues at all today other than for the first five minutes of trying to log in my nurse could not. I believe AC in the cloud is revamping to a newer modern infrastructure. I recieved an email about this on August 8, so I wonder if this may have been part of the issue today.

AC told me it was a power outage at the hosting site. Sort of scary honestly.

David Lee, MD

In our seventh year now, we have never had our infrastructure go down.

We spend a significant premium for Tier4 facilities, which means they have redundant power, redundant batteries, redundant fiber connections, redundant on-site power generation.

We regularly upgrade hardware, including firewalls, switches, hosts, network storage, both growing with demand and endeavoring to make our infrastructure resistant to point failures.

A message from our hosting provider ITelagen:

To our valued Amazing Charts practices,

I am John O?Keefe, the CEO and founder or ITelagen Healthcare. We provide HIPAA secure hosting to over 3,000 providers nationwide including all of Amazing Charts? hosted clients. As you know, Amazing Charts & ITelagen experienced a significant service outage this week. As CEO of ITelagen, I want to offer a profound apology.
I also want to explain what happened, what we?ve learned, and what we?re going to change. First, you should know that our services are now stable. There has been no data loss and there were no security breaches.

Here?s what we know.

Around 5:30 AM EST on Monday, August 14th, we experienced a significant issue in our VMWare cloud platform that affected 25% of our cloud infrastructure. This affected a significant number of servers in our Nevada datacenter and it prevented delivery of service to our customers. Our internal attempts at intervention were unsuccessful, and we were unable to bring that portion of our cloud infrastructure back online.

By 7 AM EST on that Monday AM, our datacenter team contacted our cloud hardware and software partners EMC and VMWare, which both immediately assigned a Rapid Response Team to address the issue. The next hour was focused on diagnosing if the issue was hardware or software related.

By 8AM EST on that Monday AM, it was determined that the issue was software related with VMWare and that there was no hardware failure and no data loss. The next several hours the Rapid Response Teams worked with the ITelagen datacenter team focusing to restore services for our clients vs. determining the root cause of the issue.
By 12PM EST on Monday, services were restored to approximately 80% of the practices affected with the balance of the 20% being restored by the end of the day. As service was restored to each practice, our system administration team focused on resolving any residual server issues caused by the outage. The final server-related issues were resolved by approximately 6:30 PM EST on Monday.

The Rapid Response Teams then started to focus on the root cause of the issue.

Around 5:30 AM EST on Tuesday, August 15th, the same issue occurred with less severity. This time the Rapid Response Teams and the ITelagen datacenter teams we were able to restore services to 90% of the affected clients by 12PM EST with the balance of the server-related issues resolved by approximately 6:30 PM EST on Tuesday.

During that time on Tuesday the Rapid Response Team from VMWare made some adjustments to the environment which we believe resolved the issue by 9PM EST on Tuesday.

Since that time, through this AM, August 21, 2017, we have been experiencing sporadic and less severe issues that we have been addressing on a case-by-case basis and we have been working with the Rapid Response Team to address them on a case-by-case basis.

What comes next.

As of now, all our services are back online. We now have two tasks ahead of us.

The first task is to complete our RFO (Reason For Outage) analysis to further identify the root causes. We?re working with our vendors to complete this analysis.

Second, we will take the recommendations from this RFO and make any changes necessary to improve our stability and resiliency.

I hope you accept my apology. On behalf of all of us at ITelagen and Amazing Charts, I offer a profound apology. We understand how essential our services are to your practice and the care of your patients. We fully appreciate the breadth of our responsibility. We live and breathe it every day.

I?d appreciate any feedback you can share. Please email info@itelagen.com. This mailbox is monitored by ITelagen?s senior management team.

Sincerely,
John O?Keefe
CEO, ITelagen

Switch to Hyper-V. Problem solved.

JamesNT

Hyper-V?

David Lee

Or switch to KVM, or RedHat Virtualization. Same or better performance, 1/3 the price of Hyper-V or 1/10 the price of VMware. Go with the DOMINANT server OS where uptime is measured in years.

James and I choose different tools, though we are brothers.

Hyper-V comes with Windows server. As far as I know, there's no reduction in licensing costs between hypervisors. Only some cost extra like VMWare or Citrix XenServer.

Indy is quite correct. Whenever you see me and Indy throwing spit balls at each other, just ignore it.

Camaro versus Mustang.

Harley Davidson versus Honda.

One thing we all agree on, and as Sandeep pointed out, there is no cost reduction in hypervisors. Only a potential cost increase. Which is why I still can't believe anyone uses VMWare. The only time I use VMWare is when I need to run legacy stuff such as Windows 2000 Server or Windows NT server or even Windows 95.

JamesNT