A message from our hosting provider ITelagen:
To our valued Amazing Charts practices,
I am John O?Keefe, the CEO and founder or ITelagen Healthcare. We provide HIPAA secure hosting to over 3,000 providers nationwide including all of Amazing Charts? hosted clients. As you know, Amazing Charts & ITelagen experienced a significant service outage this week. As CEO of ITelagen, I want to offer a profound apology.
I also want to explain what happened, what we?ve learned, and what we?re going to change. First, you should know that our services are now stable. There has been no data loss and there were no security breaches.
Here?s what we know.
Around 5:30 AM EST on Monday, August 14th, we experienced a significant issue in our VMWare cloud platform that affected 25% of our cloud infrastructure. This affected a significant number of servers in our Nevada datacenter and it prevented delivery of service to our customers. Our internal attempts at intervention were unsuccessful, and we were unable to bring that portion of our cloud infrastructure back online.
By 7 AM EST on that Monday AM, our datacenter team contacted our cloud hardware and software partners EMC and VMWare, which both immediately assigned a Rapid Response Team to address the issue. The next hour was focused on diagnosing if the issue was hardware or software related.
By 8AM EST on that Monday AM, it was determined that the issue was software related with VMWare and that there was no hardware failure and no data loss. The next several hours the Rapid Response Teams worked with the ITelagen datacenter team focusing to restore services for our clients vs. determining the root cause of the issue.
By 12PM EST on Monday, services were restored to approximately 80% of the practices affected with the balance of the 20% being restored by the end of the day. As service was restored to each practice, our system administration team focused on resolving any residual server issues caused by the outage. The final server-related issues were resolved by approximately 6:30 PM EST on Monday.
The Rapid Response Teams then started to focus on the root cause of the issue.
Around 5:30 AM EST on Tuesday, August 15th, the same issue occurred with less severity. This time the Rapid Response Teams and the ITelagen datacenter teams we were able to restore services to 90% of the affected clients by 12PM EST with the balance of the server-related issues resolved by approximately 6:30 PM EST on Tuesday.
During that time on Tuesday the Rapid Response Team from VMWare made some adjustments to the environment which we believe resolved the issue by 9PM EST on Tuesday.
Since that time, through this AM, August 21, 2017, we have been experiencing sporadic and less severe issues that we have been addressing on a case-by-case basis and we have been working with the Rapid Response Team to address them on a case-by-case basis.
What comes next.
As of now, all our services are back online. We now have two tasks ahead of us.
The first task is to complete our RFO (Reason For Outage) analysis to further identify the root causes. We?re working with our vendors to complete this analysis.
Second, we will take the recommendations from this RFO and make any changes necessary to improve our stability and resiliency.
I hope you accept my apology. On behalf of all of us at ITelagen and Amazing Charts, I offer a profound apology. We understand how essential our services are to your practice and the care of your patients. We fully appreciate the breadth of our responsibility. We live and breathe it every day.
I?d appreciate any feedback you can share. Please email info@itelagen.com. This mailbox is monitored by ITelagen?s senior management team.
Sincerely,
John O?Keefe
CEO, ITelagen
