Home Forums Discussions Problems Slow Response Time

Anyone else having issues with slow times? Opening a chart takes about 1-2 minutes!

If you are using spindle drives with AC you're going to hurt. SSDs in a RAID 5 are the way to go. Make certain your AC server has at least 8G of RAM.

Workstations should have SSDs and 4G of RAM.

JamesNT

Server does use spindle drives (15000K RPM). Probably will talk to him about moving them to SSDs.

Other than that, the server is a 40C/80T CPU, 128GB Ram

Workstations are i7s, SSDs and 16GB

Watch that HDD controller. Assuming Dell server you're gonna want an H740 controller or better. Lesser controllers can't keep up with Enterprise SSD.

JamesNT

H700

Got the doc to bite the bullet and slapped some SSDs in the server. Now it is time to wait to see if it actually improves the performance.

You can't help but speed up dramatically with SSDs. Just the reboot times will be far better. Usually don't disagree with James and RAIDs will always be confrontational but I would go with a RAID1 or a RAID10 (my specific go to)

Also what is the speed of your switch? Are you using wireless or wired?

My largest speed upgrade so far has been a Optane 905p SSD for AC VM. Really fast random, queue depth 1 access (great for SQL hence AC). $340-399 for 1 Tb while they last.

It's not in a raid and running in a PCIe U.2 adapter card so definitely not official "enterprise". But, I keep good daily AC backups and have a hot-spare server.

Switched from Raid 1 and Raid 5 to a Raid 0 (single drive for OS since offsite backup) and Raid 10 for data.

Also the switch is just a managed gig switch, so the network teaming we are doing is just more for redundancy in case something fails and can load balance.

RAID0 is still two drives. It just gives slightly better performance than a RAID1 but RAID0 has no redundancy. It's nice to know that if one drive fails you still have the other drive to run on until you can replace the faulty drive.

RAID is one of the few areas were myself and Bert disagree. A few points and my argument:

1. Never use RAID 0. If one drive fails you loose the entire array and all data on it.

2. RAID 1 is great for the operating system drive. But little else.

I prefer RAID 5 or 6 and highly recommend it for AC installs. In every environment you will ever manage, that environment will fall into one or two categories: Heavy Reading | Heavy Writing. Amazing Charts is a Heavy Reader. With RAID 5/6 for every drive you add it cuts down read time. That's why my RAID 5 arrays always have a minimum of 4 drives. And, with RAID 5 you can lose one drive and be OK with RAID 6 it is two drives.

The reason I do not recommend RAID 10 is because of sheer cost. That cost comes in two forms: first is the cost of that many more drives. Second, that many more drives also take up more drive bays in your server so limit expansion ability. Remember, for RAID 10, *half* of the drives are for nothing more than redundancy.

JamesNT

I use the RAID 0 on the OS partition (single drive), but that is because we have a nightly full disk backup (for bare metal restore) and every 2 hours we take a backup of the SQL databases and the Files.

As for the RAID 10, I dont mind the cost because it isnt the biggest practice, so we have 4 500GB drives in it. This allows for 1TB of data storage (they are only using about 200GB) and still allows for redundancy, but still gives you benefits of read AND write speeds.

Either way, I feel like Amazing Charts needs a complete database overhaul because no matter what client of mine it is (I manage 4 that use amazing charts), it is always super slow when doing just about anything lol. One of them runs OK, but that is because they only have about 3K active patients, where the slower ones have closer to 16K active patients.

AC slowness is not because of SQL Server or the database. It is because of AC itself.

JamesNT

Aint that the truth. Even after all these upgrades (Server OS, SSD drives, RAID update) everything is still painfully slow. Even opening a most recent encounter takes about 1 to 1 1/2 minutes just to load up to the point you can click on anything. My doc already looking to trial other software.

If you need your AC data exported to PDF files (one per patient) including imported items, let me know.

JamesNT

He is now looking at EMDs, which we also have several clinics using that. Since we can hand them over a copy of the database, they can usually import things easily with mapping. But we shall see. I talked to AC support today and they are recommending an uninstall and reinstall on the server, but I feel like that is going to result in the same thing happening.

You can try turning off Hardware Acceleration. That has worked in the past.

Also, I may have missed it, but are you using wireless or wired?

I would take a patch cable (Ethernet) and plug a PC directly to the server and see what your AC does. This is the only way to completely bypass the wired/wireless network. But no one does it, so don't feel bad if you don't. But you have to troubleshoot. And that's the first step.

Pulling a chart should not take 1 to 1.5 minutes! I just opened a chart and it was less than 10 seconds. New Note was almost instantaneous. My problems with slowness come with opening imported and prescribing windows. What kind of times are you getting for those?

In troubleshooting it is so important to get back to those helping with the info.

1. Hardware acceleration
2. Are you wired or wireless (key)
3. Plug one computer to the server and see how fast it opens up.

If you had a fluke meter that would be great, but they are rather costly.

Plenty of IT expertise on this thread that way exceeds mine...
but if it takes 1.5 min to open a chart, I would say there is something unique to/ wrong with your current set up.

Sure, AC may have its issues, but what you describe is more than 2 standard deviations outside the norm.

Not sure if your client has given up. We used to have issues with eRx taking an excessive amount of time but everyone had that. I have NEVER heard of anyone having this much time with opening a chart. I doubt it is the server. I doubt it is AC. You have good SSDs and plenty of RAM on the PCs.

If we could know:

Speed of switch
Wired or wireless
Speed when connected by Ethernet cable directly to server. Just get a 50 to 100 foot Cat6 cable.
SQL is blazing fast when compared with AC so it won't be that.

Remember, no one else I have ever heard of has had an issue like this. I could see changing EMRs if it were this slow. I can almost guarantee you it is not AC. You are simply connecting to SQL.

It could very well be your network.

Agree. Check network, do performance testing, isolate each component. If not solvable ask a qualified server/networking IT consultant.

Sorry for late reply, went on vacation lol.

Switch is full gigabit, nothing is wireless, we have network teaming enabled in the server, amazing charts also runs equally horrible on the server.

I have been monitoring resources and the quad CPUs are only hitting about 3% usage, about 28GB of the 128 GB of ram is being used, network is only showing spikes when someone scans to Amazing Charts (for half a second) and disk usage is low.

We just had amazing charts do a full uninstall and reinstall of amazing charts and it's still slow. Their next troubleshooting step is to wipe out the server completely.

H700 controller means you likely have a Dell R710 controller.

I'm sorry, but that server isn't going to cut it. The H700 controller was out long before SSD drives were mainstream. That controller simply cannot handle the throughput.

I just sent you a direct message.

JamesNT

DON'T clear the server. That will do nothing except cause 24 to 48 hours of headaches.

Are you saying that when you run AC on the server it takes up 28GBs of RAM. That is crazy. It shouldn't take more than 150MBs of RAM

I would let James remote in and use SSMS and see what the time of queries are. Should be milliseconds. But if way more there is a problem. Do you have the OS working as a host so you can run a VM and isolate AC there?

Also, has AC looked at it? Are you on the latest version? If James can't look at it, my IT guy would probably look around for free. Why is it using 25GBs of RAM.

You don't even need AC on the server if it is running?

Tonight I had not only created a virtual machine on my server (which I host other VMs, also turned them off to test) and it was terrible. I even put it on my own machine (3950X AMD, 32GB RAM, 3090TI, PCIe 4.0 M.2 SSD at 3800Mbps) and it was still terrible. I dont know if it is the amount of patients or the size of his database (16K patients and 11GB Database), but even on my workstation being a standalone could not compete with it. I think for the size of his practice and the latest updates are just not working at all and it is time for him to switch.

My thought is, if I can play Cyberpunk at 4K no issues, encode a 4K 1 hour video in 15 minutes and my network is a 10GBPS network, with the amount of high speed cores I have and DDR5 RAM and amazing charts is still crawling, there has to be something wrong with amazing charts itself.

SQL Server Express has a 10G limit on database size that you have clearly exceeded and is very likely your problem. You will need to move to SQL Server Standard for further resolution.

JamesNT

Wow!

Perusing this thread because of problems and feel I should share a little

Using cloud hosted AC, recent upgrate from AC to 11.4.2, 35 second wait times to open any new aspect of the chart -- losing 2 hours of staff time a day for the last 8 days and spending lots of pajama time with AC. High speed conections (851 mbps this am) and OK computers on our end - the same from every home environment and our main office. Unusable. Newcrop "improvements a bandaid that doesn't adress lots of glitches and obvious opportunities for improvment. Swithching to Athena with a go live date of 11/1 so this is only purgatory?

Our practice is 15 years old and we work in a resort community, so 1 have 16,000 charts but really only have a core of 2000 locals. Any thoughts on if AC could run a script to inactive pts that havent been in in say the last 5 years -- if so would it help?

We are using the Sql Server Standard 2016

Yea, ever since this new update, it is painfully slow and my doc I manage also has about the same patient load in his database.

Fascinating, Teleski are you using official AC in the could? What did they have to say about it? Also tell me about your job, sounds great to us flatland skiers.

Paging AC reps this could help you troubleshoot potentially serious issues with 11.4.2

I'm running 11.3.2 on a local server without hiccups for any interested. Too timid for 11.4.2 without a good reason to upgrade. The solid lesson is to be a late adopter.

Yes, the only reason my clients upgraded to 11.4 was due to the lackluster performance they were receiving, but when it came down to it, upgrading to 11.4 was the worst mistake.

Now that 11.4.2 is out and we have applied that, it has gotten even worse than it was on 11.3.2

I found that sometimes if you have too many (deleted) messages in the inbox it wll slow the system down substantially. I contacted support and they logged in and cleaned out all the deletes over a year. It made a significant difference.
You can go to user preferences and under messages click automatically clear deleted messages older than X years and I have that checked now but not sure if that will clear it as effectively as they did remotely

You can also go directly to deleted items, highlight ANY of the messages and select Delete. A popup box will appear and the top bar will allow you to delete all the messages. The bottom bar will allow you to delete all items "Received before.....??

I recently did the delete "received before". I didn't think this feature worked but then I cut the date down so I was removing smaller bits at a time and it worked perfectly. I had messages in the deleted box over 2 years old. Not sure it has sped up the system but worth trying. I also was able to talk to Tech Support and he logged in to clear the dreaded pharmacy change request in the refill pop up. I had over 1200 requests for medication changes. I was not able to do any thing but deny them. This is a feature that is not working correctly and tech support was able to clear and turn it off.

How to do this yourself:

First delete them all: See above

Second: go to Edit --> My Preferences --> My messages (you can customize it to some or all

If you haven't gone to Edit --> and looked at all your options, you are missing a lot of options.

Keep in mind that if you have it check for interactive medications or allergies each time, your ePrescriber will take much longer to open. We just send the drugs and if they have interactions, we find out that way :-)

@Chris

No one is going to listen to you. You have been officially jinxed by the ePrescribe gremelins, thererore, everything happens to you, lol

Doc still suffering from issues but refuses to leave Amazing Charts. Done basically everything I possibly can, including moving it to a standalone gaming rig and still insufferable to use. I think there may be something in his database that is causing the issues, but Amazing Charts support doesnt want to take the time to fix it.

Rock...meet hard place.

Stop blaming AC. Your setup is just wrong most likely. Let a SQL expert come in and measure your query times. It doesn't matter if you put the system on the Pentagon's servers. How much RAM are dedicating to the SQL server? Plus RAID 10 is the way to go. Or at least RAID 10 or RAID 6. Like James said, RAID 10 not used only due to cost, which I doubt is your issue.

We have two SQL experts. JamesNT and the guy I talked about in the private messages. Nothing against James, he knows his stuff. Just giving you another option. Like James, has 20 years of experience working for major companies. You need to have them jump in use SSMS to pull a chart and see if it is SQL or AC. On mine even with Express, the queries are like 2 ms and AC is about 5 seconds. Again, getting back to the amount of RAM on SQL. It needs more than 1 GB of RAM. Probably at least 5 to 10 GB to handle the pages. You are asking AC to bring over a lot of data for speed which, of course, does just the opposite.

Please don't take this wrong, but if this is a problem with your SQL-AC setup, then AC can't fix it. They have probably never seen this problem because it has never happened before. No SSDs, that is a problem. But, still 2 minutes to open a chart? Crazy.

This is my final piece of advice. Not sure what James charges. My friend, just to give a second opinion, is very reasonable. It is your setup, pure and simple. You could have the fastest CPU and hard drives and RAID 0 for speed (no redundancy) and it isn't going to fix the problem. How much RAM is dedicated to Standard?

Someone with SQl server experience could delete your log files and get your space down to 6 GBs. Hire James. Hire my friend (sorry James -- just giving him options based on time factors). This is obviously too difficult for AC -- I don't blame them at all -- and not solved by you yet. Spend the money and have a SQL
EXPERT look at your query times when you pull a chart. I am taking you installed SQL Server Standard yourself. It is extraordinary fast, no limit that know of on RAM or database (I could be wrong), but if someone who is SQL only didn't set it all up, then there is probably an issue there.

We are using SQL 2016 Enterprise and the SQL can use as much RAM as it wants (I dont have a cap on it), but usually it is using about 28GB of RAM.

Damn. That is a lot of RAM. ITFACTOR, obviously I have been following this thread (didn't mean to be so harsh -- just AC probably will have a hard time figuring this out).

It sounds like you know what you are doing. BUT, would you be opposed to having a verified Microsoft Certified SQL Server specialist remote in and look around and see if there are any issues you may be overlooking?

Yea, just ran a test today and the SQL service gets to about 28GB. But I also can assume that he has outgrown amazing charts. He has about 16K active patients, sees about 50-70 patients a day and even his imported items folder (scans) is about 250GB in size. But he has been using AC for about 12 years?

60 to 70 patients per day? It may not be too big. Still think you should take the free offer for my friend who is a SQL expert look around and make sure everythig is set properly. Not sure why you won't do this? Or JamesNT.

See private message. I am really trying to help here.

Actually over this weekend I performed a reformat of the server, reinstalled amazing charts and restored the data and now it seems to be running smoother. Doc is going to test it today but hopefully that fixed it.

That's good news. Hope it fixes it. Pulling a chart really shouldn't take more than 5 to 8 seconds. Especially if you are seeing 70 patients.

There are two mistakes I see.

First, the purchase of SQL Server Enterprise was unnecessary. That's a near $20,000 purchase by itself. Assuming single provider practice, how you got the doctor to spring that kind of coin is beyond me. My clients would have skinned me alive.

Second, the H700 controller. That tells me you are running all this on a Dell R710. You are attempting to run modern software on 2008 model hardware. The H700 simple can't handle the throughput of those SSDs and is choking. You would be considerably better off to get a Dell R740 server with the H750 controller.

You sunk all your client's money into software when it should have been hardware.

JamesNT

He is running a Dell R820 and the SQL is something he got previously before I took over.

But I have tried convincing him to get something newer but he doesn't want to spend any money on it. Much the same to go with something like Athena or some other cloud based EMR as they are too expensive. So basically I'm stuck doing what I can with what I can.

I would request the client show you the purchase of SQL Server Enterprise. SQL Server Enterprise is one of the most expensive things Microsoft sells and they are quite serious with the licensing of that software. Microsoft audits, which I have been involved with 14 of in the past 8 years, go one of two ways:

1. You are missing some CALs, need to get a copy of Windows Standard for the extra VM you are running. No worries, just show us proof of purchase in two weeks and we are good!

2. You have over $20,000 of our stuff in production that you never paid for. We are sending over one of our lawyers that makes per hour what you make per week. May Bill Gates have mercy on your soul.

If the client is pirating that kind of software, you may wish to walk away or at least insist that the client purchase the correct software. SQL Standard is the most you will ever need.

That Dell R820 is still 2012 technology and the H700 is still your bottleneck. All servers I'm selling now are R740 or higher with H740 controllers or higher. But if you are stuck with it, you are stuck with it.

James

If the doc is waiting 90-120 seconds to open every chart, HE (or SHE) is the one "stuck with it".

Hard to imagine anyone willing to live with a set up like that. If they think they are "saving money", they are deluding themselves.

James, how does the Dell T430 with an E5-2620 processor at 2.10 GHz 8 cores 16 logical processors with 96GB of RAM compare with the H740?

@ITFACTOR My recommendation has always been to allow my friend Raja or JamesNT (both SQL experts) assess your setup and possibly remote in to check the configuration of you SQL server. Both are doing it for free right now. You're getting a lot of free advice.

If I were you, I would simply do what James is recommending.

Agree with Jon.

And this is why this board is so valuable.

Bert,

I'm afraid you are not making an apples-to-apples comparison. The H740 is a hard drive controller card, only, not an entire computer.

If you gave me the specs on RAM, CPU, etc. with an H700 controller and then compared those same specs on RAM, CPU, etc. with an H740 controller, you would see a world of difference.

James

I was giving an apples to pineapples comparison. Sorry. Thanks for the info.

I think you're onto something. It's likely something inside AC.

Most server processors and going to be slower than your test bench. If it's slow for one user, it's going to be slower when you add the rest. New hardware won't make a difference.

Upgrading from SQL Express to Standard/Enterprise doesn't really make a massive performance difference given how AC is programmed. Most people end up getting because they exceeded the 10GB database limit rather than performance.

I would dig around and see where it locks and slows down. That will give you a hint of what needs to be cleaned up.

The T430 comes with a E5-2620V4 which is pretty old now. It came out in 2016. If you have a server with that processor in it, the warranty is probably over or is going to end very soon.

Just a heads up

They are covering it at least another 3 years.

Nice if AC reps would chime in:

What processors/storage/architecture is AC cloud running?

Does AC play well with AMD cpus?

Is AC essentially single threaded? Then 2012 20-core slowish processor will choke on multiple users.

Is there an upper limit to self-hosted database size/user #? If so how does AC cloud circumvent the issues?

We could ask Jon Bertman to help interface with AC? Come up with our top 10 questions / requests for AC

I am confused. How is it an Amazing Charts issue when no one else has the problem?