Home Forums Discussions General Discussion Why is 8.2.2 slow and how do we fix it?

OK. So here is what I think is going on with 8.2.2. And, the biggest caveat is I am not a SQL expert nor a programming expert.

Reading through the blog article is very helpful. Ignore the 3rd party stuff and just concentrate on how application caching is related to SQL is related to databases. Of course, SQL Server and the databases are one in what they call an "instance." So, instead replace SQL above with the SQL memory cache.

The speed of any AC application should be related to three things. How AC retrieves and writes information to the database. The SQL memory cache itself and the AC cache. The AC development team should control the first and third parts. SQL Server shouldn't be an issue and the queries stored in its memory aren't likely to make much of a difference. Again, IMHO.

So, first let's understand how the data in the databases on the server in the AC folder and AC works. An analogy would be say all of the information in your database was physically stored in your house. You get to your office and you open AC and a chart. Now you need data to enter into the chart. But, it's at home. So, you hop in your car, drive back home and find the data for Joe Smith and bring it back. Now, you open another chart. The data is at home. You drive home and get it. This is akin to opening a chart in AC and its going all the way to the database "hitting the database" on the hard drive on your server. Wouldn't it make more sense to bring all that data with you and load it in AC its cache. This is what it is doing when you see it preloading data for speed. But, you don't need to bring everything from your house, because you may not need it. If you do, you will have to run home and get it. But, now it will be loaded into AC's cache, so it will be faster next time. So, forgetting the analogy, a lot of data is preloaded into AC, but a lot of data is still in the database (or saved in SQL's memory cache -- but that is more complicated for now.) But, again, anything that keeps the query or the call from AC for data from hitting the database makes it faster. So, if it can use data from the SQL cache, then it is much faster. Just like the RAM in your computer is much faster than pulling data from your drive. So, keep in mind, that there is the AC cache, the SQL cache (these are different) and the physical databases on the server.

So, everytime while AC is running, that you ask for information: demographics, meds, summary of problems, etc., AC will look at the cache to see if it is right there. If not it goes to SQL and the database and gets it. That takes longer. But, now that you retrieved those summary problems or demographics which wasn't preloaded, it will be nearly instantaneous. So, do this. Go to a version which is working quickly. Open a chart of a busy patient. The chart loads relatively quickly pulling a lot of data from the preloaded cache. Now, click on Summary tab. Even in a fast version, there should be a small amount of lag. One second maybe. Now click on the demographics tab. A small lag. Now click on the summary tab again. It should be nearly instantaneous. The info is in AC's cache. Demo tab same way.

Now, here is an interesting but probably unnecessary fact. Especially on a slow program. Say you open a chart, and as we have been dealing with, it takes seven seconds. In our analogy, this would be because it needs additional data, and now you have to run home to get it. Or, go to the database. It opens, that data along with the preloaded data is retrieved and cached. Next time should be much quicker. You close it and open it again, and it is much faster. Maybe three seconds. Now click on another chart that hasn't been opened. It will not take seven seconds. (it shouldn't anyway). Why? Because some of the data you pulled from the database is common to the second chart.

OK, so you get the idea. The biggest factor here should be how AC works with the cache. My hypothesis is that somewhere in the development of 8.2.2, the way AC works with the cache both in storing data in and removing or deleting it is not optimal. Having data stored in the cache keeps you from having to grab data over the network and means less overhead for SQL. If fewer staff are using SQL and only the cache, then it will be more efficient.

The hypothesis above is hard coded. Meaning it will need to be fixed in a later version.

So, if this is the case, and AC knows it to be the case (again IF), then they know what has to be worked on. Which means we can all take a sigh of relief.

http://blog.idera.com/sql-server/pe...caching-to-boost-sql-server-performance/

The thing that seems weird to me is that even the long-term, heavy-duty AC champs like Bert and Sandeep don't seem to be able to crack the aft cabin door and get any real answers from the folks who really write the code.
Are they on Mars?
Couldn't they they just give us a little teaser? Security is tighter than the NSA. Where is Edward Snowden when we need him?

For my part, delays of 5-30 seconds every time you change a window or save or open a new file are simply a non-starter. That, plus all the rest of the frustration of "modern" medicine will add up to my changing occupation to garbage transfer station attendant, or something similar.

Electronic medical records will not truly be "meaningfully useful" until it works like Amazon or Google -- any information you want is immediately available across all hospital and clinic databases. My cardiologist friend in Sweden says their record works like that -- nationwide. I don't know if that is really true, and since he doesn't do family medicine, I don't know if primary care is so transparent, but he tells me it is.

Of course, in USA with HIPAA, privacy issues (for the doctors, not the insurance companies), proprietary EMR's and all the rest -- that is a dream that can't be realized unless we get out of our box. I don't know what that would require -- but it won't happen in my practice lifetime, and it reduces the effectiveness of EMR to marginally helpful in a single office, and useless for tracking patients as they travel around.

Presumably, the EMR could also help us switch to an "outcomes based" reimbursement system -- but everything I see along those lines relies on claims data, and is no better than the 1970's paper systems.

When I read articles about EMR, it is either totally geek-speak, or MBA-speak, but never doctor-speak. Doctors don't matter to EPIC or Cerner -- and now I'm beginning to wonder about AC.

The few patients who seem to like these systems only really want one thing -- easy, and especially, FREE access to their doctors and their medical records so they can have a private email conversation at any time about any subject, and get their bronchitis treated without coming in to the office.

Agreed. The downside of having a job that does pay a large figure (even by today's standards) is it is difficult to change jobs. I likely would in a heartbeat.

Bert, I still need to try my speed patch on your setup.

I'm getting the previous speeds on hard drives now.

Any time. I hope I am wrong. I did order 8 more GBs. Using NewEgg Premium now, lol.

Is that 8 on top of the 200 you already have?

ACtually 8 on top of the 208. Upgraded last week. Much faster, just needed a tad more.

A client was kind enough to throw me a copy of their production database this past weekend. Once I get some VM's stood up and do some testing, I'll weigh in.

JamesNT

Are there instructions out there somewhere about how to upgrade SQL Server Express on Amazing Charts to 64 bit version?

Is there any evidence that it actually helps performance?

Cancel my previous question regarding SSD drives, I hadn't done my homework. That was addressed multiple times earlier in the thread.

Maybe someone else can chime in, but I found no significant difference between 64 Bit EXPRESS and the current version. Full 64 Bit SQL Standard is a different story.

Thanks for the feedback, Sandeep.

Yes, I don't think it makes a huge deal with SSDs and databases, but it definitely gives you increased performance all around.

SQL Standard gives you many advantages, but I still don't think it will solve the issues we are having until 8.3. Given, there will always be an upgrade path to 8.3, if you have 8.2.2, you can just sit tight for a bit. Except for those with severe performance issues.

One thing for sure; it is MUCH easier to convert from 2012 Express to 2012 Standard as it is actually designed to allow that. But, it is costly, and then one day AC will move up to 2014. God please don't, lol.

I really wanted to make the upgrade from 7.1.3 to 8.2.2 before the upcoming AC gig in Seattle. I could probably live with the delay problem, I've a fairly fast network and server. However, the return of the problem with new prescriptions being added to the incoming medication list (blending list that patient was on coming into visit with additions during visit) is a deal-breaker for me. Been there and done that with an earlier version of AC, and hated the confusion. (And no, I'm afraid I essentially never finish and sign-off the chart note during the encounter. Perhaps I should, but I'm no where near that efficient, I start running behind by the 3rd or 4th geriatric multiple-problem patient of the morning or afternoon, if not earlier.) (Sorry for the rant.)

I don't blame you. In effect, a lot of the beta testing for AC is done on this board, lol. I would wait for 8.3.

One of the biggest problems is not being able to go back easily with some of these versions.

You may be able to live with the delay. I have a lightning fast network, and I still have delays. They are getting somewhat better with a few tweaks here and there. And, I have yet to deploy it fully. I simply won't until my testing shows normal speed. At least AC development thinks they know most of the issue.

The name escapes me, but Pri-Med does make that Lite version which is problem oriented.

Got invite to beta test 8.2.3 guess I'll bite and report back.

Sorry got disinvited to beta test so for the time being no updates from me.

I know AC must hate me. I wouldn't worry about Beta testing. Most of this is done on the board. How else can you put out 8.2 with the slowness issues. Even if it is 2% to 3%. That is 2% to 3% too many.

Just want to re-mention that I have same slowness in opening and closing charts (several seconds each) and I'm running AC cloud, provided by AC. So I wouldn't suggest wasting time tweaking server and database; if the AC hosting service has this problem, it's presumably inherent in this version of AC.

Michael
NY

How could they hate you, you probably do as much work on issues with AC as a full time employee, without a benefit package.

8.2's speed is a pain, but it really isn't all that bad. It is VERY stable, and it works. Yes, it's slower (all of the V8's and much of the V7's are slower than the V6 series), but not that much (perhaps I have simply gotten used to it)

Thanks Wendell. Hopefully you are right.

Also, do you think that for those who experience slowness, some may be as like yours, but others have a longer lag?

We are currently using ver 8.1.1. We will upgrade to 8.2.3 when available. For us, AC is VERY slow.
I can't imagine that this only affects 2-3% of users. this probably represents those users who have complained!

How slow is slow? If you open a chart for the first time AC is open how long does it take? If you then open ANY chart after that, how long does it take?

I agree on the other assessment.

Using AC cloud 8.2.2

First time opening a chart when program just started: 8 secs.

After that, opening any chart: 4 secs
Closing any chart: 3 secs

Michael
NY

Which would make sense, if it is the cache. Which is good, because they are aware of that issue.

Bert
Between 5-10 seconds to open and 3-7 seconds to close chart at signing. Certain screens generate quickly and others are somewhat laggy.
Marc

The behavior you should see based on my working on it, is that the first chart should be rather slow, and the second and then on until you close it, should be faster. Not sure if you have that. The signing is weird as is the closing.

Each tab will be slow and the following tabs should be faster. Actually, even on a fast version you should experience that, but it would be more like 1.5 seconds first time you do a tab (like Recent Encounter, Demographics, etc.) then almost instant after.

Who knows with ePrescribe. I have seen it related to the amount of meds.

I would check your deleted items and make sure they aren't over 5,000 or so. I would delete them anyway. You may already have that as the default.

We decided to take the plunge last week and go from V6.3.3 to V8.

To summarize briefly: installation went smoothly, with only a couple of hiccups. We have a couple issues with usability, but nothing too serious in very preliminary use. V8.2.2 is definitely slower than 6.3.3, despite better hardware. The difference is noticeable and sometimes annoying, but not disastrous.

Installation was done on our own, without scheduled help from AC support. We did get prompt help from support with a brief issue. The process was smooth and relatively easy, even compared to past upgrades. Installing on the new ?server? on our peer-to-peer network took about 40 minutes. Downloading actually took most of that time. Each client (we have 6) took about 20 minutes. The biggest delay was caused by an unrelated networking issue (resulting from our prior mistake in buying two Win8 machines; an error we won't repeat). For a couple machines I used Revo for a ?clean install? but for most, I just installed on top of the old version and that went fine.

Out of concern due to reports of speed problems, we decided to upgrade the ?server? on our P2P network before the AC upgrade. We got a Dell XPS 8700 running Win7 Pro with 8gb of RAM and an i7 processor. We did not change our clients which are all Dells with Win 7 and 8, with a range of slower processors and memory. Our network is all hard-wired.

There are so many variables and considerations regarding speed that it is difficult to objectively compare one version of AC to another? let alone one practice?s experience to another. I timed an assortment of tasks on our old system with 6.3.3 and the new one with 8.2.2. This was not an ?apples-to-apples? comparison by any means; the new ?server? is faster and higher-powered. I realize that this limits the value of the comparison times, but my goal was not to have a fair test: it was to avoid upgrading and then find that the new version was unbearably slow.

The biggest issue for us is chart opening as well as closing after signing-off. Each can now take 4-8 seconds; several seconds longer than prior to the ugrade, and long enough that you notice that you are sitting and waiting. Opening the letter writer takes several seconds longer than it did in V6.3.3 (this of course is an all new LW with that upgrade). On the positive side, the longest delays are for ?one-time events?. For example, opening the program takes significantly longer than it used to, and logging on is a bit longer. Opening the first chart after logging on is slower than all subsequent ones. Since you may only do these operations once a day, these delays are not so significant. Pulling up messages or a schedule and creating reports is quick. I have not done much e-prescribing but it seems to be only slightly slower. The size of a chart does not seem to make a significant difference in time opening or closing. As in the past, opening the chart to certain tabs (e.g. by hitting F9 to open) is quicker than double-clicking the name and going to the Most recent (actually, current) encounter. This still saves a bit of time.

What I have not done is count how many times we open and close charts in a single day. I am sure I do this well over 100 times a day. 250 might be more accurate (including work on patients seen that day plus prescribing, checking labs, looking at old notes, dealing with messages, etc). 250 times 5 seconds gives a rough estimate of 20 minutes of daily lost productivity related to this issue alone. This is major, and I certainly hope AC makes good on their promise to address it. On the other hand, it is not enough to make me want to roll back the upgrade. Hopefully as we use the program more in coming weeks, the benefits of the upgrade will become more apparent and the slowness less so.

Two other issues bear mentioning if you are moving from V6.3.3 or below to V8. There are plenty of warnings (and instructions, including videos) from AC about migrating insurance, which is part of this upgrade. After upgrading, it first seems that all your old insurance information is intact, though not in the newer codified format. In fact, if you print a patient?s insurance information, nothing appears on the demographic page. I assume the same is true if you try to use AC for billing. Migrating insurance is something that cannot be started until after you upgrade; at that point you have a fair amount of ?figuring out? and grunt work to be done before you are fully functional. This is not work that can be done in a sandbox, and can only be accomplished after you upgrade and as you try to resume normal work and billing.

To some degree the same is true of the new Letter Writer. It is a big improvement on the old one, and it offers far more flexibility and a much better finished product. On the other hand it takes some time to understand its workings and to develop a process for creating a series of templates. This also cannot really be done until after the upgrade.

My goal here was to share my personal experience with the people above who are working on the issue who are far more technically skilled than I.

Thanks for the update. Very detailed. That is good. The only thing I never understand, is it takes me about ten minutes to install server and five minutes to install clients.

Opening the program is a long time with the Crystal Reports and preloading cache. Not preloading inactive patients would help.

I don't think I am going with v8 until it is just as fast as v6.1.2

JBS,
Thanks for the report on upgrading from 6.3.3 to 8.2.2.

The 4-8 second delays probably don't seem that much to some, but I can see how they would be aggravating. For example, I frequent see a wife and husband in same room (2 visits back to back). I try to deal with one patient at a time, but frequently the first patient forgets something after she is done, then I have to close the second patients chart, go back to the first patients chart, address the issue, then close the first patients chart, then open the second patients chart again. 8 seconds is not enough time to make small talk, and too much time to just stare at the patient or the computer.

8 seconds is simply 7 seconds too long.

I have done incremental upgrades. 6.3.3 to 6.5.? to 7.0.?, etc. I have done some beta testing and some were simply moving to the next stage.

Because of this, I have not always remember some of the changes along the way. A good example would be the letter writer. I have been using it for so long that I don't think about the learning curve to how it is different.

Another is the time differences. I have always stated that V8 felt slower. But was it that much slower than 7? It was, but not by a whole lot. I go back to V1 and will tell you that it will blow all other versions out of the water in terms of speed. But it didn't have an Rx writer, orders, MU or any of the myriad of things we now use. It's good to see a detailed description from 6.3.3 (which I would call one the last of the "old" system (i.e SQL 2008)) to the most recent version.

While I have said that V8 is slower, I have also said "Well, it's not really all THAT slow." Actually opening and closing is not all that bad. What's annoying to me is going from current charts to demographics. Glaciers move faster. Opening the program is slower but doesn't happen all that often so not really a big deal. The other issue is Account Information (the last tab) which seems to have always taken longer than reasonable, now it is more unreasonable, but again, not by much.

Some of it I take in stride. I chart while I am in the room, so closing the chart is no big deal, I am still talking when opening and closing the charts. I get the process going or ending and the focus is more on the visit at the time. When I am changing screens to check on either demographics or summary for immunizations, now THATs annoying. If I were charting later, I thing the opening and closing would be more of a nuisance.

I understand that speed increases are in the works. I sincerely hope so, but it's not really a game changer. There are a lot of improvements from certainly V6 to make it worthwhile, perhaps more for a PCP than specialist like Jon, but even true in his case. There truly have been a lot of changes.

Another thing. I know Jon, he is often critical of AC's shortcomings (often with good reason.) He is on our Clinician Advisory Board. While he comments on some timing issues, it would appear the road to 8.2.2 was rather positive. Let's see how it goes in with longer use.

And, to be fair, Wendell is also on the board and he is always VERY positive.
Which is a good thing.

The tabs SHOULD become fast after the first tab has been opened. At least that is my experience and what I have been told by my SQL friend.

But, it is extremely slow, and any new bells and whistles should not make it even a microsecond slower. That's just not how things work. Windows 95 was faster than 3.1. Windows XP was faster. Vista was a hiccup in the road. Then Win 7 Pro blew them all away. I never read anywhere in the past that Win 7 is slower than XP, but it is worth it, because it has more bells and whistles.

Sure, some programs get bulky, say Internet Explorer compared to Firefox. But, once they are loaded into RAM, they are both pretty fast.

I think for many, the improvements are not worth the speed issues. Your view as far as going up incrementally is helpful, but for those of us like me who are still in the stone age (v6.1.2), the speed change is very noticeable.

But, from my interactions here on the board with support and development and via email, I think they are aware it is a caching issue, and hopefully can fix it.

It really isn't hopefully, it is a must. Everything is relative. Rememeber Dial-up? But, things gets faster not slower.

I agree with your basic point that 8 seconds is too long when we need to open and close charts so often. Not to get off track on the "open two charts at once" debate, but remember that while one spouse's chart is open, you can e-prescribe, open and use the Letter Writer, or write orders on the other one.

I have finally finished another project I was working on and got two VM's stood up. Let's see what I find out in the upcoming days.

JamesNT

I certainly agree with Wendell that I am "often critical of AC's shortcomings". In fact, as a member of the Clinical Advisory Board, I view that as one of my responsibilities. I am also not shy about touting AC strengths, either.
As I said, the installation process is much improved. I am also excited that AC is finally turning to addressing some of the old, long-term complaints from users. On the other hand, that should not come at the expense of a much slower EMR. Fortunately, it appears that AC agrees, and is working to address the issue.

Another point regarding speed is that different processes matter to different users. Wendell is not bothered by slow opening and closing because of the way he uses the program. It does seem to bother several people writing above. As a specialist, the completion of every note is followed by (automatically) opening the letter writer to either cc the referring doctor or send a letter. Waiting 8 seconds to sign-off and then more to open the letter writer is hard to overlook.

any updates re chart speed from those beta testing 8.2.3?????

Still on 6.3.3 and no reason to upgrade, until ICD-10 hits. I will likely have to upgrade the hardware for 8.2v.

Suggest you sit tight with 6.3.3 until it is clear the speed problems in V8 are fixed, while upgrading hardware may make 8.2.2 a little better the guru's here seem to think it is a software/programming problem not a hardware problem, there are 'free' icd 9 to icd 10 conversion websites you can use to get icd 10 codes.
Just my 2 cents, from a current speed challenged V8 user.

The short answer is that there is no longer ONE answer for folks looking (or having) upgraded.

There are all of the subjects that those of us who have been around are well aware of - processors, RAM, network, DNS, configuration.

Add to that the increased impact of MU2(the MU Tax) on later versions, certain interfaces (that can consume over a GIG of RAM and significant CPU at every run), and we are trying to diagnose a multi-variable problem.

We have practices with 6-8 computers that notice slowness after upgrading and we have practices with 10-12 computers that have a work stoppage and need immediate help.

There are some cases where 64bit Express makes a difference, and others where it has no effect.

Full SQL has the most marked improvement, but that alone does not deal with the MU2 tax.

For some practices, we are evaluating the option of staying on 6.3.3 from a cost-benefit perspective, and coding encounters completely outside of AC. I'll be doing a more detailed session on it next month, but some practices will be better off having someone other than the Physician/Provider doing the coding, especially with the advent of ICD10.

One simple thing: be sure you have a gigabyte switch. I thought we had one, but time flies when you are buying computer equipment, and we did not. Upgrading the switch helped certain computers in certain ways. Things are still slow though.
Thanks to Bert for thinking of this for our particular situation.