Home Forums Discussions Problems Imported items speed

The speed of opening imported items is becoming ridiculous in large charts. Here are some questions:

In most competent database programs, searches are done on the server. Only the results are sent over the network, making the speed of the network itself (wired or wireless) pretty much immaterial. What is happening on AC?

We have AC certified IT support maintaining our system. What we have is the best they can do. We don't scrimp on hardware. The access to imported items is not just slow; it is two or three magnitudes slower than I would expect.

Is there some essential fix? not just wired vs. wireless which seems to make little difference? that makes this useable??

Wow, that was profound

He was copying and pasting from the imported files section...I actually feel the speed issue at times on that as well....found out my computer cant listen to I heart radio, do 2 browsers, AC, and UPDOX along with a stock ticker....so I got more RAM...

Because AC Imported Items is accessing external files, it is *very* sensitive to disk I/O performance.

That is why we have recommended and implemented SSD RAIDs for the AC directory.

When you have a complex patient with ~250 Imported Items, the time to build that screen is a few seconds versus over a minute.

I'll try again. I got disgusted with the post not sticking when I started this and just went to bed.

We have done all the hardware and network band-aids we have thought of. At best, it is painfully slow, and getting worse by the month. Eventually it will just be unusable as the practice grows. Any long term fix thoughts?

I am absolutely serious about SSD RAIDs as the solution.

PM me if you want to try a system with SSD RAIDs installed.

I second Indy's remark about SSD drives. I installed one in my workstation here at home and am able to run the following with no problems:

* Windows XP VMWare virtual machine with Visual Studio 6.0.

* Windows 7 VMWare virtual machine with Visual Studio 2010.

* Windows 8 host with the following:

* Visual Studio 2013
* Outlook 2013
* 24 copies of Internet Explorer (tabs count as copies)
* Word 2013
* Four Adobe Readers

And the list goes on. All with no problems.

However, in addition to the SSD's, do take the time to look over your network. Make certain there are no "weak links." This way you'll get full benefit of your SSD's. Your entire network should be Gigabit Ethernet. Any 10/100 network adapters should be replaced.

JamesNT

I still think you should be slightly hesitant about SSDs. I have them on eight computers. I had to replace three and still have freezing issues on one. I still don't see using them on servers.

Bert,

Apologies, I'm afraid I'm going to have to disagree with you.

Firstly, even if SSD drives were 10% less reliable than traditional drives (that is quite a lot and the actual number should be far less) that does not affect the overall statistical probably of failure over a 5 year lifespan between an SSD drive and traditional drive all that much. In fact, if you surveyed 100 computers with SSD drives versus 100 computers with traditional drives over a 5 year period, you may notice one more drive replacement of the SSD's versus the traditionals.

Secondly, backup, backup, backup. The cost/performance ratio of SSD's, as it continues to come into better favor, means the risk is worth it. Even at 10%.

Take a look at this article from Tom's Hardware:

http://www.tomshardware.com/reviews/ssd-reliability-failure-rate,2923.html

JamesNT

I'm not talking 5 years. I'm not talking MTTF. I am talking one day. You can quote stats and books all you want, but when you install 8 SSDs in brand new Lenovo 91p and three of them freeze constantly, that's a pretty high percentage. Of course OCZ vs Intel is a big difference.

I've had good luck with 3 machines with SSDs for the last 18 months. I did have a standard hard drive fail in the same time. All of them were implemented about the same time.

They (SSDs) are much faster.

We haven't had a SSD failure yet, but that is also why I always propose RAID1 - additional redundancy.

TO abstract the whole issue of drivers and ease the recovery from a hardware failure, our baseline is a RedHat /CentOS Host machine, a virtual instance of the AC server on the RAID1 SSDs, and snap-shot the entire server container onto the host spindle drive (and we recommend to a NAS/other computer on the network).

Absolute worse-case, the host fails hard, you install the host OS on another machine, copy over the backup of the container, and you are back in business, faster than you could install a new Windows OS and patch it.

High-end infrastructure companies now offer all SSD hosting Performance Servers and once we move to the new version of RedHat Enterprise Virtualization later this year we will offer all SSD instances as well as clustered real-time hardware failover.

If you "Feel the Need for Speed", you will be able to have it soon.

I didn't understand a word of that lol.

I think we are having the wrong debate here. I have had ONE hard drive fail in my computer life.

SSDs are extremely fast. But, in my experience (my experience), I don't trust them yet, certainly not to use them on a server. And, since my servers tend to last five years, I will likely have four years of frustration. I never said my SSDs, and I have had 12 crashed, I just know that when I replaced eight spindle HDDs with eight SDDs, immediately three of them would freeze for two minutes or have to be rebooted. Maybe the install was wrong with UEFI. Maybe my six month old Lenovo computers weren't compatible with them. I don't know. The Porsche 911 could be rated the fastest and most reliable car in the world, but if you have a bad experience with just one, you may not say you won't ever buy one again, but you will be a little leery. If I purchased 10 desktops, would I put regular drives in them? No way. I would use SSDs. I (me) am not sure if they are ready for prime time yet. Understand, I am not saying they aren't and that Google isn't using SSDs for every server they have which is at least more than three, I am just not there yet.

my 2 cents. I use a SSD on my work computer. However, I do regular backups (nightly) to a regular 7200 rpm hard drive. It is so nice to use that I will always use SSDs on my personal computers (or whatever comes next).

I have worked with 2 other SSDs that froze or wouldn't work consistantly. In general they are blazing fast but quite a hassle when customers have problems. I currently have relationships ( :-) ) with about 8 machines with SSDs.

Yeah backing up to an SSD would be crazy. ? Very ironic that jasman has eight machines and two SSDs froze.

Don't get me wrong. The work. And, the are 6Gbs. You can't beat the speed. Reboots in less than 15 seconds. Which is good since I have to reboot more often. If you get SSDs, which I STRONGLY encourage, I would recommend Intel.

Thanks for the post jasman.

sorry. the 8 machines I worked with that have SSDs 2 had freezing issues. One was a client's and the other was on one of my machines.

No need to be sorry. I understood.

Looks like I will need to make an SSD forum, lol.

OK, so let me see what you think could be gained by using an SSD. 10 times faster? 100 times? Unless we are talking a couple of orders of magnitude improvement, I just see this as rearranging the deck chairs on the Titanic.

To give us an idea of where you are starting, what is the size of your II folder, at what number of entries in a given patient are you seeing significant slowness? 10 items? 50 items?

To answer your questions I'm willing to replicate your install in our environment with 15K spindles, then again on SSD just so that we have actual benchmarks.

How does SSD fail?
Catastrophically?
Just slow down?
Can you tell it is going bad in time to do something about it?
I have had a Crucial brand in my server for almost a year. Works like a charm.
Planning to upgrade to bigger drive, Samsung Pro series -- any thoughts?

Samsung Pro is what we use - haven't had *any* fail - maybe the luck of the Irish, but I'll take it.

Intel if you own stock in them - have engineer friends there, just don't like paying the brand premium.

Was working on a server that reported degraded from the RAID controller during boot (still ran fine) - backed the instance to the installed spindle drive on the host, rebuild the array => Good-To-Go.

In general, you would have deck chairs rearranged for significantly faster transfer of guests. In other words, if you played a game of musical deck chairs, the game would be over much more quickly and quieter. But, we would have to know what HDD you have now and the SATA ports on your motherboard, e.g. a 6Gbs SSD connected to a 1.5 Gbs port will not perform at its capacity.

Tom, SSDs of every brand come with software such a s Intel Control Center that not only allow you to monitor your drive, but like S.M.A.R.T. will alert you when there are issues.

ARRGHHH! I went to do some tests. We have an imported items file of just over 19 GB. I opened the II link to a patient with about 100 labs that last week had taken me well over a minute. Today it took 12 seconds. Of course, today is Sunday, and no one else is trying to do anything.

Maybe last week was a bad week. Maybe I forgot to do my periodic restart that seems necessary. Maybe it is network congestion. Maybe it is just doing the IPIO (Innate Perversity of Inanimate Objects) routine to suck me ever deeper into the illusion that this has all been a wonderful adventure and I am actually very happy with my medical information system. Whatever. I can't ask for help in fixing something that this afternoon ain't broke. Thank you for the generous offer.

David,
Our II file is 43 GB. When I open a chart with about 100 labs (and we don't have too many like that; by the way, I am counting the labs brought in on the interface from a single date as one item, even if it is multiple lines of labs)... it takes a second or two. This is not dramatically different from a new, empty chart.
So the issue is not the size of your II file, nor is it the number of items in a given chart.

It is hard to compare apples to apples because we use a peer-to-peer set-up, with a mix of Win7 and XP machines, with no SSD's. I believe our network is far smaller than yours (we are only two providers).

I have never been able to understand why some have this issue and others do not.

Wow. That would be so great. Last week it took, as I said, in the neighborhood of a minute to open Imported items on a patient. Then maybe 20 seconds to open a particular lab. Then another 45 seconds or so while the imported items reloaded before I could select at another lab. And that was not my benchmark patient with the huge number of INR's and other labs; just someone with hepatitis C under treatment getting normal labs. You can imagine how frustrating this is. I have to write down the first set of lab values manually before trying for the second as I cannot be certain to remember all the values in between data loads.

David,

There are a couple of things that come to mind. The database itself, although I find that less likely. But, it would be easy to have someone look at the tables, etc.

But, the major thing different between Jon's and your setup is the network. My suggestion to rule out EVERYTHING in your network would be to take a laptop or PC and connect it directly to the computer with AC. You may need a crossover cable rather than a patch cable, (you used to always have to), but today's cables can allow you to use a plan Ethernet patch cable. Or, if you have a small switch lying around you can put it in the middle and take care of that problem if it arises.

If you get two to three second speed each time, you'll have your answer. If it is 20 seconds to a minute, you won't. Well you will know it's not the network.

David, we have talked offline about your network before, and I came away from the conversation with the impression that it was getting the switches upgraded and would be reliable and reasonably fast.

That said, my experience is group practices above 20-30 connections to the database, AC (in part because of the SQL Express limits) starts to get memory then I/O bound.

One partial fix is running a full version of SQL Server, the other would be faster disk, ala SSD RAID.

We regularly onboard practices with 10-20G of II, we are doing one now with 80G; I only asked about size to know what it would take to fully restore your DB.

Try the same patient tomorrow a few times as more people get logged in and see what results you get.

Which is why I suggested connecting directly to the server.

So far it is working quickly this morning, but not everyone is connected yet. But that brings up another question; why should network speed be of much importance? In my other SQL databases, all of the heavy lifting takes place completely on there server. All that goes over the network is the search request, then the reply which is basically a screen redraw of a few KB. And Bert, can't I mimic this by going to the server itself which has a copy of AC on it and running it totally on the server?

OK, here is what I have:

If I use Remote Desktop, and run AC client on the server, it is very fast. My benchmark patient opens a lab in 4 seconds, and about the same to close a lab and reload all imported items.

Over the wired network, it is 16 seconds
Over Wireless, about 18 seconds.

The 16 vs. 18 second difference is not enough to make me abandon the convenience of wireless and consider adding the cost of wiring all the exam rooms, but it seems the whole network is really slow by comparison with running the program on the server itself. I would consider wiring the rooms if the wired network We have all the usual; gigabit managed switch, cat 6 cable, etc. So, I think I need to get my IT guys to try and figure out what our true network speed really is.

That still leaves open the question: what is the program doing if it is dragging all that information back and forth over the network? It is tempting to use Remote Desktop and just run it on the server.

No, you have to have a mini network. The server will always be very fast.

I would love to get a traffic sniffer on his network to see what's going on.

JamesNT

Or my friend's $25,000 fluke meter. But start with the tiny little network.

Dave,

You would have to get a TS and licenses for everyone to use RDC. You can't run multiple staff off the server.

Go to Amazing Utilities, Advanced Options, Test speed to and from databases just for fun.

Also, make sure you NIC card in devices is set to auto-negotiation. Also, rather than test using RDC which isn't helpful at all, connect one computer to the server through a small switch and see what the speeds are. If it's fast like two seconds, put it back where it was. If it is slow, you have the answer.

Just by connecting to the server directly and it is a lot faster, you just made your switch a possible candidate. You could then use the 4 port GB switch that you know is good and run the server cable and three computers through it and see what you get. Generally wired would be faster than wireless, so that should tell you there is a bottleneck somewhere. I would definitely look at the switch.

The Amazing Utilities test runs between 0.6 and 1.4 seconds. This is mid-day with average performance. Is this fast or slow?
The traffic sniffer would be great. Don't know anyone who has one. Physically what you suggest, Bert, is possible, but I just have a reluctance to start yanking cables around in the dungeon that is our shared server room. Is there any alternate way to do this, like timing the transfer of a large file from the server to a client? Having put an awful lot of effort and $$$ into our hardware and network, not having issues with other database programs sharing the network and not having scrimped anywhere, my emotional reaction is to point to AC, not the network, but clearly I am capable of being a victim of confirmation bias.

My experiences with large group networks is that it is on the slow side. You should see ~.6-.8

On well-implemented networks for AC, I have seen .3-.5

You are correct about AC's design - it is old-school client/server in it's implementation, which means that it moves a crap-ton of data back and forth.

If you are getting far better performance from a TS session, then you may want to think about M$ or third-party terminal server as one option to price out.

Indy, thanks so much for the information. If the issue really is that there is just a ton of stuff going back and forth, then the suggestions given make sense. I had been assuming that AC's design was more what I was accustomed to, in which case network speed would not really enter in greatly, and I just could not understand why it kept coming back to that. Knowing what you have said, then the issue makes more sense, and putting in the effort that you, Bert and others have suggested is clearly worthwhile.

Don't take this the wrong way, but most people tend to put the blame on the software and SQL. Everyone uses the same AC (well different versions) and SQL, but all use different networks. There is a saying in IT which basically says, if you are having trouble connecting, it is likely DNS, DNS, DNS. And, the same with slow connectivity -- network, network, network.

As far as cables, etc. that really is rather easy. If you unplug the Ethernet cable from the server, you know whether cable end is. Wrap some tape around it. If you have a laptop, you can use it. If not, just disconnect the cables from a computer, move it closer to the server and connect both to a 1GB Netgear switch. You can get one on Newegg for about $40 to $60. Then run your speed test and do some things with AC. At home I have a 100 foot Cat6 Ethernet cable. You can always run a long cable from a computer to the switch.

Not to be redundant, but if that is fast, then move the new switch to where the real switch is and connect the server and two computers to the switch. You can connect the router if you want. If that is fast (basically the same thing as the experiment from before), then you have to wonder if it is the old switch.

Lastly, and we mentioned this before is the Link speed. If you go to the device manager and find your NIC card, right click, select properties, Advanced, you should see Link Speed & Duplex in the drop down list. You will see everything from 10 Mbs half duplex to 10 Mbs full deplex up to 1000 Mbs half and full duplex. Obviously, 1 Gbs full duplex is as fast as you can get, it is still best to let the card set the speed automatically as it will set it for the fastest speed. You would see a rather large difference between 10 Mbs Half Duplex and 1000 Gbs. Now I doubt this is the problem, because I doubt every computer has their NIC cards set to the slowest setting. But, who knows, maybe at one time, your cables handled 100 Mbs or your switch did, so someone thought they needed to match.




By the way, my speed test is consistently 0.25, and mine runs fine. But one man's fast is another man's slow. But, I can open II in less than a second.

This is the frustrating thing about AC and SQL. Obviously, AC has to use the free version of SQL Express 2005 and 2012. They can't expect for all their users to keep paying for the full or standard versions of SQL. So Express is bundled in the install. The problem is that comes with a big hit. While 2012 gives you 10 GB of space, quite a bit more than 2005, they are all crippled with the same 1 GB of memory. Standard versions come with a ton of RAM or unlimited RAM. So, when you put 40 GBs of RAM in your server, you are wasting quite a bit of RAM. SQL will take as much RAM as you give it, so in this case, you would set it to 24 or so. Maybe even 30 GBs. Now you have enough RAM to store 24 times the amount of pages or each query. This is based on the fact that statistically, you will be using the same patients more frequently for a certain length of time just like you will see Patient A five times in two weeks, and then not see him or her for two months. So, the patient query you used two weeks ago is stored as RAM in SQL. So, just like RAM on a computer, the queries you use within RAM, are going to be much faster than the queries that go to the AC folder and the AmazingCharts.mdf database. This is where a server can be more efficient than a regular OS. It goes months without a reboot. Every time you reboot, the cache is cleared. So, in this instance, pardon the pun, not rebooting is more productive. The reason a lot of users state they have better performance when they reboot is because they have 4 GBs of RAM to run AC, their billing software, all their services, background programs, etc. This is really 3.2 GBs. So when SQL takes 1 GB of RAM, now the available RAM is 2.2 GBs of RAM. That just isn't enough to go around. So a reboot puts all the RAM back in play. Which is why a 64-bit OS is so helpful. You can add a lot of RAM. It's cheap.

One thing that may help is SQL 2012 Standard. But, you would need to have it installed by an expert, even remotely. But, I would make sure that is an issue. Of course it can't hurt.