Home Forums Discussions General Discussion Bad Server! Bad Server! Behave Like You are Supposed to: a tale of woe

Tuesday morning this week, I arrived to the office at my usual time 5:45am. Checking email, I noticed two messages generated by the server's operating system Small Business Server 2008. The first date/time stamped message said there was an error with the Virtual Disk Service and it had stopped. The second date/time stamped message said the server had shut down unexpectedly. For those of you without a server, trust me when I say this is usually a VERY BAD omen that something major is wrong with the server. Since everything in my office connects to it, no big deal right?

Well I was about to see the 6am patient, so I decided this was a fluke and probably won't happen again. Afterall, restarting Windows platforms fixes 90% of problems so this issue must have resolved itself....wrong assumption. Because three times on Tuesday during the workday I was met with bewildered staff saying "I can't connect to AC or the practice management software." Gee thanks for stating the obvious, I was kicked out of AC also.

I ran back to the server and noticed the boot-up screen. Oh great, it restarted itself.....not good. However I figured this was half as bad as the server not starting up at all. So I prayed this wouldn't happen until after business hours. Ok that didn't work because it happened two more times during the day.

Being pissed at the server and mad at Windows for creating an O.S. that would crash spontaneously I naturally did the obvious thing: I went home and decided to go white water kayaking. It's a phenomenal workout and exhilarating rush. Plus I needed to work on my wave surfing skill, and all of this took my mind off the mysterious crashing server.

At this point it's important to add that I never touch my server. I was not in there adding or changing programs or settings. I did not reset anything recently. I've learned that more problems happen when frigging around with the server so I never go in there. Perhaps that is to my detriment because I'm getting rusty with the SBS platform.

I hoped I would get inspired to figure out the problem, but no inspiration came. I chatted with an I.T. patient of mine who thought it sounded like a hardware issue. Ah here it comes, the classic hardware vs. software debate. Which one is it? Software guys always blame the hardware. And obviously hardware guys always blame the software. We see it medicine all the time. Pulmonologists blame the heart, cardiologists blame the lungs, and GI docs blame the psyche. If you haven't guessed by now, my I.T. patient is a software guy.

So he says, it's probably a power supply problem. Ok fine, but I have TWO power supplies. One is redundant to the other as a backup and kicks in the instant the 1st one fails. It literally has two power cords. Really could this be the problem? well my machine is from 2008, so it could be the issue. Fortunately I have a backup server which is the exact make/model so I can steal parts. ok I have the plan of attack, it's late on Tuesday night so I'm going to have a beer and go to bed.

I awake 4am Wednesday, I'm planning to go in early and swap power supplies from one server to the other. This should prove or disprove the power supply theory. The swap is easy, they have quick release locks from the back of the machine. So it's a very fast unsnap/unsnap, then snap/snap. Hook up the power cords and restart the machines. Now I wait to see what happens. Meanwhile I check email.....drats!!! There are FIVE alert messages that the server unexpectedly shut down over night. Well it's ok, because the software guy said it was hardware. So I put the bad units in the secondary server and the good units in the primary server, so I should be fine right?

Well....it wasn't alright. By 5:30am the server shut down two times. I decide to create up to the minute backups. Yes I had backups from the night before, but I want to physically retrieve the latest backups to a portable drive in the event this sinking ship goes under. Crap, the server shut down two more times while trying to retrieve the latest backup. Sure I had these on another external drive but call me crazy. I wanted another set just in case my backups were no good (it happens).

5:45am my staff comes in. Directed by the sign on the staff entrance "find me in the server room before you do anything," Amanda asks me what's wrong. Well the server is crashing and this day is going to suck, I stated. I want you to reschedule the non-urgent patients to next week, but keep the urgent ones. I can handle a 1/4 to 1/2 schedule and handle a crashing server, but I can't do this while seeing a full day of patients.

Thus I begin moving my billing software and AC to run off the secondary server. Ok now I must admit that I'm not setup as fancy as some others on the ACUB. I don't have VMware so I can't just move an instance of AC or the billing software to my backup server. I have to install them there and restore the data. No big deal right? When you need AC to install within 5 minutes and be completely restored, I'm here to say that it takes a little longer than this.

6:00am "We couldn't reach your 6am physical and he is here." Can't you apologize for his troubles and reschedule him? "Well he drove from 30 minutes away and has not seen you in 2 years. He is way over due for his physical." Ok, fine I'll let AC install and see this guy. A watched pot never boils, and it seemed like it was taking one million years for AC to install. It wasn't but stick with me here people, time has a way of slowing way down when you are starting to freak out. Is my business going to be functional today? Hell today is out the window. Tomorrow? The rest of the week? How long until I get my business restored? And how badly will this cause next week to pile up? Oh darn, I'm going to visit my fiance in PA on Friday, will this cause my trip to be cancelled? Argh!!!!

I walk into the room and I look frazzled. The PATIENT ASKS ME IF I'M OK....talk about roll reversal. Yes, I'm fine but be patient with me sir, my server is crashing and I'm trying to save my computer before the business goes down the tubes with it. Yeah sure, I'm just dandy. How are you? Anyway, I figure I can get through this physical without pulling out necessary info from AC. He's pretty healthy and all his labs were printed the day before so that I could review and give him the hard copy (per usual office protocol). "Hey doc, did you notice I lost 20# since I saw you 2 years ago?" Gee sorry I didn't notice, in case I wasn't clear when I said my computer system has CRASHED, which means I can't compare your weight. Sorry pal. "Oh it's ok doc, so I went ahead and cut my Benazicar in half, is that ok?" Alright now I'm feeling really lost. Benazicar, Benazicar. What the hell is Benazicar? It sounds like he is trying to say either Benicar or benazapril....if only they were the same drug. Sir, do you take Benicar or benazapril? "Oh sorry doc, I don't remember." FOR THE LOVE OF GOD IT'S THE ONLY MED YOU TAKE AND YOU DON'T KNOW IT'S NAME. LESSON #1 for patients: Know the F%@#ING NAME OF YOUR ONE DRUG YOU TAKE!!!

Alright, so I've affirmed my decision to cut the patient load WAY down. We get through the visit, he cuts his Benazicar or whatever the hell he takes to 1/2, keeping a BP log and let me know if it gets above 140/90 on a consistent basis in which case I'm going to tell you to take a whole pill again. Keep up the weight loss efforts and keep working out in the gym. Good job buddy, see you in ONE year, not two. You are almost 50 and those regular checks are a little more important. "Haha, ok doc, see you NEXT year. Have a great day." Wait a minute, I'm supposed to say that?? Oh and P.S. I'M HAVING A SH#TTY DAY.

Back to the server. AC is installed, now I get to copy the Import Items folder (all 35GB) which takes quite awhile from the USB portable hard drive. Another patient seen, sinusitis. Zpack. "thanks doc." No problem buddy. Call it in girls since we don't have ERx yet nor do we have a physical Rx pad in the office. thanks all. Back to the server.

Import Items done, sweet. Restore backup utility. Error message. Oh crap. Fix the error. Hold breath.... Blip blip blip....ok done. Check a few things, it appears to be in working order on the server. Awesome. Next the billing/scheduling software, restore. Done that was easy.

Ok next to the clients, make sure to reset the path to the backup server and the new shared AC folder. Waiting....waiting....waiting....AC says "cannot connect to the path, please try again." ARGH ARGH, double ARGH. Sorry it's been awhile since I installed a clean copy of AC on the server. What did I screw up? Shared permission? check that, no it's fine. Ports? Firewall? I vaguely recall something about the ports or firewall on the server. Crap what is the port?? ARGH. Girls call AC Guardian Angels and press redial Q1second until someone answers.

At this point I lost track of time, it was probably 8:00am and I'm certain we were the 1st call of the day. If I wasn't so stressed out about this issue, I would remember his name. But the dude at ACSupport helped save the day. what's the port to open? Oh great that's right! Now stay on the phone with me and please bear with me but can you walk me through the steps to open those ports in windows firewall? It's been a long time since I had to do this. "sure not a problem." I log into the backup server from my workdesk station, configure the exceptions to the firewall. Zipity do-da I'm in like Flynn!! Dude you just saved my day! "You are welcome, is there anything else I can do to help?" No dude, gotta go but thanks again.

At this point we are pretty good. At least functional and it only took about 2 hours to restore to a functional machine. And I saw four total patients during this 2 hours. Not bad, eh? Not great, but certainly not bad.

Ok next I need to figure out what the hell is wrong with my terribly misbehaving server. Oh yes and don't forget the backup strategy to the backup server, as I can't afford to have that one crash at the least opportune time. Murphy's Law, or some such thing. I figure the backup strategy is more important than fixing a dead server with my mission critical information off it. So using Backup Assist I create backups to an external drive and set the AC backup feature.

I started looking at the log files in SBS on the broken server. I start with searching under Critical errors. Virtual Disk Service stopping and the Server restarting on it's own seem like Critical errors. OMG there are miles and miles of reported errors. Most of them not pertinent to anything. There it is, I find the dates and times of the server restarting and VD Service stopping. Great, but it gives me no explanation as to why. Well it does in the report, but it's reported in binary machine code. Crap I forgot my binary machine code translator on the kitchen counter along with my phaser and teleport coordinates. Scotty we need maximum power. "I'm giving it all she's got Captain." Mr. Scott you must give more power......oh crap I'm hallucinating.

Time to call computer guy. One of my staff is married to a computer repair guy. He sends over their MS Server expert. He digs around for an hour. I sit at my desk drooling and shaking with anxiety. I hate this crap I hate this crap I hate this crap. I can handle a guy coding, or an MI, or any of hundreds of medical scenarios. But my computer crashing....that is pushing my tolerance to the limit.

A few more patients and I'm almost done for the day. "Hey doc, can you come here for a minute?" I said look I never touched anything in the server. I never go in there. I sure as heck didn't do any recent installations or program upgrades. So what made the computer go wacky? Do you think it's a hardware problem? "No doc, it's 90% likely to be software." Really?? what?? "On monday, 55 minutes before the VD Service stopped, Kaspersky anti-virus did an automatic upgrade. Then your server responded by restarting to restart Virtual Disk." To those of you without servers, Virtual Disk is what makes the computer think the 8 hard drives on my server are actually one gigantic disk. Kinda important, see? "I don't know if that's what happened but it seems to fit the time frame." Sure and of the fifty million times it updates the virus database, why didn't it cause a crash then? "I dunno, but I think that's it. BTW here's an article I found for your version of Kaspersky and your SBS2008 version R2, it's known to cause VD Service to quit and the server to restart. Just a hunch."

I did the logical thing: I threw the server into the ocean. Okay I wanted to but refrained from doing that, sort of counter productive ya know. I uninstalled Kaspersky antivirus from the server. In the last 36 hours there were no more crashes!!!

I'm going to have that talk with the server guy. Help me get SBS2011 installed and configured, help me get file replication to the backup server going, and setup VirtualMachine so next time I plop a working copy of the primary server into the backup server and I can run diagnostics on the primary without losing so much time.

All in all, the backup server did what I wanted. It was simply present in the case of a disaster. I had a backup data recovery methodology that restored successfully onto the backup computer. I was able to see 8 patients and generate at least enough income to pay my staff and the computer guy.

11:45am, my head was throbbing. So I took 800mg ibuprofen and 0.25mg lorazepam and a shot of 14 year old Oban single malt Scotch. Then I went to the YMCA and swam 1 mile in the pool to burn up the nervous energy. Well that didn't work, so I got on my road bike and cycled 25 miles in 1 hour 15 min. Ok, now I'm feeling near baseline and very exhausted. Lay in bed for a couple hours. Go to the YMCA Board meeting (I volunteer there), get home at 6pm and fall into a very deep sleep. Today everything is working fine, no problems. YAY!!!!

thank God I only had a bad day, not a truly catastrophic business failure.
.....and if you read this entire story, God bless you my friend. It was truly a day I don't want to relive. In case you are wondering why I'm up at 3:30AM to write this story, I'm going to see my fiance today. We have been apart 5 weeks and we are dying to reunite. I start driving in 90 minutes. YAY!!!!!!!!

Sorry about your server. I'm glad that it's over. I've heard of a similar situation with McCafee and Norton, but it didn't cause restarts.

Even though your IT guy deemed it a software issue, these restarts seem more like a hardware issue. Even though it may seem like the power supply is the problem, I've seen this plenty of times with failed hard drives which is probably why your first guy suspected the power supply. Most redundant PSUs/Servers in general have a WatchDog line, which basically restarts the server if it hangs/stops responding. The watch dog line is directly connected to the power switch. So if the os were to hang/crash or a hard drive were to fail or stop responding for some reason, the server would restart to maintain uptime or in the case of a failed hard drive, continually restart. Which makes it seem like a power supply issue. These restarts would be occasional in the beginning and as the corruption spreads throughout the hard drive, the restarts become more frequent. Just a thought. I haven't seen the logs so I can't make an informed decision like your IT guy.

A couple of questions:
1. Do you have a hardware RAID controller? Are you using software RAID?
2. Did you run a disk utility from the drive manufacturer? E.g. Seatools for Seagate
3. Do you have an eSATA/USB 3.0 backup drive? Copying 35GB on an eSATA/USB drive would take about 5-10 minutes which is why Bert and I strongly recommend them over USB 2.0
4. What do you mean 8 hard drives as one big hard drive? Did you do a software RAID 0?!

Virtualization is a good idea assuming you have no hardware devices attached. I have a fax board, that's my main reason for not virtualizing. Be sure to take that into consideration. There's ways around it of course. Use a Hyper V Parent with Fax Services and join into the Hyper V Child SBS 2011. Or a separate physical fax server or UpDox or PaperPort. Whatever floats your boat.

I can't throw my two cents in here without disagreeing with Sandeep and starting the whole "hardware versus software" thing that LauerDO complained about in his post.

And what's so bad about that is I'm both an IT guy and a software developer so I have no loyalty to either the hardware or the software. I both love and hate them both equally.

And LauerDO hates me enough already. . .

JamesNT

Could have had a peer-to-peer system, swapped the "server" for another client and been up and running. Just sayin'.

Actually, that's not really fair; I don't mean to kick a man when he is down. I know from many of your prior posts that your good friend Bert helped you set up a system that has served you well for years. I don't mean to knock your system and his excellent advice; sometimes I just can't resist the urge to put in a good word for us poor P2P folks.

That's pretty much what he did. Copied the data over, reinstalled the programs, and was operational in 2 hours. 35GB of II and restoring the database takes time. Anyone who's running a server can double back to P2P. So that's not really an advantage of P2P ha. He could've just as well have done a bare metal restore to the night before and had a fully operational server. That requires the same hardware. This is where Virtualization is awesome because you can move virtual machines between physical machines if necessary.

Finally someone who drinks scotch....I have one taped to the wall near the computer (small single malt bottle) that says "In case of crash drink this before thinking".....

Crown Royal. Doesn't that count for anything?

JamesNT

How about Knob Creek, Pendleton, Basil Hayden, Woodford Reserve?

Took out the fire extinguisher and put a bottle of Johnnie Blue. Break in case of emergency.

OK, this was made way too hard on many levels.

1. This is a time for paper
2. We do have prescription pads
3. As soon as your server gives that many warnings, you stop using it and troubleshoot.

He has a hardware RAID.

@Jon, What we are trying to say is a client/server domain is not some magical and difficult concoction of computers. I have on two occasions in five years, simply turned off the server, restored AC on a pre-designated computer that has SQL, redirected the other computers and NEVER had a port problem. But, (and I make these rules up), but never look for the port -- turn off the firewalls). I have never had to open port 61067 which is also on the AC website. Most programs open ports by themselves. If not, use the exception on AC first if you don't know the port.

It is easy to back up your II onto the designated backup computer.

Here is one huge lesson, although not sure if I can convince everyone of this.

Do what Susan Bradley teaches (Google her) she is the SBS goddess. Seriously, she has Bill Gates' phone number and, more importantly, he has hers. DON'T put antivirus on your server. What? Yes, don't. There is little that goes to your server. Block the SPAM and viruses in the cloud. You have your firewall. All of your clients have antiviruses. Run MBAM and Super-Antispyware manually once per week.

This is especially true if you are using Hyper-V. Then antivirus on the physical server and none on the virtual. Have you ever looked at the list of exclusions for AV software on a server? There are more exclusions than the other way around.

And another thing. We all love to hate Microsoft. But take Microsoft and add $259.00 and you can drink all the Scotch you want while they take over and fix your computer. And, they WILL fix your computer. You already have support from Dell.

I'm with Bert. I do not put Antivirus on any of my servers. I have too many legitimate programs that most antivirus programs would freak out at. In order to explain why AV software screams at perfectly legitimate programs, it will have to be earlier in the day and I can't have this much wine in me at the same time, but there are legitimate reasons.

I especially do not put AV on my Hyper-V hosts. Besides, why bother. My hyper-v hosts NEVER see the Internet, and besides a baseline installation of Windows Server 2008 R2, they are just servers with 4 or 5 gigantic files on their drives (the vhd files of the guests).

JamesNT

Furthermore, yes we all love to hate Microsoft. Buy why are we dishing on MS when the cause doesn't appear to be MS?

JamesNT

Hi Sandeep,
to answer your questions (in capitols, I'm not yelling rather differentiating my text from your text.)

1) Do you have a hardware RAID controller? YES. Are you using software RAID? NOT TO MY KNOWLEDGE, I USED DISK UTILITIES TO SET THE HARDWARE RAID CONTROLLER PRIOR TO SBS INSTALLATION.

2. Did you run a disk utility from the drive manufacturer? E.g. Seatools for Seagate. YES, NO HARD DISK ERRORS.

3. Do you have an eSATA/USB 3.0 backup drive? Copying 35GB on an eSATA/USB drive would take about 5-10 minutes which is why Bert and I strongly recommend them over USB 2.0. NO I DON'T, BUT I DEFINITELY SEE THE TIME ADVANTAGE AND THAT'S A FUTURE PLAN WHEN MY PERSONAL LIFE SETTLES DOWN. (GETTING MARRIED IN A FEW MONTHs)

4. What do you mean 8 hard drives as one big hard drive? Did you do a software RAID 0?! I WAS OVERSIMPLIFYING, SORRY. THE OS IS ON A RAID 1, 2 DRIVES PLUS 1 HOT SPARE (3 TOTAL DRIVES). THE DATA IS ON A RAID 5, 4 DRIVES PLUS 1 HOT SPARE 5 TOTAL DRIVES).

Maybe I misunderstand how virtual disk functions. If for example 3 drives are setup as a RAID 5, data is distributed across those 3 drives in a redundant manner so 1 can go down yet the data remains retrievable. However the OS "sees" 1 disk, not 3. If I don't understand virtual disk correctly, please educate.

Thanks James for the input, I'm keeping antivirus off the server.