Home Forums Discussions Problems Need IT consutant

I am in need of help (beyond the psychological kind):

We have a LARGE group -- 9 providers, over 35 workstations including support staff, a zillion printers, 3 offices, 5 servers (one for terminal services, one for AC, one for our billing software and 1 for billing software database and not sure what 5th server is for), two internet pipes in case one goes down, VPN configured between offices, and using RDP to communicate using AC from satellite offices to main offices with server(s).

I understand the system is complicated BUT we seem to have daily problems, glitches, errors that cause us to call our IT people several times daily. I know they are VERY bright but I am concerned that they have IT DISEASE -- ADD with inability to work other than in crisis environment. This is costing us between $3.5-$6K / month to keep our system up and running!!

I would love to hire someone who could really look at our system in detail and come up with either a more stable way of doing things, and / or a checklist of things our IT people need to do on a set schedule (i.e. read error logs, update programs, schedule backups and test, etc). They obviously need to be someone with a LOT of experience with large systems, very logical and compulsive and could do this work by logging into our system for examination. It is possible our IT folks are doing a great job, and I suspect they are, but need to have another set of really competent eyes look and confirm and /or improve our system.

Any and all recommendations greatly appreciated.

BEst,

James P. Clayton, M.D.
Red Hook Family Practice
US Virgin Islands

Are you paying your current IT team to do proactive monitoring and documentation or do you instead pay them on a time/materials basis when a problem arises? If T&M, there may be a disincentive for them to fix the problems so they don't resurface.

with a more complex, unstable setup like this, a couple basics should be in place before I would hire another person. Ensure your IT group has:

1. Developed detailed, written documentation of your physical and logical IT architecture. This means documenting the existing hardware configurations as well as documenting software and OS configurations, key data flows and backup procedures. Everything from servers, network switches, and the clients.
2. Documented incidents each time and regularly reviewed the data for trends. Documenting these incidents includes noting who is having problem, which location and piece of hardware, capturing problem description, & documenting root cause once fixed.

This is the bare minimum to expect from your current IT service provider. When you have this documentation and the problem history, then a new or existing IT support person can start to pinpoint mis-configured or bottleneck components of architecture. It also prepares you for changing IT service providers if this becomes necessary.

Eric brings out some excellent points, and the under-current is that as an busy practice already having challenges, adding/changing a key provider can amplify the problems.

I do wonder if their architecture hasn't kept up [from a plan/clarity point of view] with the systems growth and complexity.

There are some diagnostic nuggets that would helpful to understanding current circumstances, to wit:
<>are users logging in against the domain controller?(domain versus local password)
<>are users running as users without administrators rights?
<>what kind of traffic monitoring/shaping are you running at each location (e.g. MRTG, squid, Sonic, Watchguard, Cisco)
<>is there a plan for regular maintenance & patching?
<>is there a plan for regular scanning virus&HIPS checks?

I could go on, but I'm a recovering engineer, and I need to know my weaknesses.

With information like this you could have an off-line conversation with a systems professional, and provider that person a LogMeIn or join.me session to take a quick peak around the system.

My guess is that you need to team your local "boots on the ground" with someone with a more holistic perspective on multi-location architecture & system, and help them start being more pro-active in their work.

You'll need some mrtg (or similar) network traffic data, but I also wouldn't rule out network through-put latency from your local provider as being part of the problem.

Indy and Eric have some excellent points.

I would love to know what's in your system's event logs.

As I have pointed out elsewhere, there is no excuse for having those kinds of problems at such a high frequency - regardless of operating system.

I PM'd you my number. If you wish to discuss what's going on, I'd be happy to offer what advice I can.

With myself, Indy, Eric, Bert, and others here, you could also try to describe the issues you are having in posts and maybe some of us can point you in the right (or at least better) direction.

JamesNT

Thank you SOO MUCH, everyone, for replying so quickly. My local IT guy is knowledgable and I suspect the system configuration is correct. I like the idea of asking my IT person to map out the entire system and will request it. Once done I would love to have someone else peek into my system and draw up a reasonable plan/preventative algorithm that my IT guy has to follow to prevent baddness from happening.

Examples: Trojans entering system despit SonicWall with AV and individual servers and workstations with SymmantecAV. Trojans removed with Malware but then Malware not placed on active mode so trojans get in again, past Symmantec and propagate throughout system.

Updates installed irregularly and last time no updates for server(s) and our practice manager updated it crashed (because of outdated system software that was not updated).

System modifications done Sunday night so that when we arrive MOnday live we beta test what was done and frequently end up with BIG problems. Has happened way too often. (We are off line on weekends, all weekend from Friday night).

Setting changed but then mysteriously revert back to dyfunctional way (i.e. printer settings, user settings, etc.)

System logs not routinely checked -- sometimes every other day then someimes not for several weeks until something bad happens.

Sonicwall and internet providers -- when one provider slows down to crippling slow speeds but does not crash the Sonicwall does not switch to the other provider -- have to call IT guy to manually switch. Only switches automatically IF on provider goes down completely.

COST: We are now paying a minimum of $3K per month and some months up to 5-6K. Even for a medium size office this seems like a LOT and would love to know if this is in the realm of reasonable IF I had a stable system?

I could go on but I think that is the general drift.

Tnx,

James

Most of the issues above sound like problems with either the configuration and change management procedures followed by your office. Change management allows you to schedule and plan for changes to your IT configuration (no updates during normal working hours). For instance, doing any updates on Friday night so testing can occur on Saturday. Good change management also gives you backout procedures that, if needed, can be finished before work starts again on Monday morning.

For a small office with 35 clients, there are many methods to lock down the environment so people don't change the setup at unscheduled times. Removing administrative permissions from regular users so users can't easily install software, making all clients use same build image, and using internet based email vs. local pop3 email clients are some fixes applied to bring stability in larger installations.

Also, on the Sonicwall incident, this gets back to configuration management. Typically, the IT provider would align and document setup with how you wanted network data flows to work. The IT provider would then review the proposed configuration against expected ways to fail system to be sure it meets your needs. In your setup, the IT provider likely had to chose one of the 4 methods available for Outbound Load Balancing (Basic Active/Passive Failover, Per Connection Round-Robin, Spillover-Based or Percentage-Based). After failure he now knows the correct configuration choice but what is to prevent a bad choice later? A configuration plan would address this.

On the cost aspect, you are basically paying a full time person to maintain your IT ($35-60K per year). I'd say that is about right for that many clients if you don't have in house expertise to deal with configuration, change and other issues. If you can develop a person interested in IT and willing to act as "front-line" response, or 1st level support, you may be dollars ahead. Then, this onsite power user can call the IT guys if they encounter more complicated issues they can't fix. The more expensive IT guys would be your "2nd level" support.

James,

Wow. A full plate you have. [nod to Yoda]

To summarize what I am hearing:
<>Security - OS configuration/mitigation/cleaning/prevention
<>Network - edge configuration/traffic shaping/monitoring
<>Server Management - patches/upgrades/testing
<>Proactive management - logs/testing/spot-checks

Your first task is to gather the info that will allow you to effectively communicate to another pro what all is going on.

To be honest with you about expectations, in order to remediate the problems you have (including the ones you don't even know are lurking out there) will take some time, and multiple passes through your systems. Because much of this is interrelated, you will need an overall plan and priorities, knowing that things will continue to break and impact progress.

Trust me it is do-able, but it will take some patience and forbearance.

This is way over my head, so I am just going to ask some simple questions. First, it isn't necessarily a large setup (I know IT people who deal with hundreds of workstations). But, it does sound complicated.

When I hear 35 workstations and 5 servers, I immediately think that's great that a server isn't sharing the whole load, but I also think that five is a lot. I guess it is because I don't know the topology of the setup.

You say three offices and five servers and terminal services, but I don't know where the servers are, what office is connecting to where, what office is the main office (I suppose I would say "main office" "Sat1" and "Sat2"

Also, there seems to be TS and VPN, yet you still have people using remote desktop to log into AC.

I am not understanding why malware is getting by your antivirus programs and firewalls.

As mentioned, every user should be a domain user with no admin rights and logged into the domain. I know this is overboard, but I would be tempted to run all WIN7 with AUC on.

Also from your description it sounds like IT comes when you call them. If that is the case, spending upwards of $72,000 a year seems like a lot. If I spent that much, I would want a guarantee that no Trojans were coming into the system.

With 35 workstations, how many are infected? Do you just need to make an image and reformat every PC with the same setup?

Again, I don't know where all the servers are, and I am sure a real IT person would be excited that everything is separate. But, it seems a bit much to have billing software on one software with billing database on another. Then AC on another, etc.

The last thing I would like to know is the credentials of your IT. Do they have this after their name:

CCNP/CCDP/Cisco Wireless LAN specialist, Cisco Security Specialist, Cisco Sales Expert /MCSE. Major areas of expertise: frame-relay, LAN switching, Campus Network Design, PIX firewall, ASA firewall, VPNs, network management...
CCIE-Security Candidate (passed written exam 350-018)
Add Microsoft MVP 2004/2005/2006/2007/2008/2009 to the list

or just "Really smart IT guy." I really am not trying to be sarcastic, but one has to wonder what the credentials are.

I am not sure how many IT people are in the area where you live.

Well, you have three very knowledgeable people helping, but hope this helps.

Thank you EVERYONE for your suggestions -- they have given me a roadmap to get started. As it turns out, all users were given adminiatrator rights which is what allowed the trojan into our house. This has since been corrected.