Paramedics and Neurosurgeons

By Doug Kaye

August 20, 2002 -- (WEB HOST INDUSTRY REVIEW) -- In their marketing brochures and sales pitches, many MSPs project images of well-integrated services in which the people who build and tune your servers are the same—or at least on the same team—as those who provide 24x7 support. The vendors are trying to give you that warm, cozy, one-stop-shop feeling.

But look closely at most MSPs’ org charts and profit-and-loss statements, and you’ll see a very different reality. You’ll likely find there’s not just a single group, but two very separate groups, each with its own vice president, profit center and unique business model. One model is based on proactive or professional services, while the other is for reactive 24x7 services,

The tension between these business models creates two related problems for MSPs. I call them the escalation-bleed and the charter-blur problems. Understanding these problems and how MSPs solve them can help you evaluate vendors and manage your ongoing outsourcing relationships.

Proactive vs. Reactive Services

Proactive services are those you typically pay for on a time-and-materials basis. The proactive staff members are the “professional services” people who configure systems and manage your servers. Post installation, these folks upgrade and patch your operating systems and applications and tune your databases. The more you want, the more you get, and the more you pay. From a purely P&L perspective, an MSP wants you to utilize as many proactive services as possible.

Reactive services, on the other hand, are typically billed on a flat-rate basis. You pay a monthly fee for each server or application under management. The reactive staff are those that man the network operations center (NOC) 24x7. Their jobs are to respond to outages, failures and slowdowns, and to get your site back up and running as quickly as possible. They won’t fix the underlying causes of an application failure on a Saturday night, but they’ll hold the pieces together with chewing gum and duct tape until the proactive team comes in on Monday morning. Due to the fixed price, of course, vendors are financially motivated to keep the services delivered through this channel to a minimum.

On first blush, it may appear as though the vendor’s objectives are out of phase with delivering the best value to the customer. You’d think that trying to do as little as possible in its NOC, and shifting tasks to its professional-services team would put an MSP at odds with its own customers. But as we’ll see, this isn’t the case. Ultimately, what’s good for the customer is also good for the vendor, but there are two tendencies that threaten every MSPs’ ability to manage its operations for the mutual benefit of itself and its customers.

Escalation Bleed

The first tendency, which I call escalation bleed, is when the NOC staff escalates problems they can’t resolve to the proactive team staff who typically (a) have more advanced skills, (b) have more intimate knowledge of each site, and (c) are typically more expensive. Once an MSP brings the proactive team in to solve reactive problems, it’s losing money.

As Mike Palmer, vice president of professional services at MSP Totality put it in a recent interview, “I’m looking at a 24x7 [NOC] environment that has a tendency to escalate issues outside the [NOC]…Seeing tickets being escalated is watching your money being thrown out the window.”

The solution to the escalation-bleed problem benefits both the vendor and its customers. It consists of three components: (1) a clear separation between the teams, (2) top-notch documentation provided by the proactive team to the reactive team, and (3) sufficient skills within the NOC/reactive team to perform the necessary triage without the need to escalate to the proactive team. If all three components are in place, not only does the vendor make more money, but the customer also receives better and timelier reactive services.

Steve Holmes, regional vice president of professional services at Exodus, pointed out when I spoke to him recently, that there has to be a “line of demarcation” between proactive and reactive service teams. These are fundamentally different roles and the people in each have different skill sets. The reactive team is “responding to issues [and] recovering systems versus [the proactive team, which is] designing and consulting,” he said. Holmes pointed out that even his customers want to keep the line of demarcation very clear. They want strict change management and explicit restrictions on who can modify their systems.

Charter Blur

The second problem is essentially the inverse of the first. In this charter-blur scenario, the NOC team goes beyond their charter of providing triage, and attempt to implement permanent corrective or preventive measures. The danger originates from three sources: (1) the reactive NOC team staff don’t know enough about the design and history of the site to make such changes, (2) they don’t have the long-term view of the site that the proactive team has, and (3) they often don’t have the required skills or experience to do it right.

By their nature—as well as their motivation and compensation—the NOC staff are encouraged to restore sites quickly, using whatever tools and resources may be available. They’re the triage team: the patch-‘em-up and move-‘em-out guys. These are the people you want to be there when things go wrong and you’re in a panic. But they’re not also the people you want to handle the more strategic aspects of your site, where you want a team that specifically doesn’t take shortcuts. For proactive measures you want people who will do things in an orderly and well-documented manner. You want a staff concerned with stability, reliability and knowledge transfer.

The solution to the charter-blur problem is to maintain another line of demarcation: a leash on the reactive team that limits the scope of their work to whatever it takes to restore service, but prevents them from crossing the line into preventive corrective actions.

Evaluations

The best vendors are those that have figured out how to maximize both their profits and the benefits to their customers. Their first secret to success in this area is having the knowledge necessary to build, configure and manage web sites that require little reactive support: i.e., sites that suffer as little downtime as possible. When sites keep running, everyone wins.

The second secret is the segregation of the proactive and reactive teams, not the overlapping and integrated vision portrayed by the sales pitches and marketing blurbs. Check the org chart: Is there a separate vice president managing each team? That’s a clue that the teams are well segregated.

Remember: Paramedics don’t make good neurosurgeons, and vice versa.