It's the Screaming

Tim Bray observes:
The community of developers whose work you see on the Web, who probably don’t know what ADO or UML or JPA even stand for, deploy better systems at less cost in less time at lower risk than we see in the Enterprise. [emphasis in original]
That's a pretty bold claim.

Oh, certainly, the Enterprise makes a botch of IT projects on a titanic scale, and Tim gives a partial list of very high profile failures. I think this data is tainted. Firstly, these project failures (such as the UK NHS NPfIT programme) are, well, very high profile—but they aren't representative. Plenty of corporate IT projects do very OK. Secondly, the specific examples Tim gives are actually government projects. Government IT is the reductio ad absurdum of Enterprise IT. These might not be the most informative examples to look at.

On the other side of the matter, Tim cites the successes of Facebook, Google and Twitter. These are also not representative. Quickly, name three failed web 2.0 startups...that's a hard ask because those startups that don't succeed sink without trace. Which, if you believe in the Y Combinator argument, is the point. A web 2.0 startup needs so little initial investment that western civilisation can afford to throw them away in bulk without noticing. The result is a sort of Monte Carlo method for product innovation. I wonder how that's working out for them now that money is no longer so cheap.

But still, something is very wrong with enterprise IT. Would the "web 2.0" way of working help?

Maybe, maybe not. I think Tim has made the mistake of assuming that a technique that he's seen work in one context will necessarily work in another. And should be deployed there post-haste. The IT industry collectively makes this mistake about every seven years.

Some commentators have noted some of the differences between the Web and the Enterprise. In summary: web 2.0 startups don't have to integrate with a heterogeneous ecosystem of legacy systems older than the combined ages of their founders. Web 2.0 startups don't have to operate within multiple tight and complex regulatory regimes. Some were even so unkind as to observe that Tim is one of the devisors of XML and so some of the chest–deep mud that Enterprise developers have to slog through is his fault.

I think there are two more issues that Enterprise groups have to deal with that web 2.0 startups don't, and don't have a good story for.


I'm going to say that enterprise IT faces a much tougher scale problem than do web developers. Part of the web 2.0 model is that the initial user base is the founders plus their immediate circle. The problem that startups face is that if they are one of the (very) few that make it, their user base may grow very quickly. Their problem is scaleability. This is a tough problem.

The problem for the Enterprise is scale.

Citigroup, Bank of America and JP Morgan Chase all have slightly less than a quarter of a million employees. If you are deploying a "corporate IT app" at one of those organisations then you have to plan for all quarter of a million of them to hit your app at 9am their local time tomorrow. And all day and everyday thereafter. That's a different problem. And by the standards of corporate IT this is no–where near as bad as it can get.

You know that NPfIT project that Tim holds up as an exemplar of corporate IT failure? In 2008 the NHS had 1.12 million non–medical staff, 99,000 medics and dentists and 34,000 General Practitioners. Imagine that they are all going to start using your app tomorrow. To help them treat patients. Of which there are several million a week handled by NHS services. Putting up the fail whale isn't going to cut it.

And that's the real problem that corporate IT have the deal with: the screaming.


Let's say that your web 2.0 startup thingy that you put together in record time in your buddy's mum's spare room falls over. What's the worst thing that can happen? A few snarky tweets? The odd complaining blog post? The whole point of your business model is that your service (and company) are disposable, at least in the early days. It's not as if you have a revenue stream to protect. As if!

Now lets imagine what happens when your FX app you just deployed into Mammon Inc's trading floors falls over. What happens then is that a head of desk in Farawayvia phones you up at 3 am your time and will not stop screaming at you until you get it fixed. I exaggerate for comic effect, but not by much. If you are a Web 2.0 type then you have the luxury of fixing stuff up in something like your own time. If you are a Corporate IT type then you might well have someone who believes that you, personally, are trying to steal their bonus from them raging at you until you get the damn thing working again.

Or that you, personally, are trying to kill their patient.

If you were a founder of a web startup and the users of your service had your personal mobile number and believed that five minutes of downtime could cost them personally millions of bottletops (the currency of Farawayvia, ticker symbol BTP), or lead them to a malpractice suite over a dead patient, do you think you would behave the same way that other founders do?


So, what's the answer? I don't there is any "the answer". Not a technical one, anyway. There are a bunch of tools and techniques that I've come to think will help a lot, and I encourage my clients who happen to be in corporate IT to use them. I fact, Tim mentions some of them in his post. He notes that startups get a lot of help from:
dynamic languages and Web frameworks and TDD and REST and Open Source and NoSQL at varying levels of relative importance
Too right. But, as Gerry Weinberg says: it's always a people problem. It always is. It's the screaming.

Get people into a place where they believe they can do the right thing and they mostly will. The technology can then, to a surprising extent, look after itself. This is the really big advantage that startups have, I think: no–one to tell them not to do the right thing. It's not that corporate IT is full of people saying "don't do the right thing" (although in the worst case that does happen). Rather it's that the social context inside any organisation big enough to call itself an "enterprise", without any actual malice on the part of any individual, works against the right thing.

I think it was Ron Jeffries that I first heard say that most of the advice given to software developers to help them do better is about as useful as telling them to be taller. Telling folks who work in corporate IT to behave more like the folks in web startups seems no more useful than that to me.

On the other hand, if one does end up working in a really toxic environment that does work against the right thing it can be worth looking around. Look around to see who, exactly, is holding the gun to your head to make you put up with it.


Lex Pattison said...

So - your defense against operating using 'agile' principles and iterative development is that complexity and unnecessary redundancy is necessitated by legacy systems, regulatory regimes and the fear that an ignorant user base is going to scream and yell at you. Seems to me iterative development is the key to avoiding large scale failures, and operating safely with legacy systems.... iterate slowly away from their implementation... iterate slowly within the regulatory mechanisms. Your argument is a massive Fail Wail - pure rationalization.

Steve said...

A perhaps simpler way of putting Lex Pattison's comment would be this:

Consider if the NHS NPfIT system was rolled out first to a small userbase. Perhaps one hospital and a few surrounding GPs. It could prove itself in the field with incremental updates before being rolled out further.

Now consider that instead of commissioning software, they commissioned protocols for secure transmission of patient records and certification schemes for software implementing them (this has to be done anyway) and allowed every GP in the country to choose their own software supplier.

The problem for these approaches is they launch with a whimper, not a bang. There's no grand opening, no point at which someone can stand up and say "I have done it, promote me." This to me seems to be the root cause of the problem

keithb said...

@Lex, I reproduce here the reply I made to your same comment on reddit:

Not at all.

I actively encourage my clients who have legacy systems, regulatory regimes and the fear that an ignorant user base is going to scream and yell at you to overcome those problems using agile principles. The word 'agile' is in the name of the business unit I manage. Clients hire me because I'm a consultant specializing in Agile practices.
It seems also to me that iterative development is the key to avoiding large scale failures, and operating safely with legacy systems.
What I don't recommend is that folks working in an economic, regulatory and what-all-have-you context wildly different from that of a web 2.0 startup should expect that web 2.0 startup practices will work just fine for them.

Sorry if that wasn't clear.

keithb said...


I have no doubt that the desire for a "bang" is behind many large failures, especially in the public sector.

Politicians love to announce grand projects. Civil servants love to manage huge budgets. Neither has a good way of measuring outputs so they measure inputs instead.

As a result, telephone directory-like RFP documents go out (and only the largest professional services firms can handle them) and telephone directory-like proposals come back. The only question that government departments can bring themselves to ask of their suppliers demands a kind of answer that greatly increases the chances of failure. There doesn't seem to be a way out of this.

Anonymous said...

I worked in an "Enterprise" software development (product + services) environment for 7 years and in "web 2.0" for 5 years after that. Having had that experience and reading the post; why would anyone willingly work in enterprise software development? I make a lot more money than I ever did in enterprise related work (and I made *a lot* even there), I work when I want and on what I want. Ofcourse you need profitable ideas, but I appear to only have those.

Enterprise development seems largely the domain of people with not very much inspiration and a high tolerance for stress and, indeed, screaming.

Steve Freeman said...

Is this the wrong way around? My impression is that systems which are directly linked to someone with authority who cares generally work--perhaps not well, but enough.

The real problem is where decision-making in enterprise IT is just too far from any responsibility from its consequences. The classic example is underpowered development kit: IT purchasing manages their costs down, but the company as a whole loses value. As we all know this is rife in large enterprises, just consider the respect most development teams hold for most corporate architecture groups. The result has been that every generation, institutions oscillate between centralised solutions and local workarounds. Once it was PCs, now it's teams using Facebook.

Even if Tim Bray overstates the case, Web 2.0 (whatever that means) has been an excellent wake-up call to show the possibilities of a new generation of technology. It will also trigger another generation of screaming, as in "Startup X wrote an entire CRM system in a month and sold it for $500M, why are you taking so long!"

keithb said...

@Steve F

If I have understood your comment correctly then no, I don't think its the wrong way round.

Let's not confuse someone being in a position to shout at the developers of a system with that someone having authority over and caring about the development of that system. And even more so with them being willing to and actually doing anything to help with it.