The Process for Programming (and why it might not help)

Paul Johnson writes about the infamous 1:10 productivity ratios in programing, and suggests that we don't know how to program, that is, there is no process for programming. 

I take the view that on the contrary, we do know how to program, and there pretty much is a process for programming, but it isn't much recognized because it doesn't do what so many people so desperately want a process for programming to do.

The process for programming that I have in mind is very much like the process for doing maths—I suspect that this is why (reputedly) everyone who goes to work as a programmer at Microsoft is given a copy of How to Solve It. What both activities have in common is a need to do imaginative, creative problem solving within strict constraints.

Paul lists a bunch of activities, of which he suggests that "the core activity is well understood: it is written down, taught and learned". One of these is "curing a disease".

I find that questionable. Have you ever watched House? Apart from being a very entertaining update of Sherlock Holmes, what goes on in the early seasons of House is (within the limits of television drama, to the eyes of this layman) pretty close to the actual investigations and diagnoses I have read in The Medical Detectives,  The Man Who Mistook His Wife for a Hat and a few other of the bits of source material. To a surprising extent, they aren't making that stuff up. See how the diagnostic team blunder about? See how they follow false leads? See how some apparently unrelated fact combines in the prepared mind with a huge mass of prior experience to prompt new ideas? See how they must take a chance on something that they don't know will work?

There are definite tools that are used again and again, primarily differential diagnosis (which, BTW, makes the average episode of House into a good short course on debugging), and the various physiological tests and so forth, but mostly it's a synthesis of knowledge and experience. If you ever get the chance to hear Brian Marick tell his story about how vets learn to recognize "bright" vs "dull" cows you'll get a good idea of just how the teaching of medicine works. And it works to a large extent by the inculcation of tacit knowledge.

Tacit knowledge is terribly hard stuff to deal with, it's in the "unknown knowns" quadrant. One of the things that makes teaching a skill difficult is the manifestation of tacit knowledge*.  A bit part of what a lot of people want a "process" for programming to do is to make explicit exactly that tacit knowledge. 

But there is only one way to do that really well, which is exactly why things like teaching hospitals exist. The way to do it is to try repeatedly with feedback from an expert. The external feedback is crucial. You may have heard somewhere that practice makes perfect. In fact, all practice does by itself is make consistent. It's the expert feedback that leads to improvement. 

This one reason why folks are unhappy with what we do know about how to program: it cannot be learned quickly from a book, from a one-week course. There is another reason, perhaps worse: what we know about programming does not (and if parallel with other disciplines based on skill and judgment, it never will) equip programmers to produce a given outcome on demand, on time, every time.

The idea has got around (originating perhaps with "One Best Way" Taylor) that a process for any given job should be like a recipe—follow it exactly and get the same result every time. In some spheres of work such things are possible. Programming does not seem to be one of them, no matter how much we might wish it.

* Design (and other kinds) of patterns are in part about capturing this stuff, which they do variously well. I suspect that this is part of the reason why more experienced folks who haven't seen them before say things like "that's obvious" or "anyone who doesn't know/couldn't work that out shouldn't be a programmer".