Observations on Estimation

Teams following a process like Scrum tend to estimate the "size" of stories as an aid to figuring out a commitment for a sprint. My view is that this is a transitional practice, and that the aim should be to learn how to make stories all roughly the same size so that commitments (also a transitional practice) can be determined by counting.

While all of that is going on teams that want to use a numerical scale to estimate (rather than, say, "t-shirt" sizing) tend to choose a scale, a sequence of licit values from which estimates must be drawn. The various planning tools that demand a numerical field be filled in tend to force this issue.

I've noticed a tendency for "expert" level practitioners to want to use some clever non-linear scale, maybe Fibonacci numbers (1,2,3,5,8,13), maybe a geometric series (1,2,4,8,16) and they will have some sophisticated reason why this or that series is preferred. And I've noticed that a lot of teams aren't comfortable with this. They want to use a linear scale.

It seems to be traumatic enough that the estimates don't have units, or even dimensions. The idea that estimates are dimensionless but also structured can be a double cause of confusion.

Anecdote: a team had been estimating and planning and delivering consistently for a good long time. Their velocity was fairly constant, but drifted over time (fair enough). One day it turned out that their velocity happened to be numerically equal to the number of team members times the number of days to the next planning horizon. Someone noticed this and with a huge sigh of relief the team concluded that these mysterious "units" in which they estimated were actually man-days in disguise. Now they finally understood what they were estimating! And they promptly lost the ability to estimate: their next planning session was all over the place and it took some time for their planning activities to converge again. My inference was that it's actually quite important that estimates are dimensionless.

Anecdote: a User Experience expert at a client had been involved in some research whereby (as a side effect) members of the general public had to create a scale that made sense to them within which to rank the usability of features. These folks were presented with different generic objects and asked to give them a "size", and then to give a corresponding "size" to some other generic objects in order to create a scale that made sense to them, which would then be applied to the merit of the system features that were the actual target of the research. They created linear scales.

[After seeing this he added the observation that this process was in aid of avoiding what often happens with the strongly disagree, disagree, no preference... type of scale which is either polarised or bland results, neither of which is that useful]

That surprised me at first, since I know that the physics of our sensory apparatus are generally non-linear, and memory is non-linear and so forth. But thinking about it some more I realised that our experience tends to seem to be linear, even if the underlying phenomena aren't.

Meanwhile, if one did want to use a particular scale for estimating the size of stories, why not use one of the series of prefered values? They are very well established in engineering and product design and offer interesting error-minimising properties. On the other hand, it might be a real struggle to get a team to decide if a story was a 1.6 or a 3.15

I don't have a grand narrative into wich to fit these observations, but here is another related anecdote about estimation.

Genius at work

Over the years there has been a lot of propaganda claiming that Google both creates the only working environment in which double-super-genius types can function at peak effectiveness and then goes out of its way to hire only double-super-geniuses to work in it. How, then, does this happen:

We periodically receive updates to that list [of malevolent sites] and received one such update to release on the site this morning. Unfortunately (and here's the human error), the URL of '/' was mistakenly checked in as a value to the file and '/' expands to all URLs.

This sort of thing happens because of bad processes. It doesn't matter how smart folks are, if they work in a bad way, poor results will ensue.

Of course, I don't know exactly what occurred, but I'll bet you a pound that one lone double-super-genius is left to make changes to this file by themselves. 

Update: Jonathan Wolter raises some questions about testing such changes.

The Process for Programming (and why it might not help)

Paul Johnson writes about the infamous 1:10 productivity ratios in programing, and suggests that we don't know how to program, that is, there is no process for programming. 

I take the view that on the contrary, we do know how to program, and there pretty much is a process for programming, but it isn't much recognized because it doesn't do what so many people so desperately want a process for programming to do.

The process for programming that I have in mind is very much like the process for doing maths—I suspect that this is why (reputedly) everyone who goes to work as a programmer at Microsoft is given a copy of How to Solve It. What both activities have in common is a need to do imaginative, creative problem solving within strict constraints.

Paul lists a bunch of activities, of which he suggests that "the core activity is well understood: it is written down, taught and learned". One of these is "curing a disease".

I find that questionable. Have you ever watched House? Apart from being a very entertaining update of Sherlock Holmes, what goes on in the early seasons of House is (within the limits of television drama, to the eyes of this layman) pretty close to the actual investigations and diagnoses I have read in The Medical Detectives,  The Man Who Mistook His Wife for a Hat and a few other of the bits of source material. To a surprising extent, they aren't making that stuff up. See how the diagnostic team blunder about? See how they follow false leads? See how some apparently unrelated fact combines in the prepared mind with a huge mass of prior experience to prompt new ideas? See how they must take a chance on something that they don't know will work?

There are definite tools that are used again and again, primarily differential diagnosis (which, BTW, makes the average episode of House into a good short course on debugging), and the various physiological tests and so forth, but mostly it's a synthesis of knowledge and experience. If you ever get the chance to hear Brian Marick tell his story about how vets learn to recognize "bright" vs "dull" cows you'll get a good idea of just how the teaching of medicine works. And it works to a large extent by the inculcation of tacit knowledge.

Tacit knowledge is terribly hard stuff to deal with, it's in the "unknown knowns" quadrant. One of the things that makes teaching a skill difficult is the manifestation of tacit knowledge*.  A bit part of what a lot of people want a "process" for programming to do is to make explicit exactly that tacit knowledge. 

But there is only one way to do that really well, which is exactly why things like teaching hospitals exist. The way to do it is to try repeatedly with feedback from an expert. The external feedback is crucial. You may have heard somewhere that practice makes perfect. In fact, all practice does by itself is make consistent. It's the expert feedback that leads to improvement. 

This one reason why folks are unhappy with what we do know about how to program: it cannot be learned quickly from a book, from a one-week course. There is another reason, perhaps worse: what we know about programming does not (and if parallel with other disciplines based on skill and judgment, it never will) equip programmers to produce a given outcome on demand, on time, every time.

The idea has got around (originating perhaps with "One Best Way" Taylor) that a process for any given job should be like a recipe—follow it exactly and get the same result every time. In some spheres of work such things are possible. Programming does not seem to be one of them, no matter how much we might wish it.

* Design (and other kinds) of patterns are in part about capturing this stuff, which they do variously well. I suspect that this is part of the reason why more experienced folks who haven't seen them before say things like "that's obvious" or "anyone who doesn't know/couldn't work that out shouldn't be a programmer". 

Doing Design

In an extract from the new edition ("2.0", indeed) of Software Conflict Robert L. Glass revisits his prior discussion of "design" in software. He focuses on the undoubted fact that design is a cognitive process. It's perhaps a sad reflection on the state of the industry at the time he was writing that this should have been a noteworthy observation.

Where to begin? How to proceed?

Anyway, it seems that Glass undertook a commendable self-assessment of his stance on design, found it wanting and turned to the research of Bill Curtis and Elliot Soloway. As reported by Glass, after a series of "protocal studies" came up with roughly this description of what programmers do when they design:

1. Construct a mental model of a proposed solution to the problem
2. Mentally execute the model to see if it solves the problem
3. It doesn't, so see which parts of the problem the model fails on and enhance it
4. Repeat until good enough

Strangely, Glass calls this a "well defined process" and looks to the implications of this "new concept of design as a mental process."

By the way, Google wants "protocal" to be a typo for "protocol" but apparently it isn't. "Protocal" seems to mean "talking about what you're doing while you do it."

The problem with the design process as described above is that it has by itself nothing to say about where these mental models come from, or how they are compared or improved. Glass quotes Charles Simonyi on this, thus:

The first step in programming is imagining. Just making it crystal clear in my mind what is going to happen.

Which all, as a description of the design "process", seems only little more useful than the Feynman Problem Solving algorithm (*). Especially since one of Glass's goals in studying this matter was to find a better way to teach design.

Tools for the mind?

In the 2006 update to this essay Glass reports on the outcome of the Curtis/Soloway research: nothing. Apparently, they gave up. Glass states:

Because the nature of design, they discovered, is so intensely cognitive, happening inside the mind at mind speed, the researchers could conceive of no useful tools to help in that process!

What!? No useful tools to help with a process "inside the mind". Come now! I'm not sure I believe that this is what two smart guys like that concluded, unless by tool they meant some clanking monstrosity that would, you know, run on a computer.

Well, programmers aren't the only people who do design, nor who introspect upon their own design activity. In 1964 (about 20 years before the research Glass is talking about) Christopher Alexander produced his doctoral thesis, Notes on the Synthesis of Form, which describes how the "imagining" takes place within a culture and a tradition (the design patterns movement inspired by Alexander seeks to make these things explicit for the society of software builders, and others) in both self-conscious and un-selfconcious modes, but more importantly explains how to traverse the loop from less good to more so. Laurent Bossavit gives a good overview of these ideas. Alexander even went so far as to write some computer programs to help architects and planners use these ideas. That would be a "tool". Oh dear.

As Prof. Sir Tony Hoare has put it

If only we could learn the right lessons from the successes of the past, we would not need to learn from our failures.

(*) The Feynman Problem Solving Algorithm

1. Write down the problem
2. Think very hard
3. Write down the solution

Test-first and flow

Frank Sommers has proposed a conflict between test-first and "flow". To answer the question he asks at the end of the article "Do you really write your tests first?" I find it easiest to quote Jason Yip from the ensuing discussion on artima: "Yes I do... except when I don't and I pay for it every time..." These days I even think differently about the two activities: test-first is programming. Anything else may involve the creation of code, but really is exploratory noodling about--which has its value but the results should be discarded.

I wonder where Frank learned his test-first? I suspect that he hasn't ever sat down and paired for a significant time with a master of the art. It is a skill and can't be picked up just from reading about it and having a bit of a go. Also, test-first is only a transitional practice to test-driven, which is where the big wins come from.

In any case, I find Frank's appeal to the theory of flow to explain why he doesn't enjoy test-first programming rather bizarre. He quotes Csikszentmihalyi on some of the characteristics of flow:

• The activity must present just the right amount of challenge: If the activity is too hard, we become frustrated; if too easy, boredom ensues.

• There must be clear and unambiguous goals in mind.

• There needs to be clear-cut and immediate feedback about the activity's success.

• Finally, our focus during the activity must center on the present - the activity itself.

And suggests that they are in conflict somehow with the practice of test-first. I find the exact opposite, and I'm not alone in that. I'm going to talk more about TDD than merely test-first, because I don't practice the one without the other any more.

I find that test-driven development is the way that I turn programming into an activity with the right amount of challenge. I'm just not smart enough to program any other way. I did somehow manage to get paid to program in the days before I adopted TDD, but I no longer understand how that can possibly have happened. YMMV.

In writing a TDD test I am explicitly setting an clear an unabiguous goal for the next few minutes programming.

The red bar -> green bar -> refactor cycle provides very clear-cut and very immediate feedback on my programming activity.

And, as Frank rightly points out, test-first works best on a very small scale. In fact, TDD encourages you to program in very small steps (and, perhaps counterintuitively, the smaller the steps the faster you make progress). So I find that writing the tests first encourages me to center on the present.

Fascinating that two people can have such utterly opposed experience with the same technique.

Agile film-making?

The possibility of parallels between these disciplines came
up on the XP egroup recently, which remineded me of this wiki posting I made a few years ago. Seems the idea is gaining
more currency.

Agility, Architecture and Scale

Is "scale later" a fatal mistake? And further, a symptom of the indiscipline and sloppy software development that some folks hide behind claims of Agility, according to Aditya.

Well, Aditya is commenting upon this advice from 37signals:

In the beginning, make building a solid core product your priority instead of obsessing over scalability and server farms. Create a great app and then worry about what to do once it's wildly successful. Otherwise you may waste energy, time, and money fixating on something that never even happens. [emphasis in original]

That's sound Agile and/or Lean advice. Aditya counters with a reference to this piece by Werner Vogels, CTO of Amazon.com Clearly, Werner knows a thing or two about large systems. And clearly, any new functionality that his teams implement must be capable of handling the sort of load that Amazon's customer base generates, within Amazon's large scale, high performance, highavailability system. But Amazon have been in business for over a decade as I write this, and that's not the scenario that 37signals are talking about. They are talking about what to do when you're in the first year of your new web-based business and you need to get a product in front of paying customers right now, or else there isn't going to be any scalability problem ten years down the line. Or even two.

Pragmatic?

37signals is a Ruby (on Rails) house, so they're joined up with the PragmaticProgramming movement. In The Pragmatic Programmer Hunt and Thomas present some techniques that will lead to a system that will have some degree of scalability:

• tip 13 Eliminate effects between unrelated things
• tip 41 Always design for concurrency
• tip 45 Estimate the order of your algorithms
• tip 46 Test your estimates

These essentialy architectural practices have other drivers, and other benefits, but they will also introduce into your system the points of flexibility and cleavage planes through into which mechanisms for improving scalability maybe inserted. Inserted later.

Now, the pragmatists hardly look as if they are indisciplined and sloppy, if they are following those tips. But there are always those other guys. For instance, while many fine developers would agree that often optimization is your worst enemy, there'll always be someone to come along with a counter story of a time when the optimization really should have been done. Similarly, the Extreme Programming advice that You Aren't Gonna Need It, and Do the Simplest Thing that Could Possibly Work are countered with stories about times when folks think that using them would have failed. Furthermore, folks claim that these practices are used by bad programmers to justify, well, being indisciplined and sloppy (usually because XP doesn't explicitly metion whatever thechnique their most recent book was about). In my experience it requires a great deal of discipline on the part of the typical programmer to apply YAGNI and DtSTtCPW, so action oriented, operationally biased and generally in love with doing the next cool, clever thing are they.

In isolation, out of context, these practices can be damaging. But one remedy is Worst Thing First. Notice the examples that Ron gives:

In the PlanningGame, users assign priority to stories, among other reasons, because of risks they perceive. "System must be able to pay 60,000 people in two hours. High priority." Developers assign technical risk similarly. "System must be able to marshal up to 60,000 networked PCs simultaneously for 1 minute each. High risk." We sit together and build consensus on what is risky, what needs a SpikeSolution, what order to do things. Is there a better way to identify worst things than to get together and work on it?

Those are performance and scale requirements. Being addressed up front. Because the customer wants them, so you really are going to need them.

Abstract Scalability

But that's scale, not scalability. Werner Vogels defines scalability this way:

A service is said to be scalable if when we increase the resources in a system, it results in increased performance in a manner proportional to resources added

This is an abstract quality referring to a hypothetical situation. A terribly difficult thing to do engineering about. When might you need to feel confident about this scalability? Well, if you were running an web 1.0 startup and your business model relies on capturing a dominant slice of the global market for your particular genius product in very short order, you might. You might very well need to be seen (by your VCs) spending money on the big iron needed to handle the load that would have to come to earn the revenue that would show them any kind of return.

On the other hand, if you know that the first day of the rollout of your new product it's going to get banged on by all n tens of thousands of employees of Yoyodyne worldwide, then you'd better be sure that it will operate at that scale. And if you're Werner Vogels and you expect your historical growth in user base to continue at whatever rate it is, you have a definite idea of what extra capacity you'll have to accommodate and when. But abstract scalability? Nah.

The key to the message from 37signals, and from XP is this: how far do you think Google would have got if the first thing Larry and Sergey had done in 1996 was to think "you know, this could get huge, so first up we should design a filesystem to handle huge data volumes across many machines and then devise an algorithm for distributing certain classes of problems across huge clusters" instead of building the search engine itself?

Sustainable Pace Supported

At last, pointers to hard number studies in support of the XP practice of "Sustainable Pace".

Once again, other industries are far ahead of software development in the depth and maturity of their thinking about work. This pull quote sums up the article nicely:

A hundred years of industrial research has proven beyond question that exhausted workers create errors that blow schedules, destroy equipment, create cost overruns, erode product quality, and threaten the bottom line. They are a danger to their projects, their managers, their employers, each other, and themselves. Any way you look at it, Crunch Mode used as a long-term strategy is economically indefensible. Longer hours do not increase output except in the short term. Crunch does not make the product ship sooner--it makes the product ready later . Crunch does not make the product better--it makes the product worse

Now, there have been times when I, as a development manager, have pushed back against pressure from my management to suggest to my reports that they enter "crunch mode" for an extended period (oh, say, the lasquarterer of disappointingng year). Once, I put together a spreadsheet that, although I didn't know it at the time, embodiesomethingng similar to Chapman's model, as mentioned by Evan. The covering email that announced this spreadsheet explained that I intended it to be used for scenario modelling: think up some scheme for deploying overtime, plug it into the model, see what improvement (or, more likely, decline) in productivity would come about. Or better still goal seek to the improvement desired, see what the model had to say about the assumptions that would have to be true for that improvement to materialise, and then judge if those assumptions were reasonable.

Those of you reading this who have any substantial experience in the software industry will not be surprised to learn that I nearly lost my job as a result.

And Evan addresses this:

Managers decide to crunch because they want to be able to tell their bosses "I did everything I could." They crunch because they value the butts in the chairs more than the brains creating games. They crunch because they haven't really thought about the job being done or the people doing it. They crunch because they have learned only the importance of appearing to do their best to instead of really of doing their best. And they crunch because, back when they were programmers or artists or testers or assistant producers or associate producers, that was the way they were taught to get things done.

There is a circle of abuse here, (real abuse; lives relationships and families are damaged), that must be broken if the software industry is to flourish.