Red and Green

Brian Di Croce sets out here to explain TDD in three index cards. It's a nice job, except that I'm not convinced by the faces on his last card.

Brian shows a sad face for the red bar, a happy face for the green, and a frankly delirious face for the refactoring step. There's something subtly wrong with this. We are told that when the bar is green the code is clean, which is great. But the rule is that we only add code to make a failing test pass, which implies a red bar. So, the red bar is our friend!

When I teach TDD I teach that green bar time is something to get away form as soon as possible. Almost all the time a green bar occurs when a development episode is incomplete: not enough tests have been written for the functionality in hand, more functionality is expected to go into the next release, or some other completeness condition is not met. 

It's hard to learn form a green bar,  but a red bar almost always teaches you something. Experienced TDDer's are very (and rightly) suspicious of a green-to-green transition. The green bar gives a false sense of security.

Generally speaking, in order to get paid we need to move an implementation forward, and that can only be done on a red bar. Wanting to get to the next red bar is a driver for exploring the functionality and the examples that will capture it. 

I tell people to welcome, to embrace, to seek out the red bar. And that when the bar is red, we're forging ahead.

TDD at QCon

Just finished a presentation of the current state of my TDD metrics thinking at QCon. The slides [pdf] are up on the conference site, video should be there soon, too.

Tests and Gauges

At the recent Unicom conference re Agility and business value I presented on a couple of cases where I'd seen teams get real, tangible, valuable...value from adopting user/acceptance/checked-example type testing. David Peterson was also presenting. He's the creator of Concordion, an alternative to Fit/Library/nesse. Now, David had a look at some of my examples and didn't like them very much. In fact, they seemed to be the sort of thing that he found that Fit/etc encouraged, of which he disapproved and to avoid which he created Concordion in the first place. Fair enough. We tossed this back and forth for a while, and I came to an interesting realization. I would in fact absolutely agree with David's critique of the Fit tests that I was exhibiting, if I though that they were for the purpose that David thinks his Concordion tests are for. Which probably means that any given project should probably have both. But I don't think that, so I don't.

Turning the Tables

David contends that Fit's table–oriented approach affords large tests, with lots of information in each test. He's right.  I like the tables because most of the automated testing gigs that I've done have involved financial trading systems and the users of those eat, drink, and breath spreadsheets. I love that I can write a fixture that will directly parse rows off a spreadsheet built by a real trader to show the sort of thing that they mean when they say that the proposed system should blah blah blah. The issue that David sees is that these rows probably contain information that is, variously: redundant, duplicated, irrelevant, obfuscatory and various other epithets. He's right, often they do. 

What David seems to want is a larger number of smaller, simpler tests. I don't immediately agree that more, simpler things to deal with all together is easier than fewer, more complex things, but that's another story. And these smaller, simpler tests would have the principle virtue that they more nearly capture a single functional dependency. That's a good thing to have. These tests would capture all and only the information required to exercise the function being tested for. This would indeed be an excellent starting point for implementation. 

There's only one problem: such tests are further away from the users' world and close to the programmers' . All that stuff about duplication and redundancy is programmer's talk. And that's fair enough. And its not enough. I see David's style of test as somewhat intermediate between unit tests and what I want, which is executable examples in the users' language. When constructing these small, focussed tests we're already doing abstraction, and I don't want to make my users do that. Not just yet, anyway.

So then I realised where the real disagreement was. The big, cumbersome, Fit style tests are very  likely too complicated and involved to be a good starting point for development. And I don't want them to be that. If they are, as I've suggested, gauges, then they serve only to tell the developers whether or not their users' goals have been met. The understanding of the domain required to write the code will, can (should?) come from elsewhere. 

Suck it and See

And this is how gauges are used in fabrication. You don't work anything out from a gauges. What you do is apply it to see if the workpiece is within tolerance or not. And then you trim a bit off, or build a bit up, or bend it a bit more, or whatever, a re–apply the gauge. And repeat. And it doesn't really matter how complicated an object the gauge itself is (or how hard it was to make—and it's really hard to make good gauges), because it is used as if it were both atomic and a given. It's also made once, and used again and again and again and again...

Until this very illuminating conversation with David I hadn't really fully realised myself quite the full implications of the gauge metaphor. It actually implies something potentially quite deep about how these artifacts are built, used and managed. Something I need to think about some more.

Oh, and when (as we should) we start to produce  exactly those finer–grained, simpler, more focussed tests that David rightly promotes, and we find out that the users' understanding of their world is all denormalised and stuff, what interesting conversations we can have with them then about how their world really works, it turns out. 

Might even uncover the odd onion in the varnish. But let's not forget that having the varnish (with onion) is more valuable to them than getting rid of the onion.

Curiously Apt

A friend is embarking upon a conversion MSc into the wonderful world of software development. He's become interested in the currently en vogue paradigms of programming, their relationships and future. It seems to him (he says) that OO is very much about standing around a whiteboard with your friends, sipping a tall skinny latte while your pizza goes cold. And by contrast  functional programming is like sitting alone, crying into your gin with your head held in your hands over some very, very, hard maths. 

Perceptive. He'll go far. And not just because he's already had this other successful career dealing with actual customers (which should be a stronger prerequisite for becoming a commercial developer than any of that comp sci stuff).

Compiler Warnings

This Daily WTF reminded me of a less than glorious episode from my programming past. At a company that shall remain namless (and doesn't exist anymore) I was working on a C++ library to form part of a much larger application.

One fine day a colleague came stomping (that's the only verb that suits) over to my desk and boldly announced that "your code has crashed my compiler".  Somewhat alarmed at this I scurried (yes, scurried) back to his desk. "Look" he said, and did a little cvs/make dance. Lo and behold, when the compiler got to my code indeed it fell to silently chugging away. "See," he resumed, "it's crashed." 

"No," I said, "it's just compiling." 

"So where," he asked, "are the messages?"

"What messages?" I replied. He scrolled the xterm up a bit.

"These. All the warnings and stuff"

"Doesn't generate any," I said.

He boggled. "What, have you found some way of turning them off?"

"No," I said, "I just wrote the code so that it doesn't generate any warnings."

He boggled some more. "Why," he eventually managed to gasp, "would you bother doing that?"

I didn't last long there. It was best for all concerend, really.

Golden Hammers vs Silver Bullets?

The old wiki page on Golden Hammers has popped up on reddit. One commentator suggests that a golden hammer seems to be the same as a silver bullet.

Probably. But I find the two phrases suggestive when placed side-by-side. To me, it seems as if a golden hammer is a tool that's very familiar, simple to apply, and words best on things within striking distance: everyday problems. All these screws sticking out all over, bam, bam, bam. A golden hammer is something we already have, and it worked great lots of time before, so lets carry on using it for whatever comes next.

A silver bullet, on the other hand, seems like something that is directed in from the outside. It is a projectile, the sender has no control over it once launched. The silver bullet is new and alien. It requires complex tooling to make it work. It is deployed against hairy monsters that jump out at you. Nothing ever worked on them before, maybe the silver bullet will?

A golden hammer, then, would be the over-applied old tool of a worker who doesn't learn new ones. The silver bullet is the disengaged outsider's agent of violent change. 

I love having new distinctions to play with.

Agile 2008

Just a reminder that the call for submissions to the Agile 2008 conference closes on the 25th for Feb. I strongly encourage those who are interested in agile methods to go, and I strongly encourage those who are going to submit sessions. 

I'm assistant producer (vice Steve Freeman) of the "stage" called Committing to Quality, which is a home for sessions concerned with the strong focus on internal and external quality in agile development.

I've been watching the submission system since the call opened and there are some really interesting sessions being proposed. Looks as if it's going to be a good conference.

See you there!

UK Conference Season

It's getting round to be conferece season again. If you happen to be in the UK in the next few months, maybe I'll see you at UNICOM, where I'll be talking about some adventures with automated testing.

Or perhaps at QCon, where I'll be presenting the latest news on my metrics work and joining in a panel with Beck and others, both part of the XPDay Taster track, a cross-over from the XPDay events.

Or even at Spa (that oh so magical automated testing again).

Tolstoy's Advice on Methodology

So, Tolstoy tells us that all happy families are all the same, whereas each unhappy family is unhappy in its own way. I think that the same applies to development projects: all successful projects are the same, unsuccessful ones fail in their own ways.

This occurred to me while chatting with one of my Swiss colleagues recently. The Swiss end of the business does a lot of successful RUP engagements: that's right, successful RUP. The reason that they are successful is twofold. Firstly, they always do a development case so they always carefully pick and choose which roles, disciple and the rest they will or will not use on a project. Secondly, they understand that RUP projects really are supposed to be really iterative and really incremental, that almost all the disciplines go on to a greater or lesser extent in all phases. A (different) Swiss colleague once asked me what the difference was between Agile and RUP. My only semi-flippant answer was that if you do RUP the way Philippe Kruchten wants you to do it, then not much.

Another take on DSLs, or "Why Ruby doesn't quite do it for me"

If you liked that, then you might also like this.

Research, huh! What is it good for?

...absolutely nothing, according to this screed. Yes, I know it's half a year old or so, but I only just came across it. Which is a nice segue, because it (and that I felt the need to make that qualification) partly illustrates something that annoys me not a little about the industry: a short memory. Also, not looking stuff up. While the overall message (that contemporary Comp Sci research is a titanic fraud that should be abolished) is both shrill and apparently right-libertarian propaganda, I have a degree of sympathy with it.

We are asked to consider those mighty figures of the past, working at institutions like the MIT AI Lab, producing "fundamental architectural components of code which everyone on earth points their CPU at a zillion times a day". OK. Let's consider them. And it's admitted that "some of [them] - perhaps even most - were, at some time in their long and productive careers, funded by various grants or other sweetheart deals with the State." No kidding.

Go take a look at, for example, the Lambda the Ultimate papers, a fine product of MIT. Now, MIT is a wealthy, independent, private university. So who paid for Steele and the rest to gift the world with the profound work of art that is Scheme? AI Memo 349 reports that the work was funded in part by an Office of Naval Research contract. The US Navy wanted "An Interpreter for Extended Lambda Calculus"? Not exactly. In 1975 "AI" meant something rather different and grander than it does today. And it was largely a government-funded exercise. This Google talk gives a compelling sketch of the way that The Valley is directly a product of the military-industrial complex, that is interventionist government funding. Still today, the military are a huge funder of research, and buyer of software development effort and hardware to run the results upon, which (rather indirectly) pushes vast wodges of public cash into the hands of technology firms. Or even directly: Bell Labs, for instance, received direct public funding in the form of US government contracts.

In the UK, at least, the government agencies that pay for academic research (of all kinds) are beginning to wonder, in quite a serious, budget-slashing kind of way, if they're getting value for money. So, naturally, the research community is doing some research to find out. One reason that this is of interest to me is that my boss (or, my boss's boss anyway) did some of this research. Sorry that those papers are behind a pay gate. I happen to be a member of the ACM, so I've read this one and one of the things it says is that as of 2000 the leading SCM tools generated revenues of nearly a billion dollars for their vendors. And where did the ideas in those valuable products come from? They came, largely, from research projects. What the Impact Project is seeking to do is to identify how ideas from research feed forward into industrial practice, and they are doing this by tracing features of current systems back to their sources. Let's take SCM.

The observation is made that the diff [postscript] algorithm (upon which, well, diffing and merging rely) is a product of the research community. From 1976. With subsequent significant advances made (and published in research papers) in 1984, '91, '95 and '96. Other research ideas (such as using SCMs to enforce development processes) didn't make a significant impact in industry.

Part of the goal of Impact is to:

educate the software engineering research community, the software practitioner community, other scientific communities, and both private and public funding sources about the return on investment from past software engineering research [and] project key future opportunities and directions for software engineering research and practice [and so] help to identify the research modalities that were relatively more successful than others.

In other words, find out how to do more of the stuff that's more likely to turn out to be valuable. The bad news is that it seems to be hard to tell what those are going to be.

I focus a little on the SCM research because that original blog post that got me going claims that

most creative programming these days comes from free-software programmers working in their spare time. For example, the most interesting new software projects I know of are revision control systems, such as darcs, Monotone, Mercurial, etc. As far as Washington knows, this problem doesn't even exist. And yet the field has developed wonderfully.

I would be very astonished to find that a contemporary (I write in early 2008–yes, 2008 and I still don't have a flying car!) update of that SCM study would conclude that the distributed version control systems were invented out of thin air in entirely independent acts of creation. They do mention that the SCM vendors, when asked, tended to claim this of their products).

The creation of Mercurial was a response to the commercialization of BitKeeper, and BitKeeper would seem to have been inspired by/based on TeamWare. Those seem to have all been development efforts hosted inside corporations, which is cool. I'd be interested to learn that McVoy at no time read any papers or met any researchers that talked about anything that resembled some sort of version control that was kind-of spread around the place. The Mercurial and and bazaar sites both cite this fascinating paper [pdf] which cites this posting. Which tells us that McVoy's approach to DVCS grew out work at Sun (TeamWare) done to keep multiple SCCS repositories in sync. Something that surely more people that McVoy wanted to do. SCCS was developed at Bell Labs (and written up in this paper [pdf], in IEEE Transactions in 1975)

One of the learnings from Impact is that what look like novel ideas from a distance in general turn out, upon closer inspection, to have emerged from a general cloud of research ideas that were knocking around at the time. The techniques used in the Impact studies have developed, and this phenomenon is much more clearly captured in the later papers. So what does that tell us?

Well, it tells us that its terribly hard to know where ideas came from, once you have them. And that makes it terribly hard to guess well what ideas are going to grow out of whatever's going on now. So perhaps there isn't a better way than to generate lots of solutions, throw them around the place and see waht few of them stick to a problem. Which is going to be expensive and inefficient–upsetting for free-marketeers, but then perhaps research should be about what we need, not merely what we want (which the market can provide just fine, however disastrous that often turns out to be). Anyway, once I heard somewhere that you can't always get what you want.

Back to that original, provocative, blog posting. It's claimed that, as far as the problems that the recent crop of DVCS systems address that "As far as Washington knows, this problem doesn't even exist." Applying a couple of levels of synecdoche and treating "Washington" as "the global apparatus of directly and indirectly publicly-funded research", the it would perhaps be better to say that "Washington thinks that it already payed for this problem to be solved decades ago". Washington might be mistaken to think that, but it's a rather different message.

Rubbing our noses in it this time

How hard can it be? What I want (and I know I'm not alone in this) is a 12" (ok, these days 13" if you must) MacBook Pro. With an optical drive built in. And user upgradeable RAM and hard-drive. And battery. With a pukka graphics card.

Such a machine would already be as thin and as light as I care about, and as capable as I want. But I can't have one, apparently. I could have this new "air" frippery–and doesn't it photograph well? Don't you love they way that in that 3/4 overhead shot you can't see the bulge underneath and it looks even thinner that it actually is? But really, a laptop with a off-board optical drive? Well, maybe that's the future, what with renting movies on iTunes and what have you, but...

Apple (ie, Jonathan Ive) have gotten really good at recycling a forty year old design language for products that pundits often hate and (partly because) the public loves, but this gadget seems a little too far ahead of its time. Ah well.

The Learning Curve

I'm going to let the dust settle on my recent posting on patterns, and then do a follow up—some interesting stuff has come out of it. For that, I think, I need a bit of supporting material some of which is here.

Recently this little gem resurfaced on reddit, which prompted a certain line of investigation. Normally I spread a lot of links around my posts[*] partly as aides memoir for myself, partly because my preferred influencing style is association, and partly because I believe that links are content. But I really want you to go and look at this cartoon, right now. Off you go.

Back again? Good. This "learning curve" metaphor is an interesting one. Before getting into this IT lark I was in training to be a physicist so curves on charts have are strongly suggestive to me. I want them to represent a relationship between quantities that reveals something about an underlying process. I want the derivatives at points and areas under segments to mean something. What might these meanings be in the case of the learning curve?

Most every-day uses of the phrase "learning curve" appeal to a notion of how hard it is to acquire some knowledge or skill. We speak of something difficult having a "steep" learning curve, or a "high" learning curve, or a "long" one. We find the cartoon is funny (to the extent that we do—and you might find that it's in the only-funny-once category) because our experience of learning vi was indeed a bit like running into a brick wall. Learning emacs did indeed feel a little bit like going round in circles, it did indeed seem as if learning Visual Studio was relatively easy but ultimately fruitless.

But where did this idea of a learning curve come from? A little bit of digging reveals that there's only one (family of) learning curve(s), and what it/they represents is the relationship between how well one can perform a task vs how much practice once has had. It is a concept derived from the worlds of the military and manufacturing, so "how well" has a quite specific meaning: it means how consistently. And how consistently we can perform an action is only of interest if we have to perform the action many, many times. Which is what people who work in manufacturing (and, at the time that the original studies were done, the military) do.

And it turns out, in pretty much all the cases that anyone has looked at, that the range of improvement that is possible is huge (thus all learning curves are high), and that the vast majority of the improvement comes from the early minority of repetitions (thus all learning curves are steep). Even at very high repetition counts, tens or hundreds of thousands, further repetitions can produce a marginal improvement in consistency (thus all learning curves are long). This is of great interest to people who plan manufacturing, or other, similar, operations because they can then do a little bit of experimentation to see how many repetitions a worker needs to do to obtain a couple of given levels of consistency. They can then fit a power-law curve trough that data and predict how many repetitions will be needed to obtain another, higher, required level of consistency.

Actual learning curves seem usually to be represented as showing some measure of error, or variation, starting at some high value and then dropping, very quickly at first, as the number of repetitions increases.

Which is great is large numbers of uniform repetitions is how you add value.

But, if we as programmers believe in automating all repetition, what then for the learning curve?

[*] Note: for those of you who don't like this trait, recall that you don't have to follow the links.

The Problem with Problems with Patterns

There seems to be a new venom in the informal blogsphere-wide movement against patterns. I'm not the only one to notice it.

Commentators are making statements like this:

In certain programming cultures, people consider Design Patterns to be a core set of practices that must be used to build software. It isn’t a case of when you need to solve a problem best addressed by a design pattern, then use the pattern and refer to it by name. It’s a case of always use design patterns. If your solution doesn’t naturally conform to patterns, refactor it to patterns.

What Reg originally said was "Most people consider Design Patterns to be a core set of practices that must be used to build software." Most? Really most? I challenged this remarkable claim, and Reg subsequently re-worded it as you see above. Quite who's in this certain culture I don't know. And apparently neither does Reg:

We could argue whether this applies to most, or most Java, or most BigCo, or most overall but not most who are Agile, or most who have heard the phrase "Design Patterns" but have never read the book, or most who have read the book but cannot name any pattern not included in the book, or...

But it seems off topic. So I have modified the phrase to be neutral.

But if that "certain culture" doesn't exist, then what's the problem? And if it does exist, then why is it so hard to pin down?

There are anecdotes:

When I was [...] TA for an intro programming class [...] I was asked to whip up a kind of web browser "shell" in Java. [...] Now, the first language that I learned was Smalltalk, which has historically been relatively design-pattern-free due to blocks and whatnot, and I had only learned Java as an afterthought while in college, so I coded in a slightly unorthodox way, making use of anonymous inner classes (i.e., shitty lambdas), reflection, and the like. I ended up with an extremely loosely coupled design that was extremely easy to extend; just unorthodox.

When I gave it to the prof, his first reaction, on reading the code, was...utter bafflement. He and another TA actually went over what I wrote in an attempt to find and catalog the GoF patterns that I'd used when coding the application. Their conclusion after a fairly thorough review was that my main pattern was, "Code well."

So, maybe there are some teachers of programming (an emphatically different breed from industry practitioners) who over-emphasize patterns. That I can believe. And it even makes sense, because patterns are a tool for knowledge management, and academics are in the business of knowing stuff.

It's where Patterns came from

But what's this? "Smalltalk [...] has historically been relatively design-pattern-free due to blocks and whatnot" That's simply not true. In fact, the idea of using Alexander's pattern concept to capture recurrent software design ideas comes from a group that has a large intersection with the Smalltalk community. There's a Smalltalk-originated pattern called MVC, you may have heard of it being applied here and there. Smalltalk is pattern-free? Where would such confusion come from? I think that the mention of blocks is the key.

That Smalltalk has blocks (and the careful choice of the objects in the image) gives it a certain property, that application programmers can create their own control-flow structures. Smalltalk doesn't have if and while and what have you baked in. Lisps have this same property (although they do tend to have cond and similar baked in) through the mechanism of macros.

Now, this is meant to mean that patterns disappear in such languages because design patterns are understood to be missing language features, and in Lisp and Smalltalk you can add those features in. So there shouldn't be any missing. The ur-text for this sort of thinking is Norvig's Design Patterns in Dynamic Programming.

The trouble is that he only shows that 16 of the 23 GoF patterns are "invisible or simpler" in Lisp because of various (default) Lisp language features. This has somehow been inflated in people's minds to be the equivalent of "there are no patterns in Lisp programs", where "Lisp" is understood these days as a placeholder for "my favourite dynamic language that's not nearly as flexible as Lisp".

But there are

And yet, there are patterns in Lisp. Dick Gabriel explains just how bizarre the world would have to be otherwise:

If there are no such things as patterns in the Lisp world, then there are no statements a master programmer can give as advice for junior programmers, and hence there is no difference at all between an beginning programmer and a master programmer in the Lisp world.

Because that's what patterns are about, they are stories about recurrent judgments, as made by the masters in some domain.

Just in passing, here's a little gem from Lisp. Macros are great, but Common Lisp macros have this property called variable capture, which can cause problems. There are a bunch of things you can do to work around this, On Lisp gives 5. Wait, what? Surely this isn't a bunch of recurrent solutions to a problem in a context? No, that couldn't be. As it happens, it is possible to make it so that macros don't have this problem at all. What you get then is Scheme's hygienic macros, which the Common Lisp community doesn't seem too keen to adopt. Instead, they define variable capture to be a feature.

Now, it's tempting to take these observations about the Lisp world and conclude that the whole "no patterns (are bunk because) in my language" thread is arrant nonsense and proceed on our way. But I think that this would be to make the mistake that the anti-patterns folk are making, which is to treat patterns as a primarily technical and not social phenomenon.

Actually, a little bit too social for my taste. I've been to a EuroPLoP and work-shopped (organizational) patterns of my own co-discovery, and it'll be a good while before I go to another one–rather too much of this sort of thing for my liking. But that's my problem, it seems to work for the regulars and more power to them.

Contempt

Look again at Reg's non-definition of the "certain programming cultures":

[...] most, or most Java, or most BigCo, or most overall but not most who are Agile, or most who have heard the phrase "Design Patterns" but have never read the book, or most who have read the book but cannot name any pattern not included in the book,[...]

doesn't it seem a little bit as if these pattern-deluded folks are mainly "programmers I don't respect"?

This is made pretty explicit in another high-profile pattern basher's writing

The other seminal industry book in software design was Design Patterns, which left a mark the width of a two-by-four on the faces of every programmer in the world, assuming the world contains only Java and C++ programmers, which they often do. [...] The only people who routinely get excited about Design Patterns are programmers, and only programmers who use certain languages. Perl programmers were, by and large, not very impressed with Design Patterns. However, Java programmers misattributed this; they concluded that Perl programmers must be slovenly [characters in a strained metaphor].

Just in case you're in any doubt, he adds

I'll bet that by now you're just as glad as I am that we're not talking to Java programmers right now! Now that I've demonstrated one way (of many) in which they're utterly irrational, it should be pretty clear that their response isn't likely to be a rational one.

No, of course not.

A guy who works for Google once asked me why I thought it should be that, of all the Googlers who blog, Steve has the highest profile. It's probably got something to do with him being the only one who regularly calls a large segment of the industry idiots to their face.

And now don't we get down to it? If you use patterns in your work, then you must be an idiot. Not only are you working on one of those weak languages, but you don't even know enough to know that GoF is only a bunch of work-arounds for C++'s failings, right? I think Dick Gabriel's comment goes to the heart of it. If patterns are the master programmers advice to the beginning programmer, codified, then to use patterns is to admit to being a beginner. And who'd want to do that?

Compromise

So, Allan Kelly has picked up on something that I kind-of pretty much did say at XpDay this year—but not quite. I certainly am "part of the UK Agile set", especially if we emphasize the "UK" bit. What I'm not is part of the gang of independent consultants in London who are many of the high-profile early adopters in the UK, and it's true that I never worked for Connextra or any of the other highly-publicized, London-based "Agile" shops. Why is this worthy of note? Only because it relates to a sign of un-wellness in the community that I perceived at the session where Allan heard me say that. It was a double-header, a somewhat strange work-shoppy type of thing, followed by a panel. It was while speaking on this panel that I said what Allan noticed.

The point that I was, perhaps unsuccessfully, trying to make was that the disenchantment with the current state of the Agile world is perhaps more to do with the geographically constrained, hot-house world of ex-Connextra, ex-another place, folks circulating around the same bunch of clients in London than with anything wrong with the community at large.

In particular, this idea that any "mojo" has been lost, or that by accommodating corporate culture in amongst Agile adoption some "compromise too far" has occurred seems very off base to me. We pretty much agreed on the panel that the question "Have you compromised your Agility?" is a silly one, since "Agile" is primarily a marketing buzzword and it labels what is pretty much a bag of platitudes. How can we tell that these are platitudes? Well, they don't help us choose between reasonable alternatives: the hardest-core hard-core SSADM, PRINCE-2 wonk around, if asked in those terms, would probably tell you that working software is more valuable than comprehensive documentation (and probably add "you idiot" under their breath). They might not necessarily behave in a way aligned with that judgment mid project, that that's a whole other story. Fretting about whether or not you've compromised a platitude doesn't seem like the way forward.

There was a lot of talk about "Excellence", too.

And all of this (along with "wither the Agile Alliance?" and "what can we do about Scrum?") may seem like a topic of crucial import, if you spend a lot of your time inside the echo-chamber. Are we Agile? No, I mean, are we really Agile? Truly? Are we Excellent? Is this the real, true, expression of the Principles? I've seen long-lived Agile teams tie themselves in knots (and tear themselves apart, and get—rightly—caned by their business sponsors) over this sort of self-absorbed stuff.

Now, I have in mind a team I know that adopted some (not all) Agile principles. Are they purely Agile? No. Are they living the dream? No. Did they have to make compromises between textbook Agile and their corporate culture to get where they are? Yes. Might they do better if they hadn't to? Yes. But...

Are they three times more productive than they were before? Yes! Is the internal and external quality of their system hugely greater than before? Yes! Do their management revel in being able (for the first time ever) to believe, trust and make intelligent decision based on their status reports? Yes!

I am most reluctant to embrace, in fact I absolutely repudiate, a model of what it means to be Agile that demands that I call them failures because their (ongoing) improvement required compromise with their corporate culture. I'm not terribly interested, these days, in fine-tuning some on-going Agile adventure. I'm interested in taking what are now increasingly well-tried techniques and using them to get the big wins, the big increases in quality and productivity that so many shops are crying out for.

Architecture is contextual, but don't play dumb

A great post from Jacob on the contextual nature of architectural choices. Shock news: he doesn't think that dependency injection frameworks are right for him and his business.

He makes a good case for in-house development being the place where YAGNI is the primary principle, at least so far as the choice to spend effort (= money) on technology choices like that. I sympathize: I've seen in-house teams (with a declared adoption of XP, no less) get into huge rows with their business sponsors because their devotion to a certain architectural stance cost them productivity. They burned up man-decades on trying to get the damn thing to work, ended up with an incomprehensible codebase and then, because it was the right thing, started all over again, only with a determination to get it right this time. IIRC, they eventually ended up implementing this framework three times. Productivity went down and down. The "coach" and "customer" couldn't be in the same room at the same time. So sad. Key to this falling-out was the idea that the (paying) customer isn't entitled to an opinion about how the work is done.

Meanwhile, a side discussion at one XPDay session this year touched on the question of taking into account a need that the customer has told you they have, but that they haven't given a high enough priority to for it to be in the current release. It would be wrong, according to one understanding of YAGNI, to implement this release in a way sympathetic to that upcoming goal. But why? If the customer has told you that some need is coming (soon, that is, as in "next release", and not "in the next five years"), and this lets you make a smarter choice about implementing the nearer-term goals...well, it would be foolish to ignore that information, wouldn't it?

YAGNI means "don't spend six months building a framework to support six days development", it doesn't mean ignore highly certain knowledge about the immediate future direction of the system.

Erlang for Finance

So, the forthcoming "Hardcore Erlang" book just got a lot more interesting.

The example application (which, apparently, will really get built) will now be the core of a stock exchange. This is very exciting to me. Recently I made a career change from spending most of my time working with telecoms systems to spending most of my time with financial systems. And I have been flabbergasted at the time, expense and painful tears of frustration that go into building financial systems, particularly trading systems, given that their basic function is so simple. A lot of this seems to be because the "enterprise" technology stacks turn out (oh the irony) to be dramatically ill-suited to the task.

These systems receive messages and emit messages, perhaps in different formats, perhaps onto a different transport. They do various kinds of fairly simple processing on messages, typically an "enrichment" of a message with some extra data looked up from somewhere. They filter out some messages. And so on. They are, in point of fact, very very similar to switches, bridges, routers and firewalls. They even have similar soft real-time, high availability, high throughput characteristics.

Over in the telecoms world, for some strange reason they don't build routers out of J2EE stacks. They don't handle voice packets with XSLT. Maybe there's a lesson there for someone. And maybe having a stock exchange (or something like out) out there written in the kind of technology that routers are written in might help unwedge a few conversations. I live in hope.

Exemplary Thoughts

So, I was asked to write up the "lighting talk" on examples and exemplars I gave at Agile 2007. That was a short and largely impromptu talk, so there is some extra material here.

Trees

It used to be that botanists thought that the Sugar Maple, the Sycamore and the Plane trees were closely related. These days they are of the opinion that the Sycamore and Sugar Maple are closely related to one another, but the Plane is not to either. This is one example of the way that our idea of how we organise the world can change.

As it happens, this confusion is encoded in the binomial names of these species: the Sugar Maple is Acer saccarum, the (London) Plane tree is Platanus x acerifolia, while the (European) Sycamore is Acer pseudoplatanus. Oh, and if you are a North American then you call your Plane trees "Sycamores" anyway. And further more, not one of these trees is the true Sycamore: that's a fig, Ficus sycomorus.

Botanists and zoölogists are the masters of classification, but as we see they have to modify their ideas from time to time (and these days the rise of cladistics is turning the whole enterprise inside out)

Greeks

A well-known dead Greek laid the foundations of our study of classification about two and a half thousand years ago, in terms of what can be predicated of a thing. It all seemed perfectly reasonable, and was the basis of ontological and taxonomic thinking for many centuries. This is interesting to us who build systems, because the predicates that are (jointly) sufficient and (individually) necessary for an thing to be a member of a category in Aristotle's scheme can be nicely reinterpreted as, say, the properties of a class in an OO language, or the attributes of an entity in an E-R model, and so forth. All very tidy. One small problem arises, however: this isn't actually how people put things into categories! It also has a terrible failure mode: as we can see from all this "acerifolia" and "pseudoplatanus" stuff in the tree's names, the shape of the leaves was not a good choice of shared characteristic to use to classify them. It is of this mistake (amongst other reasons) that the unspeakable pain of schema evolution arises.

The Greeks, by the way, already knew that there were difficulties with definitions. After much straining between the ears, an almost certainly apocryphal story goes, the school of Plato decided that the category of "man" (in the broadest sense, perhaps even including women and slaves) was composed of all those featherless bipeds. Diogenes of Sinope ("the cynic") promptly presented the Academicians with a plucked chicken. At which they refined their definition of Man to be featherless bipeds with flat nails and not claws.

In 1973 Eleanor Rosch published the results of some actual experiments into how people really categorise things, which seem to show that the members of a category are not equal (as they are in the Aristotle's scheme), a small number of them are dominant: Rosch calls these the "prototypes" of the category. And what these prototypes are (and therefore what categories you recognise in the world) is intimately tied in with your experience of being in the world. And these ideas have been developed in various directions since.

One implication of the non-uniformness of of categories is that they are fuzzy, and that they overlap. The import for us in building systems is that maybe the reason that people have difficulty in writing down all these hard-and-fast rules about hard-edged, homogeneous categories of thing as many requirements elicitation techniques want is because that's just not a good fit for how they think about the world, really.

Germans

But perhaps examples do. Examples can be extracted from a person's usual existential setting which means that they can be more ready-at-hand than present-at-hand. This is probably good for requirements and specifications (it's not universally good: retrospectives force a process in which one if usually immersed to be present-at-hand, and this is good too). Also, people can construct bags of examples that have a family resemblance without necessarily having to be able to systematize exactly why they think of them as having that resemblance. This can usefully help delay the tendency of us system builders to prematurely kill off options and strangle flexibility by wanting to know the nethermost implication and/or abstraction of a thing before working with it.

And maybe that's why example-based "testing", which is really requirements engineering, which is really a communication mode, does so much better than the other.

I'm proposing a session on this very topic for Agile 2008. I encourage you to think about proposing a session there, too.

Gauges

The "test" word in TDD is problematical. People are (rightly) uncomfortable with using it to describe the executable design documents that get written in TDD. The idea of testing has become too tightly bound to the practice of building a system and then shaking it really hard to see what defects fall out. There is an older sense of test, meaning "to prove", which would help but isn't current enough. Fundamentally, though, these artefacts are called tests for historical reasons (ie, intellectual laziness). One attempt to fix this vocabulary problem has the twin defects of going too far in the direction of propaganda, and not far enough in the actual changes it proposes.

In any case, I'm more interested in finding explanatory metaphors to help people use the tools that are currently widely available and supported than I am in...doing whatever it is to people's heads that the BDD crowd think they are doing. Anyway, I've found that it's a bit helpful to talk about test-first tests being gauges (as I've mentioned in passing before. Trouble is that too few people these days have done any metalwork.

A Metaphor Too Far

So, the important thing about a plug gauge or such is that it isn't, in the usual sense, a measuring tool. It gives a binary result, the work piece is correctly sized to within a certain tolerance or it isn't. This makes, for example, turning a bushing to a certain outside diameter a much quicker operation than it would be if the machinist had to get out the vernier micrometer and actually measure the diameter after each episode of turning and compare that with the dimensioned drawing that specifies the part. Instead, they get (or assemble, or make) a gauge that will tell whether or not a test article conforms to the drawing, and use that.

And this is exactly what we do with tests: rather than compare the software we build against the requirement after each development episode, we build a test that will tell us if the requirement is being conformed to. But so few people these days have spent much time in front of a lathe that this doesn't really fly.

But, flying home from a client visit today my eye was caught by one of those cage-like affairs into which you dunk your cabin baggage (or not). It would be far too slow for the check-in staff to get out a tape measure, measure your bag, and compare the measurements with the permitted limits. So instead, they have a gauge. From now on (until I find a better one), that's my explanatory metaphor. Hope it works.

Agle 2007

Next time, I'll be writing more about the sessions I attended, and that will have a rather more up-beat tone, since they were pretty good. This post contains my general observations about the conference, though, and they are not quite so good.

So, a somewhat belated write-up of Agile 2007, since I've been as sick as a dog pretty much since I got back from it.

I'm sure that had something to do with spending a week in a refrigerated dungeon. It wasn't clear, upon arrival, where the conference was going to be; the hotel didn't look big enough. Turns out that the conference centre is in the basement. And the sub-basement. And the floor below that. No natural light, no clocks, no external sounds or environmental cues. This plus jet-lag contributed to a disconnected, floating feeling. That and the mammoth programme.

1100 people attended this year, and it is is seemingly the belief of the conference committee that this requires a very full programme to keep them all busy all the time. Individuals and their interactions, eh? No, no, no, session:coffee:session:lunch:session:coffee:session:awkward evening hi-jinks, that's the way. Apparently, next year is going to be even bigger.

And this programme (and session materials) has to be fixed far, far in advance. Responding to change, eh? There's nothing like eating your own dogfood, and is nothing like eating your own dogfood. Trouble is, to get enough sessions to fill that many slots you have to accept a lot of sessions, which means that the bar is necessarily lower than it might otherwise be.

There is no there there

The biggest problem with the venue was that, being spread over three (four, if you count the main hotel atrium) floors it had no identifiable centre, no focus for circulation, so fewer opportunities for the ad-hoc meetings that make these shows so valuable. The nearest thing to a "crush" was the CWAC ("Conference Within A Conference"), a rather half-hearted OpenSpace-ish sort of affair in the most remote part of the centre. More of a "Conference Tacked On The Side Of A Conference" Apparently, the organizers of the CWAC chose that room themselves on the basis that it was broken up by pillars, of which I can see the sense. But really, the committee should have had the intestinal fortitude and spinal rigidity to say "no, we'll figure something out with the pillars, but the place for for the open space is at the heart of the conference."

General observations: fewer "rock stars"; more vendors; more women; more vendors; more non-programmers; more vendors; good international presence; more vendors.

Cheap at Half the Price

Did I mention the vendors? Some years ago I was speaking at Spa, and amongst other things was asked to join in an evening entertainment whereby a bunch of us had to give a speech both for and against a topic. I drew "extreme programming" (which was still a hot topic at the time ;) and one of the points against it that I made was that while it's all well and good that Beck tells us

Listening, Testing, Coding, Designing. That's all there is to software. Anyone who tells you different is selling something.

but in fact almost everyone in the room was selling something. And furthermore, "no-one", I said, "is going to get rich charging commission on the sale of these things" and I threw the handful of index cards that held my notes for the talk to the ground. Martin Fowler I noticed was nodding vigorously at that point.

Well, these days there are all these folks who very definitely are selling something: great big honking lumps of tool intended to "simplify" the planning, management and execution of Agile projects.

Like Cheese?

So what does that all add up to? I think it adds up to a community that has become "mature". Maturity is one of those concepts that the petit bourgeoisie use to rationalise their fear and loathing of freedom and imagination. This has its up side: a "mature" flavour of Agile is going to be a much easier sell to a large range of large corporate clients (I include government departments under this heading) than the Zen/Hippy flavour, but it doesn't have anything like the capacity to be a radically dynamic force for truth and light in the world. I looked briefly into Mary Poppendieck's session, which sounded very interesting but was completely full and I didn't want to stand for 2 hours. I'm beginning to feel that there's something slightly creepy about the rush to embrace Lean principles in the agile community, because Lean is all about maximising throughput by minimizing waste. Now, most development teams need to improve their production, it's true, but isn't there a bit more to life than that?

Brian Marick was handing out posters listing the items that he feels are missing from the Agile Manifesto, which are: discipline, skill, ease, joy. Presumably, in that order. Doesn't that sound like a good deal?

Complexity at Agile 2007

This thread came from here, and will be continued...

Just walked out of the first keynote at Agile 2007: I don't need to hear about an amature mountaineer's love-life (yes, really) no matter how "inspirational" it's supposed to be. What have we come to?

Anyway, that gives me time to get ready for the latest adventure in test-driven development and complexity.

later...

Well, that didn't go quite a smoothly as I'd hoped. The room I'd been given was in some sort of bunker under the hotel and while it did technically have a wireless network connection the feeble signal that we had was unable to support as many laptops as there were in the room. So it was a bit of a struggle to get folks set up to use the tool.

However, some people did manage to gather some interesting data of which I hope that some will be shared here. Certainly, folks have found some interesting correlations between the numbers that the tool emits and their experiences working with their code. Especially encouraging is that folks are applying the tool to long lived codebases of their own and looking at historical terends. These are the sorts of stories that I need to gather now.

Note: the tool is GPL (and the source is in the measure.jar along with the classes). Several folks are interested in C# and Ruby versions, which I'd love to see and am happy to help with.

I sat down with Laurent Bossavit and we experimented to see if we could get equally interesting results from looking at the distribution of size (ie, counting line ends) in a codebase, and it turns out not. Which is a shame, as that would be easier to talk about, but is also what I expected, so that's kind-of OK.

A lot of good questions came up in the session, pointers to where I need to look next: Is there a difference between code written by solo programmers vs teams? Do you get the same effect from using functional (say, Fit style) tests as from unit tests? Is there any correlation with test coverage? Exactly what effect does refactoring have on the complexity distribution? Thanks all for these.

Laurent seemed at one point to have a counterexample to my hypothesis (which was a very exciting proposition), code that he knew had been done with strong test-first, but had a Pareto slope of about 1.59 (and an amazing R^2 of 1.0). But on closer examination it turned out that the codebase was a mixture of solid TDD code that by itself had a slope of 2.41, and some other code (which we had good reason to believe was a, poor and b, not test-driven) that by itself had a slope of 1.31

Unfortunately, I wasn't able to make it to the research paper session where this thing was discussed, or this. But I need to catch up with those folks. In particular, the IBM group report that with TDD they don't see the compelxity of the code increase in the way that they expect from experience with non TDD projects.

What "Ivory Tower"?

Strange rumblings on the Ruby front.

I recall walking through the grounds at Imperial, heading for the first XpDay in the company of Ivan Moore. As we passed by he noted that there was an ivory tower for me to move into. Now, I am prone to a little bit too much theoretical musing on things, it's true, but I though this comment was a bit rich coming from someone with a degree, MSc and PhD in Computer Science.

Anyway, this "ivory tower" notion is an curious one and seems to be very much a Yes Minister sort of thing: I pragmatically balance theory and practice, you pay too much attention to unnecessary detail, he lives in an Ivory Tower. The phrase has strong overtones of academic detachment. Strange, then, that this posting should suggest that Smalltalk was ever in one. Smalltalk was developed in an industrial research lab paid for by a photocopier company. An early (maybe the earliest) adopter of Smalltalk, where some of the folks responsible for spreading certain ideas from the Smalltalk world more widely worked, was an electronic engineering firm.

Currently (mid 2007) Smalltalk is (still) being used commercially in the design of gearboxes, it's being used in the pricing of hairy financial instruments, it's being used to do hard stuff for money. I've even earned money for writing Smalltalk myself, within the last 5 years.

Lisp, now, Lisp did have to escape from an ivory tower. And that didn't work out too well, so it tried to get back in. But the door had been closed behind it. Ouch.

Well, if, as is suggested, "Ruby is a unix dialect of Smalltalk" then it would seem that being unixified is not such a completely good thing. Really, spelling blocks with {}s instead of []'s is neither here nor there (although having that invisible magic parameter for them is really bad). the theory is that being being now outside the "VM" (I think that Giles probably means "image") then Ruby-is-Smalltalk plays much better with others. That's true enough. Like certain other refugee systems taking shelter in the unix world, Smalltalk really wants to be your whole world. But we kind-of know, and certainly the unix way is, that that's not a great model.

What's a shame, though, is that if Ruby is Smalltalk then it is Smalltalk with the most important thing taken out: life. And life comes with the image, and not from objects (or worse yet, merely the instructions for buliding the objects) being trapped in files. Sorry, but that's the way it is. So, until there's a ruby environment as lively as a Smalltalk image, with all its browsers and such, I can't see the ruby-is-smalltalk metaphor doing anything but spreading confusion and disappointment.

Certification, testing, RDF, semantics, a post-lunch haze

Reg "no relation" Braithwaite suggests here a certification for programmers, to cover:

• Continuous integration
• Black box testing
• White box testing
• Design for testing
• Probabilistic testing
• etc. [you get the idea]

Basically, all the things that demonstrate that your system is sound.

Most interestingly, Reg gives this caveat:

Like everyone else in favour of certification, I have my own ides about what skills and knowledge you need to demonstrate to get your certification. Unlike everyone else, I think I would fail my own certification if I didn’t do a whole lot of studying. [my emphasis]

Smart. I have this suspicion that a lot of the angst that surrounds the sporadic talk of "Agile" certification comes from (an unwillingness to admit to) fear of not making the cut. Too many programmers have way to much ego invested in their own assessment of how brilliant they are, and the notion that someone is going to come watch them do their job and maybe say "d'you know what, this isn't very good" is absolutely terrifying to them. Of course, the industry is beset with certifications that aren't worth the mouseclick it takes to get them, and the industry is also beset with development managers who don't do their own hiring and allow HR departments to abuse certifications during recruitment. The Agile corner of the IT world is not alone in this.

But that's not what I want to write about today.

When you eliminate the impossible, whatever remains--however improbable--must be the truth

Over at enfranchisedmind Brian had something to say about Reg's post that segued from certification through testing to static typing. Ah yes, testing vs static typing. Brian presents a list of quite impressive qualities of a program that can be enforced by OCaml's static type system, even things like "That a Postgresql cursor can only be created and accessed with a transaction." That's pretty sweet.

But, you know, correctness of the code in those sorts of terms is only part of the story. And perhaps not even the important part. Brian pretty much says this himself:

One thing I have realized recently. And that is that our job [as programmers] is not to produce working code. It’s a heretical idea, but it really isn’t. Our job, as software developers, is to solve problems.

Emphatically yes. He also says:

The mantra here [..] is “make illegal states impossible to represent”. Or, to quote my old CS teacher, “if you don’t mean it do that, don’t make it possible to do that!” And more so than any testing, even unit testing, static types are on the front line for doing that.

Which is great!

Thing is, though, that Aristotle wouldn't like systems engineering much (he'd probably have liked programming, but that's a whole other thing), because merely having legislated out of existence all the illegal states we can think of doesn't mean that we're guaranteed to do the right thing. As in, the thing that the customer will be willing to pay us for.

And that brings me on to something I've been meaning to write down about RDF.

Mere Semantics

A friend of mine is pondering a career change to work in the field of "document management", and he called me the other day to pick my brains about the Dublin Core. Dublin Core is fine by me, because it's meta-data. Nice to have that in a uniform machine readable format. But then there's this sort of thing:

here we in essence have the reason [that duck typing doesn't "scale"]: one has to limit the context, one has to limit the objects manipulated by the program[...] Enlarge the context, and at some point you will find objects that don't fit the presuppositions of your code. [...] the solution [...] requires one very simple step: one has to use identifiers that are context free. [...] then they will alway mean the same thing, and so the problem of ambiguity will never surface.

The proposal, which appears to be a serious one, is to write code like this:

<http://a.com/Duck> a owl:Class;
      rdfs:subClassOf <http://a.com/Bird>;
      rdfs:comment "The class of ducks, those living things that waddle around in ponds" .

and so forth.

We'll draw a veil over the matter of a.com possibly having a different notion of ducks that b.com (in which case the dreaded UFO wobbles into view, a philosophical hub-cap suspended from a cognitive string), and consider instead what it means to imagine that this sort of scheme would capture the semantics of what it means for an object to be a duck.

I suspect that if one has studied a lot (but possibly not quite enough) of computer science then one ends up thinking that something like RDF is what "semantics" means: an arbitrarily far ramified graph of things joined together by relations (that are things, that are joined together...)

Disclosure: I have studied exactly 0 Computer Science. I have nil qualifications in the area. I mean, I've read this, and these, and so forth, but I've never gained a qualification in Comp Sci.

But I have built a few systems. And based on that experience, what I think the semantics of a fragment of program text is, is the change in the users' behaviour after the text is executed. And this is necessarily a localised, context-dependent thing.

Which brings us back to tests. Now, I don't much like the word "test" these days for the kind of executable document that gets written using *unit, or FIT or whatever. These days I tend to talk more about them in terms of being gauges. Anyway, the great thing about this sort of test is that it is explicit, operational, concrete and example- and exemplar-based. This is greatly at odds with the common understanding of "specification", which tries to have a sort of abstract, universal, infinitude to it but is excruciatingly hard to get right. The other approach is much easier to get right, and for good cognitive reasons, it turns out.

And this is why static typing, of whatever flavour of whizzyness, comes at the systems problem from only one, and I'd suggest the less valuable side. Prohibiting the wrong thing isn't enough, we must also encourage the right thing. And to do that we must write the right thing down. And the right thing is intimately tied up with the existential detail, of some individual in some situation at some time.

This is why I quite like Reg's proposed certification (which, by the way, I very much doubt I could get either). It isn't a certification for QA's, it's a certification for showing that you know what to do, and then showing that you've done it.

How web 2.0 will cost us money

I'm writing this on a 1GHz G4 Ti PowerBook with 1Gb RAM. It's so old, I finished paying for it some time ago.

Until about a year ago I had no desire to upgrade: I don't edit video, I rarely gimp huge images. It's travelled half-way around the world with me (and survived the plunge out of a 747 overhead locker with great aplomb). The machine compiles code of the size I can write by myself fast enough for me not to care about it, it can even do that while playing back music. It can just about drive a 1680x1050 monitor (so long as noting too visually exiting happens). But these days, browsing the web with this machine is increasingly painful as the 2.0 sites get more and more JavaScript intensive, and as that trend spreads to more and more sites. Try doing some general browsing with JS turned off and see how many plain old websites--not a social tag clound in sight--just don't work at all without it. This is a sad state of affairs.

I might add that editing this blog posting is slightly more painful than I'd like, firefox's cpu usage is peaking at about 50%, which is ludicrous.

When I started my professional programming career I was well pleased to have a SPARCStation 5 as my desktop machine. Check out those numbers: 110 MHz! You'll still see those boxes occaisionally today (they're very well built), hidden away in data centers running some admin demon or other. For a while Sun sold headless SS5's as web servers, imagine that. At the time the thought of a "super-computer" grade laptop like the pb I have here would have been laughable. And now it's a crying shame that all this capacity is being burned up in the name of this sort of thing (95% CPU), clever as it is. Which is why I salute Ted for this little investigation, and find this survey of the art rather dismaying for its implications.

Complexity Results from Spa 2007

The story came from here

Spa 2007 attendees saw this presentation, then ran the measure tool over a range of code and kindly recorded some of their observations in comments here. Many thanks to them.

Especially rewarding is Duncan's observation regarding the time-variance of the figures across a refactoring episode. Several attendees also confirmed my suspicion that there's something odd about language implementations.

The story continues here.

Confusion

Matthew Skala published (some time ago) an excellent article highlighting some of the ontological confusion that arises when programmers' and ordinary people's thoughts overlap. I whish I could remember where I first came across the notion that a lot of the problems that arise when that happens (say, when trying to explore a customer's wants and needs of a new IT system) are due to the spectra of abstract <-> concrete being exactly backwards for programmers and real people. If you have a notion where that might have been, please let me know.

To illustrate: there used to be this subject called "Systems Analysis" that folks heading for programming jobs used to get taught. You could probably be a lot older than me and have been taught it, but not be much younger and not have. Which I suspect is why there was any interest at all in this book. To quote Nat Pryce,

Until I started working in "enterprise IT" I didn't realise that people didn't do this.

Well, back in the day I was taught with examples that pretty much sent the message that, say, a "bank account" was a terribly terribly asbtract thing, and an (as it was then) OMT box denoting a class representing bank accounts was a bit more concrete, and that a C++ class definition representing the OMT class was more concrete than that, more concrete still was an object that was an instance of that class. And most concrete of all was the bit pattern in memory that implemented that object. OK. Imagine the embarrassment of trying to explain this view of the world to someone with a crippling overdraft. "No, you see, a bank account is a terribly abstract thing..."

In space, no-one can hear you scream

Another exemplar: This page is clearly a labour of love, and quite impressive in itself. I'm not sure it does anything to make monads any more tractable, however. What it does do is express very clearly this interesting aspect of programmers' thinking, albeit through the reverse mechanism of trying to avoid abstraction.

• "I shall use a metaphor of space stations and astronauts to cut down on the degree of abstraction"
• "A space station is just a metaphor for a function"
• "These astronauts could be anything, Americans, Frenchmen, dogs, gorillas, whales, whatever"
• "The other thing is to note that is our space stations are typed"

and so on.

"A space station is just a metaphor for a function" But I have zero personal experience of space stations, whereas I do have personal experience of functions, so I struggle to find the former more concrete than the latter. Thinks: "the typed space station takes a whale in a space suit..."

I highlight this not to mock Eric (to whom more power and good luck), but because of the crystal clarity with which this language highlights the deep, deep strangeness of how programmers (including myself) communicate with each other as regards what is abstract, never mind with real people.

"What do you read, my lord?"

This sort of thing wouldn't matter so much if the notions of the world convenient to us as programmers didn't keep leaching into the real word. Here's a story that turns out to be about time, identitiy and state (more ideas that we have some sometimes fishy ideas about): I was in a particular bookshop, since gone out of business and this story might partially explain why, and I saw a copy of a Chris Ware book that I didn't have. The copy on the shelf was pretty badly beaten up, so I took it to the till and asked the woman there "do you have another one of these that's in better condition?"

Ware's books sometimes have the unusual feature that the barcode is on a wrapper and not on the book proper, and this copy had lost such wrapper. Well, she searched every surface of the book that she reasonably could to find a bar code to scan, but failed. She handed the book back to me with an apologetic shrug and said "I'm sorry, I don't know what this is". I'll repeat that: "I don't know what this is." A state of profound confusion for someone who works in the book trade, standing in a bookshop, with a book in her hand.

What was meant was something like "the IT system foisted upon me by the senior management of his retail chain will not allow me to perform any actions with this object because I cannot scan it". But what she said was "I don't know what this is" which is perhaps less explicit but more telling. The truth is that the IT system of that shop is not able to distinguish books. It likely has some sort of idea of how many items with a particular barcode have been scanned in, and how many scanned out and therefore how many may reasonably be expected to be found in the shop at any given time. But actual books are beyond it. In the world of this stock control system, rows in a database table are more real, more concrete, than the items on the shelves, and woe betide you if you try to do anything to the latter without the former. It seemes as if the history of this object had rendered it (in the context of the bookshop) without an identity. Remarkable!

And so the person working in the shop has been placed in a world where the tangible objects that are the essence of that business cannot be worked with for want of that key. A book, you see, is a terribly abstract thing...

FizzBuzz

So, there's this thing called the FizzBuzz test, which it turns out that a lot of people applying for "programmer" jobs can't pass. In a spirit of adventure, I had a go at it myself:

(require (lib "1.ss" "srfi")) ;iota
(define (fizz-buzz n)
(let ((multiple-of-three (= 0 (modulo n 3)))
   (multiple-of-five (= 0 (modulo n 5))))
(if multiple-of-three
   (display "fizz"))
(if multiple-of-five
   (display "buzz"))
(if (not (or multiple-of-three multiple-of-five))
   (display n))
(newline)))

(for-each (lambda (n)
       (fizz-buzz (+ 1 n)))
     (iota 99))

It took me a bit longer than the "under a couple of minutes" that Imran wants a good programmer to do it in (but this is not written on a piece of paper, this was written in DrScheme, and I ran it, and it works) but less time than the 15 minutes that "self-proclaimed senior programmers" sometimes take--which I suppose makes me a middling-good semi-senior programmer, which is about right.

The discussion of this on CodingHorror pulls together a lot of amazed reactions from various places, but none of them quite seem to grasp the nettle which is not so much that there a lot of really bad programmers around, but that there are a lot of IT professionals around who apply for "programmer" jobs even though their work experience doesn't actually include any programming.

This is a relatively new thing, and seems to be different from the old "data processing" vs "real programming" divide: I'll bet a grizzled COBOL walloper could polish off the fizzbuzz problem pretty quickly (and it would probably compile first time, too) But if your "IT" job consists, as many do these days, of using rich visual tools to script up interactions between enterprise components supplied by the tool vendor, or maybe filling in the templates used to generate the code that generates the code that gets scripted up by someone else, or any one of a number of other similar things then you certainly work at building IT systems, but what you do is not programming. And really, that's OK, kind-of. It isn't anything to be ashamed of, in and of itself.

The economic and cultural conditions that have caused such dreary jobs to exist and be worth having perhaps are something to be ashamed of, but this is not the fault of the people filling the jobs.

 

Fitted up

Anyway, back in the late 1990's when I was persuing a post-grad course in Software Engineering there was talk of "component foundaries", and the idea that there would be people like the toolmakers and machinists or yore working in these foundaries. And then there would be something like fitters who would assemble working systems out of those components. Please accept my apologies if you are reading this in a country that has not turned its back on manufacturing (except in certain specialized areas) and still know what I'm talking about. Being a fitter was a fine profession, back in the day, but no-one would hire a fitter as a toolmaker (although a fitter might aspire to be and retrain as, a toolmaker). This change now seems now to have happened in the IT industry, without being widely recognised.

Maybe we suffer from a sort of false democracy, a misguided anti-elitism in the industry. And maybe this is a capitulation to those who would commodify our work. Maybe we need to face up to these sorts of distinctions a bit more honestly and stop expecting that everyone who has anything at all to do with building systems will be skilled as a programmer, should be expected to be able to write code from scratch. It just may not be what they do. Ever. If you want to hire someone who can write code from scratch, the pool of "IT professionals" is not the only place to look, nor is everyone in it a candidate.

 

Plug

By the way, I do want to hire such people, and our hiring process for all technical positions involves a session of actual pair programming, at a workstation, writing real code that has to work, and be shown to work. And the problems we set are a damn sight more complicated that fizz-buzz. If you like the sound of being in the group of people who are successful at such a test, and if you live within, or would be willing to move within, commuting distiance of the inside of the M 25, then why not send me a CV?

More Real Engineering

Via 37 Signals this delicious quote, one of several on this page:

If a major project is truly innovative, you cannot possibly know its exact cost and its exact schedule at the beginning. And if in fact you do know the exact cost and the exact schedule, chances are that the technology is obsolete.

As the discussion on 37S suggests, the word "obsolete" is rather strong. But then, the chap quoted here was Director of the Lunar Module Programme at Grumman. Working on Apollo probably gave you a rather different idea of what was obsolete from that of the average person. Or even the average engineer.

But the essence of the quote is absolutely right. And this is a crucial difference between manufacturing and creative engineering. Which of these things you think software development most resembles will have a big impact on what you think a plan looks like, and what sort of things you think are reasonable estimates to use to build that plan. That's not to say that in the case of innovation all bets are off: the Apollo Programme did meet JFK's deadline for putting a man on the moon and returning him to Earth (just).

But the next time you are starting a development project, getting set for iteration zero, or the inception phase, or whatever you call it, think about which parts of the project are innovative, and which are business-as-usual. Hint: using RoR to render SQL conversations over HTTP this time round instead of J2EE is not innovation. That's just no longer attempting to do you job with one hand tied behind your back. Second hint: the innovative bits are the ones that the business sponsors are really exited about and that the architect (don't pretend you don't have one) thinks are going to be "interesting" and that the project manager has to have a stiff drink before contemplating.

There are better ways to deal with the uncertainty that comes along with the huge promise of innovation, and, well... less good ones. You owe it to yourself (and your business sponsor) to figure out which aspects of the project you should be doing which way. Oh, and if the answer is mostly business-as-usual you owe it to the business to ask why the project is being done at all.

Packages?

Chatting down the Tuesday Club last night the notion came up that Java packages really don't carry their weight. The aren't modules, they do lead to the top of your source files being infested with clutter, they aren't units of deployment, they do cause endless hair-pulling about whether a class belongs in com.baz.bar.foo.wombat.banana.frammitz or would it be better in com.baz.bar.foo.wombat.papaya.frammitz.toothbrush Life is too short.

What would be so bad about having all the classes that form your application in one package? It would be nice to have some tool support for filtering views onto that soup of classes, but why have those views (no reason to have only one) hard-coded in the source files?

Further discussion lead to the conclusion that you might want to have one named package, called something like "application.export", in which would be placed classes that are the subjects of external dependencies. You might have one other named package for your application, to avoid the Date problem. But really, no-one, no-one, is doing the thing that the com.yoyodyne.abomination.ghastly.verbose.madness pattern is meant to enable: guarenteeing that there will be no name clashed when your runtime pulls down a bunch of classes from a random place on the network. Think of all the gateways that exist between any .jar file you might think of and your runtime environment. How many opportunities to fix any such clashes exist between finding that the name conflict exists and getting a binary deployed?

Real Engineering

Glenn Vanderburg has put up a great article about the flaws in some of the simple-minded comparisons that sometimes get made between software development and "real engineering". I reminded of some of those shows put out by the Discovery Channel about gigantic civil (usually) engineering feats. I don't have a TV so I ony really see any when I'm travelling and want to zone out for a bit, but I always look for these shows. What I love about them is that, in amongst all the impressive toys and hurculean efforts there is drama. This is really why they make good television, almost more than the spectacle. And they have drama because these big engineering projects always go wrong. They overrun their budgets, miss deadlines, assumptions turn out to be false and invalidate plans, tools don't work as advertised, techniques turn out to be misapplied, and when you get right down to it, it turns out to be the collective inventiveness of the teams doing the work that makes the project a success. Real engineering. The difference between us inthe IT industry and those folks is not that we have spectacular failures and they don't, it's that they learn from theirs.

Simple pleasures vs the singularity

It's a shame that the title of this reddit post, through sloppy use of an implied universal quantification, allows for a cheap refutation. And a shame, too, that the chap in the original article asked his question of Kurzweil in such a way as to admit an answer focussing on the instrumentive.

I have to tell you, a lot of talk about the Singularity gives me the creeps. Most folks who talk about it at all do so in a mode suggesting that it is 1) inevitable and 2) desirable. I don't find this encouraging. But then, I'm not really a technologist's technologist, even though I make my living in IT, so a lot of more-technology-is-better-until-that's-all-there-is talk leaves me cold. And I think back to when I lived on a particularly lovely part of the south coast of England, and to weekends spend pottering about the New Forest (which, for the neophiles amongst you, was "new" in 1079 CE), walking hand-in-handwith my girlfriend, helping her two kids explore the interesting things to be found in woodland ponds, throwing balls for the dog, and so forth. What is the Singularity going to offer that's better than that? The Far Edge Party and it's ilk? I don't think so.