What "Ivory Tower"?

Strange rumblings on the Ruby front.

I recall walking through the grounds at Imperial, heading for the first XpDay in the company of Ivan Moore. As we passed by he noted that there was an ivory tower for me to move into. Now, I am prone to a little bit too much theoretical musing on things, it's true, but I though this comment was a bit rich coming from someone with a degree, MSc and PhD in Computer Science.

Anyway, this "ivory tower" notion is an curious one and seems to be very much a Yes Minister sort of thing: I pragmatically balance theory and practice, you pay too much attention to unnecessary detail, he lives in an Ivory Tower. The phrase has strong overtones of academic detachment. Strange, then, that this posting should suggest that Smalltalk was ever in one. Smalltalk was developed in an industrial research lab paid for by a photocopier company. An early (maybe the earliest) adopter of Smalltalk, where some of the folks responsible for spreading certain ideas from the Smalltalk world more widely worked, was an electronic engineering firm.

Currently (mid 2007) Smalltalk is (still) being used commercially in the design of gearboxes, it's being used in the pricing of hairy financial instruments, it's being used to do hard stuff for money. I've even earned money for writing Smalltalk myself, within the last 5 years.

Lisp, now, Lisp did have to escape from an ivory tower. And that didn't work out too well, so it tried to get back in. But the door had been closed behind it. Ouch.

Well, if, as is suggested, "Ruby is a unix dialect of Smalltalk" then it would seem that being unixified is not such a completely good thing. Really, spelling blocks with {}s instead of []'s is neither here nor there (although having that invisible magic parameter for them is really bad). the theory is that being being now outside the "VM" (I think that Giles probably means "image") then Ruby-is-Smalltalk plays much better with others. That's true enough. Like certain other refugee systems taking shelter in the unix world, Smalltalk really wants to be your whole world. But we kind-of know, and certainly the unix way is, that that's not a great model.

What's a shame, though, is that if Ruby is Smalltalk then it is Smalltalk with the most important thing taken out: life. And life comes with the image, and not from objects (or worse yet, merely the instructions for buliding the objects) being trapped in files. Sorry, but that's the way it is. So, until there's a ruby environment as lively as a Smalltalk image, with all its browsers and such, I can't see the ruby-is-smalltalk metaphor doing anything but spreading confusion and disappointment.

Certification, testing, RDF, semantics, a post-lunch haze

Reg "no relation" Braithwaite suggests here a certification for programmers, to cover:

• Continuous integration
• Black box testing
• White box testing
• Design for testing
• Probabilistic testing
• etc. [you get the idea]

Basically, all the things that demonstrate that your system is sound.

Most interestingly, Reg gives this caveat:

Like everyone else in favour of certification, I have my own ides about what skills and knowledge you need to demonstrate to get your certification. Unlike everyone else, I think I would fail my own certification if I didn’t do a whole lot of studying. [my emphasis]

Smart. I have this suspicion that a lot of the angst that surrounds the sporadic talk of "Agile" certification comes from (an unwillingness to admit to) fear of not making the cut. Too many programmers have way to much ego invested in their own assessment of how brilliant they are, and the notion that someone is going to come watch them do their job and maybe say "d'you know what, this isn't very good" is absolutely terrifying to them. Of course, the industry is beset with certifications that aren't worth the mouseclick it takes to get them, and the industry is also beset with development managers who don't do their own hiring and allow HR departments to abuse certifications during recruitment. The Agile corner of the IT world is not alone in this.

But that's not what I want to write about today.

When you eliminate the impossible, whatever remains--however improbable--must be the truth

Over at enfranchisedmind Brian had something to say about Reg's post that segued from certification through testing to static typing. Ah yes, testing vs static typing. Brian presents a list of quite impressive qualities of a program that can be enforced by OCaml's static type system, even things like "That a Postgresql cursor can only be created and accessed with a transaction." That's pretty sweet.

But, you know, correctness of the code in those sorts of terms is only part of the story. And perhaps not even the important part. Brian pretty much says this himself:

One thing I have realized recently. And that is that our job [as programmers] is not to produce working code. It’s a heretical idea, but it really isn’t. Our job, as software developers, is to solve problems.

Emphatically yes. He also says:

The mantra here [..] is “make illegal states impossible to represent”. Or, to quote my old CS teacher, “if you don’t mean it do that, don’t make it possible to do that!” And more so than any testing, even unit testing, static types are on the front line for doing that.

Which is great!

Thing is, though, that Aristotle wouldn't like systems engineering much (he'd probably have liked programming, but that's a whole other thing), because merely having legislated out of existence all the illegal states we can think of doesn't mean that we're guaranteed to do the right thing. As in, the thing that the customer will be willing to pay us for.

And that brings me on to something I've been meaning to write down about RDF.

Mere Semantics

A friend of mine is pondering a career change to work in the field of "document management", and he called me the other day to pick my brains about the Dublin Core. Dublin Core is fine by me, because it's meta-data. Nice to have that in a uniform machine readable format. But then there's this sort of thing:

here we in essence have the reason [that duck typing doesn't "scale"]: one has to limit the context, one has to limit the objects manipulated by the program[...] Enlarge the context, and at some point you will find objects that don't fit the presuppositions of your code. [...] the solution [...] requires one very simple step: one has to use identifiers that are context free. [...] then they will alway mean the same thing, and so the problem of ambiguity will never surface.

The proposal, which appears to be a serious one, is to write code like this:

<http://a.com/Duck> a owl:Class;
      rdfs:subClassOf <http://a.com/Bird>;
      rdfs:comment "The class of ducks, those living things that waddle around in ponds" .

and so forth.

We'll draw a veil over the matter of a.com possibly having a different notion of ducks that b.com (in which case the dreaded UFO wobbles into view, a philosophical hub-cap suspended from a cognitive string), and consider instead what it means to imagine that this sort of scheme would capture the semantics of what it means for an object to be a duck.

I suspect that if one has studied a lot (but possibly not quite enough) of computer science then one ends up thinking that something like RDF is what "semantics" means: an arbitrarily far ramified graph of things joined together by relations (that are things, that are joined together...)

Disclosure: I have studied exactly 0 Computer Science. I have nil qualifications in the area. I mean, I've read this, and these, and so forth, but I've never gained a qualification in Comp Sci.

But I have built a few systems. And based on that experience, what I think the semantics of a fragment of program text is, is the change in the users' behaviour after the text is executed. And this is necessarily a localised, context-dependent thing.

Which brings us back to tests. Now, I don't much like the word "test" these days for the kind of executable document that gets written using *unit, or FIT or whatever. These days I tend to talk more about them in terms of being gauges. Anyway, the great thing about this sort of test is that it is explicit, operational, concrete and example- and exemplar-based. This is greatly at odds with the common understanding of "specification", which tries to have a sort of abstract, universal, infinitude to it but is excruciatingly hard to get right. The other approach is much easier to get right, and for good cognitive reasons, it turns out.

And this is why static typing, of whatever flavour of whizzyness, comes at the systems problem from only one, and I'd suggest the less valuable side. Prohibiting the wrong thing isn't enough, we must also encourage the right thing. And to do that we must write the right thing down. And the right thing is intimately tied up with the existential detail, of some individual in some situation at some time.

This is why I quite like Reg's proposed certification (which, by the way, I very much doubt I could get either). It isn't a certification for QA's, it's a certification for showing that you know what to do, and then showing that you've done it.

How web 2.0 will cost us money

I'm writing this on a 1GHz G4 Ti PowerBook with 1Gb RAM. It's so old, I finished paying for it some time ago.

Until about a year ago I had no desire to upgrade: I don't edit video, I rarely gimp huge images. It's travelled half-way around the world with me (and survived the plunge out of a 747 overhead locker with great aplomb). The machine compiles code of the size I can write by myself fast enough for me not to care about it, it can even do that while playing back music. It can just about drive a 1680x1050 monitor (so long as noting too visually exiting happens). But these days, browsing the web with this machine is increasingly painful as the 2.0 sites get more and more JavaScript intensive, and as that trend spreads to more and more sites. Try doing some general browsing with JS turned off and see how many plain old websites--not a social tag clound in sight--just don't work at all without it. This is a sad state of affairs.

I might add that editing this blog posting is slightly more painful than I'd like, firefox's cpu usage is peaking at about 50%, which is ludicrous.

When I started my professional programming career I was well pleased to have a SPARCStation 5 as my desktop machine. Check out those numbers: 110 MHz! You'll still see those boxes occaisionally today (they're very well built), hidden away in data centers running some admin demon or other. For a while Sun sold headless SS5's as web servers, imagine that. At the time the thought of a "super-computer" grade laptop like the pb I have here would have been laughable. And now it's a crying shame that all this capacity is being burned up in the name of this sort of thing (95% CPU), clever as it is. Which is why I salute Ted for this little investigation, and find this survey of the art rather dismaying for its implications.

Complexity Results from Spa 2007

The story came from here

Spa 2007 attendees saw this presentation, then ran the measure tool over a range of code and kindly recorded some of their observations in comments here. Many thanks to them.

Especially rewarding is Duncan's observation regarding the time-variance of the figures across a refactoring episode. Several attendees also confirmed my suspicion that there's something odd about language implementations.

The story continues here.

Confusion

Matthew Skala published (some time ago) an excellent article highlighting some of the ontological confusion that arises when programmers' and ordinary people's thoughts overlap. I whish I could remember where I first came across the notion that a lot of the problems that arise when that happens (say, when trying to explore a customer's wants and needs of a new IT system) are due to the spectra of abstract <-> concrete being exactly backwards for programmers and real people. If you have a notion where that might have been, please let me know.

To illustrate: there used to be this subject called "Systems Analysis" that folks heading for programming jobs used to get taught. You could probably be a lot older than me and have been taught it, but not be much younger and not have. Which I suspect is why there was any interest at all in this book. To quote Nat Pryce,

Until I started working in "enterprise IT" I didn't realise that people didn't do this.

Well, back in the day I was taught with examples that pretty much sent the message that, say, a "bank account" was a terribly terribly asbtract thing, and an (as it was then) OMT box denoting a class representing bank accounts was a bit more concrete, and that a C++ class definition representing the OMT class was more concrete than that, more concrete still was an object that was an instance of that class. And most concrete of all was the bit pattern in memory that implemented that object. OK. Imagine the embarrassment of trying to explain this view of the world to someone with a crippling overdraft. "No, you see, a bank account is a terribly abstract thing..."

In space, no-one can hear you scream

Another exemplar: This page is clearly a labour of love, and quite impressive in itself. I'm not sure it does anything to make monads any more tractable, however. What it does do is express very clearly this interesting aspect of programmers' thinking, albeit through the reverse mechanism of trying to avoid abstraction.

• "I shall use a metaphor of space stations and astronauts to cut down on the degree of abstraction"
• "A space station is just a metaphor for a function"
• "These astronauts could be anything, Americans, Frenchmen, dogs, gorillas, whales, whatever"
• "The other thing is to note that is our space stations are typed"

and so on.

"A space station is just a metaphor for a function" But I have zero personal experience of space stations, whereas I do have personal experience of functions, so I struggle to find the former more concrete than the latter. Thinks: "the typed space station takes a whale in a space suit..."

I highlight this not to mock Eric (to whom more power and good luck), but because of the crystal clarity with which this language highlights the deep, deep strangeness of how programmers (including myself) communicate with each other as regards what is abstract, never mind with real people.

"What do you read, my lord?"

This sort of thing wouldn't matter so much if the notions of the world convenient to us as programmers didn't keep leaching into the real word. Here's a story that turns out to be about time, identitiy and state (more ideas that we have some sometimes fishy ideas about): I was in a particular bookshop, since gone out of business and this story might partially explain why, and I saw a copy of a Chris Ware book that I didn't have. The copy on the shelf was pretty badly beaten up, so I took it to the till and asked the woman there "do you have another one of these that's in better condition?"

Ware's books sometimes have the unusual feature that the barcode is on a wrapper and not on the book proper, and this copy had lost such wrapper. Well, she searched every surface of the book that she reasonably could to find a bar code to scan, but failed. She handed the book back to me with an apologetic shrug and said "I'm sorry, I don't know what this is". I'll repeat that: "I don't know what this is." A state of profound confusion for someone who works in the book trade, standing in a bookshop, with a book in her hand.

What was meant was something like "the IT system foisted upon me by the senior management of his retail chain will not allow me to perform any actions with this object because I cannot scan it". But what she said was "I don't know what this is" which is perhaps less explicit but more telling. The truth is that the IT system of that shop is not able to distinguish books. It likely has some sort of idea of how many items with a particular barcode have been scanned in, and how many scanned out and therefore how many may reasonably be expected to be found in the shop at any given time. But actual books are beyond it. In the world of this stock control system, rows in a database table are more real, more concrete, than the items on the shelves, and woe betide you if you try to do anything to the latter without the former. It seemes as if the history of this object had rendered it (in the context of the bookshop) without an identity. Remarkable!

And so the person working in the shop has been placed in a world where the tangible objects that are the essence of that business cannot be worked with for want of that key. A book, you see, is a terribly abstract thing...

FizzBuzz

So, there's this thing called the FizzBuzz test, which it turns out that a lot of people applying for "programmer" jobs can't pass. In a spirit of adventure, I had a go at it myself:

(require (lib "1.ss" "srfi")) ;iota
(define (fizz-buzz n)
(let ((multiple-of-three (= 0 (modulo n 3)))
   (multiple-of-five (= 0 (modulo n 5))))
(if multiple-of-three
   (display "fizz"))
(if multiple-of-five
   (display "buzz"))
(if (not (or multiple-of-three multiple-of-five))
   (display n))
(newline)))

(for-each (lambda (n)
       (fizz-buzz (+ 1 n)))
     (iota 99))

It took me a bit longer than the "under a couple of minutes" that Imran wants a good programmer to do it in (but this is not written on a piece of paper, this was written in DrScheme, and I ran it, and it works) but less time than the 15 minutes that "self-proclaimed senior programmers" sometimes take--which I suppose makes me a middling-good semi-senior programmer, which is about right.

The discussion of this on CodingHorror pulls together a lot of amazed reactions from various places, but none of them quite seem to grasp the nettle which is not so much that there a lot of really bad programmers around, but that there are a lot of IT professionals around who apply for "programmer" jobs even though their work experience doesn't actually include any programming.

This is a relatively new thing, and seems to be different from the old "data processing" vs "real programming" divide: I'll bet a grizzled COBOL walloper could polish off the fizzbuzz problem pretty quickly (and it would probably compile first time, too) But if your "IT" job consists, as many do these days, of using rich visual tools to script up interactions between enterprise components supplied by the tool vendor, or maybe filling in the templates used to generate the code that generates the code that gets scripted up by someone else, or any one of a number of other similar things then you certainly work at building IT systems, but what you do is not programming. And really, that's OK, kind-of. It isn't anything to be ashamed of, in and of itself.

The economic and cultural conditions that have caused such dreary jobs to exist and be worth having perhaps are something to be ashamed of, but this is not the fault of the people filling the jobs.

 

Fitted up

Anyway, back in the late 1990's when I was persuing a post-grad course in Software Engineering there was talk of "component foundaries", and the idea that there would be people like the toolmakers and machinists or yore working in these foundaries. And then there would be something like fitters who would assemble working systems out of those components. Please accept my apologies if you are reading this in a country that has not turned its back on manufacturing (except in certain specialized areas) and still know what I'm talking about. Being a fitter was a fine profession, back in the day, but no-one would hire a fitter as a toolmaker (although a fitter might aspire to be and retrain as, a toolmaker). This change now seems now to have happened in the IT industry, without being widely recognised.

Maybe we suffer from a sort of false democracy, a misguided anti-elitism in the industry. And maybe this is a capitulation to those who would commodify our work. Maybe we need to face up to these sorts of distinctions a bit more honestly and stop expecting that everyone who has anything at all to do with building systems will be skilled as a programmer, should be expected to be able to write code from scratch. It just may not be what they do. Ever. If you want to hire someone who can write code from scratch, the pool of "IT professionals" is not the only place to look, nor is everyone in it a candidate.

 

Plug

By the way, I do want to hire such people, and our hiring process for all technical positions involves a session of actual pair programming, at a workstation, writing real code that has to work, and be shown to work. And the problems we set are a damn sight more complicated that fizz-buzz. If you like the sound of being in the group of people who are successful at such a test, and if you live within, or would be willing to move within, commuting distiance of the inside of the M 25, then why not send me a CV?

More Real Engineering

Via 37 Signals this delicious quote, one of several on this page:

If a major project is truly innovative, you cannot possibly know its exact cost and its exact schedule at the beginning. And if in fact you do know the exact cost and the exact schedule, chances are that the technology is obsolete.

As the discussion on 37S suggests, the word "obsolete" is rather strong. But then, the chap quoted here was Director of the Lunar Module Programme at Grumman. Working on Apollo probably gave you a rather different idea of what was obsolete from that of the average person. Or even the average engineer.

But the essence of the quote is absolutely right. And this is a crucial difference between manufacturing and creative engineering. Which of these things you think software development most resembles will have a big impact on what you think a plan looks like, and what sort of things you think are reasonable estimates to use to build that plan. That's not to say that in the case of innovation all bets are off: the Apollo Programme did meet JFK's deadline for putting a man on the moon and returning him to Earth (just).

But the next time you are starting a development project, getting set for iteration zero, or the inception phase, or whatever you call it, think about which parts of the project are innovative, and which are business-as-usual. Hint: using RoR to render SQL conversations over HTTP this time round instead of J2EE is not innovation. That's just no longer attempting to do you job with one hand tied behind your back. Second hint: the innovative bits are the ones that the business sponsors are really exited about and that the architect (don't pretend you don't have one) thinks are going to be "interesting" and that the project manager has to have a stiff drink before contemplating.

There are better ways to deal with the uncertainty that comes along with the huge promise of innovation, and, well... less good ones. You owe it to yourself (and your business sponsor) to figure out which aspects of the project you should be doing which way. Oh, and if the answer is mostly business-as-usual you owe it to the business to ask why the project is being done at all.

Packages?

Chatting down the Tuesday Club last night the notion came up that Java packages really don't carry their weight. The aren't modules, they do lead to the top of your source files being infested with clutter, they aren't units of deployment, they do cause endless hair-pulling about whether a class belongs in com.baz.bar.foo.wombat.banana.frammitz or would it be better in com.baz.bar.foo.wombat.papaya.frammitz.toothbrush Life is too short.

What would be so bad about having all the classes that form your application in one package? It would be nice to have some tool support for filtering views onto that soup of classes, but why have those views (no reason to have only one) hard-coded in the source files?

Further discussion lead to the conclusion that you might want to have one named package, called something like "application.export", in which would be placed classes that are the subjects of external dependencies. You might have one other named package for your application, to avoid the Date problem. But really, no-one, no-one, is doing the thing that the com.yoyodyne.abomination.ghastly.verbose.madness pattern is meant to enable: guarenteeing that there will be no name clashed when your runtime pulls down a bunch of classes from a random place on the network. Think of all the gateways that exist between any .jar file you might think of and your runtime environment. How many opportunities to fix any such clashes exist between finding that the name conflict exists and getting a binary deployed?

Real Engineering

Glenn Vanderburg has put up a great article about the flaws in some of the simple-minded comparisons that sometimes get made between software development and "real engineering". I reminded of some of those shows put out by the Discovery Channel about gigantic civil (usually) engineering feats. I don't have a TV so I ony really see any when I'm travelling and want to zone out for a bit, but I always look for these shows. What I love about them is that, in amongst all the impressive toys and hurculean efforts there is drama. This is really why they make good television, almost more than the spectacle. And they have drama because these big engineering projects always go wrong. They overrun their budgets, miss deadlines, assumptions turn out to be false and invalidate plans, tools don't work as advertised, techniques turn out to be misapplied, and when you get right down to it, it turns out to be the collective inventiveness of the teams doing the work that makes the project a success. Real engineering. The difference between us inthe IT industry and those folks is not that we have spectacular failures and they don't, it's that they learn from theirs.

Simple pleasures vs the singularity

It's a shame that the title of this reddit post, through sloppy use of an implied universal quantification, allows for a cheap refutation. And a shame, too, that the chap in the original article asked his question of Kurzweil in such a way as to admit an answer focussing on the instrumentive.

I have to tell you, a lot of talk about the Singularity gives me the creeps. Most folks who talk about it at all do so in a mode suggesting that it is 1) inevitable and 2) desirable. I don't find this encouraging. But then, I'm not really a technologist's technologist, even though I make my living in IT, so a lot of more-technology-is-better-until-that's-all-there-is talk leaves me cold. And I think back to when I lived on a particularly lovely part of the south coast of England, and to weekends spend pottering about the New Forest (which, for the neophiles amongst you, was "new" in 1079 CE), walking hand-in-handwith my girlfriend, helping her two kids explore the interesting things to be found in woodland ponds, throwing balls for the dog, and so forth. What is the Singularity going to offer that's better than that? The Far Edge Party and it's ilk? I don't think so.

It's Alive! [flash of lightning, crash of thunder]

Steve Yegge, in a rather lengthy blog posting (as this one threatens to become, and with the good stuff at the end, too) mentions the characterists he believes characterise "great" software. I pretty much agree with them and here is that list, lightly glossed by me:

1. Systems should never reboot [and so in orer to be useful]
2. systems must be able to grow without rebooting [thus]
3. world-class software systems always have an extension language and a plug-in system
4. [as a bonus] Great systems also have advice, [...]hooks by which you can programmatically modify the behavior of some action or function call in the system
5. [in order for 4 to be possible and 3 to be easy] great software systems are introspective
6. [and] every great system has a command shell [so that you can get at the above easily and interactively, see below for the importance of this]

Wot no database?

Steve give examples of systems that exhibit various of these properties to varying degrees, but one whole class of system is missing from his list: RDBMSs. It's possible to get very much carried away with what relational datastores can do, and on the other hand, relegate them to being not much more than file managers. Either way, it's strange that Steve doesn't mention even one RDBMs in his lists of "favourite systems", since the industrial-strength RDBMs exhibit all of the properties that he ascribes to good systems (possibly for small values of "plug-in system"). Certainly they do much more so than, say, the JVM, which is on his list.

But then, when was the last time you saw someone with a terminal open doing real work by composing ad-hoc SQL queries (which is where RDBMs really shine)? Why have we, int eh IT industry, robbed our users of this?

Steve goes on to talk about rebooting as death, and muses about the concious/unconcious boundary and what it might mean to reboot a system on the concious side of it. Highly speculative, but then I did once have a conversation down the pub with someone who reported that someone else (sorry about this) was researching what it would take for self-aware AI to be benevolent towards us. An intruiging notion: those who subscribe to the idea of a creator seem always to assume that the creator is both infinitely superior to the creature and well disposed towards it (except for this one), or at worst indifferent. A self-aware, self-modifying AI might be expected to rapidly become superior to us. And if we're in the habit of killing its siblings as standard operating procedure...

But I digress. So does Steve, and I find that in amongst all the amature philosophising he kind of misses his own point. Towards the end he starts banging on static type systems in general and on Hindley-Milner in particular. This provoked the Haskell folks over on Reddit to show that Haskell can support the sort of systems he describes. Which goes to show how relatively uninteresting the kind of "life" that Yegge talks about is. Frameworks that support hot-swappable executables can be built in Haskell, of course they can. But doing so seems like a stunt.

Back to the Future

Thinking about persistance reminds me of an event: in a previous job I had for a time the title of "Head of Research" with a very open brief to investigate new ways of doing things. The business liked applications to be presented through a web browser so I built a prototype for one system that I had in mind using Seaside. One of the production developers looked at this prototype and asked me how it was persisted. It's persisted, I said, by never turning this box (kicks box hosting the prototype) off. That's not an acceptable solution for a production system, although something that simulates the same effect can be. And Ralph Johnson has this to say about that:

[clang!] just a moment... Oh, the irony: that blog is down (18 Jan 2007), the posting is here, and here's the google cache. Anyway, what Ralph says his own squeak list post that the blog post points to is:

I did this in Smalltalk long before Prevayler was invented. In fact, Smalltalk-80 has always used this pattern. Smalltalk programs are stored this way. Smalltalk programs are classes and methods, not the ASCII stored on disk. The ASCII stored on disk is several things, including a printable representation with things like comments that programmer need but the computer doesn't. But the changes file, in particular, is a log and when your image crashes, you often will replay the log to get back to the version of the image at the time your system crashed. The real data is in the image, the log is just to make sure your changes are persistent.

It's a shame that it has to be when your image crashes, not if, but never mind. This is a pointer to the big win that Yegge misses, in my opinion (and his pro-Haskell commentators, too) The interesting thing is not hot-swapping components without a restart. It's about being enabled to interact with computational objects ("object" in the broadest possible sense) in the same way you interact with physical objects: they should be ready-to-hand not present-at-hand.

A Smalltalk image is a long-lived thing (it's been claimed that some of the standard Smalltalk-80 image survives into contemporary Squeak images) but it has, along with a long lifetime, also a sense of liveness. The objects are just there, beavering away, or perhaps hanging out waiting for you to ask something of them. And they respond immediately. And Smalltalks have this sort of liveness built in by default, and assumed all the way down to a very low level. Working in such systems is a very different thing from working on or with others-and in my experience a much more productive and (the two go together) enjoyable one.

Smalltalk is not unique in this. Common Lisp implementations tend to be this way (and Scheme ones not, which is a shame). Self is this way. Subtext (not the one you're likely thinking of) is very much this way. And, to a limited extent, an Excel spreadsheet is this way. VisualAge for Java tried really hard to be this way, but this seemed to confuse the majority of that new breed of folks "Java Programmer".

Lots more systems should be this way too. Funnily enough, I believe that the best candidate around at the moment for getting this sort of immersed computation experience into the mainstream is JavaScript hosted in web browsers, if only more people would take it seriously.

Complexity and Test-first 3

The story came from here.

At XPDay last year I ran a session exploring some of the ideas about complexity and TDD that I've discussed here before. Attendees at the session got a very early copy of a tool, "measure", which calculates some of the statistics I'm intersted in. From that came some good feedback which I've incorporated into a new version of the tool, avilable here. Note to XPDay attendees: this is a much more informative bit of software than the one you have, I suggest you upgrade.

The session will be repeated at Spa this year.

New Tool Features

Measure now shows the parameters for both a Zipf and Pareto distribution modelling how the complexity is found in the codebase. It also shows the R-squared value for the regression through the data points. This was a particular request from the XPDay session and, with due care and attention, can be used to get some idea of how good a fit the distributions are.

Measure also now accepts a -h flag and will display some information about how to interpret the results.

New Results

Using this updated tool, I examined some more codebases (and reëxamined some from before). This table table shows the Zipf distribution paramters. It's getting to be quite difficult to find Java code these days that doens't come with tests, so some of the codebases with an N shown there are ones from SourceForge that haven't had a commit made for several years.

Codebase	Slope	Intercept	R^2	Automatedunit tests?
Jasml 0.1	0.95	3.52	0.73	N
Jruby 0.9.2	1.47	6.27	0.76	Y
Smallsql 0.16	1.54	5.87	0.80	Y
Sunflow 0.06.3	1.59	5.67	0.90	N
m-e-c scehdule α3-10	1.69	4.94	0.92	N
NanoXML 2.2.1	1.77	5.26	0.94	N
Syncbuilder1999	1.84	5.93	0.9	N
Itext 1.4.8	1.88	7.99	0.91	N
Xcool 0.1	1.93	4.60	0.84	N
Ant 1.7.0	2.25	8.90	0.93	Y
Jfreechart 1.0.3	2.30	8.58	0.94	Y
MarsProject 2.79	2.33	7.9	0.96	Y
Log4j 1.2.14	2.43	7.34	0.96	Y
Junit 3.8.1	2.49	5.99	0.94	Y
Jmock 1.1.0	2.79	6.38	0.96	Y
Spring 2.0.1	2.78	9.70	0.96	Y

So, the lower slope and higher incercept of JRuby was a surprise. But note also that the R-squared is quite low. This causes me to refine my thinking about these (candidate) metrics. A low R-squared means that the linear regression through the (log-log) points describing the complexity distribution is not good. This suggests that the actual distribution then is not modelled well by a Zipf type of relation. Maybe there's something about language implementations that produces this? It's worth checking against, say, Jython.

Meanwhile, those codebases with a high R-squared (say, 0.9 or above) seem to confirm my hypothesis that (now I must say "in the case of codebases with very strongly Zipf distributions of complexity) having tests produces a steeper slope to the (log-log) linear regression throught the distribution. And it looks as if a slope of 2.0 is the breakpoint. Still lots more to do though.

Further Investigation

I'd be particularly interested from anyone who finds a codebase with an R-squared of 0.9 or more and tests, but a slope less than 2.0

The discovery of codeases whos complexity distribution is not well modelled by Zipf is not a surprise. The fact that such codebases do not match the tests => steeper slope hypothesis is very interesting and suggests strongly that the next feature ot add to measure is a chart of the actual complexity distribution to eyball.

Note to publishers of code

This sort of research would be impossible without your kind generosity in publishing your code, and I thank your for that. If your code appears here, and has been given a low intercept, this should not be interpreted as indicating that your code is of low quality, merely that it does not exibit particularly strongly the property that I'm investigating.

The Story continues here.

Thanks to Vin for the table formatting fix

What could this be?

There's a kind of psychological tool called a "projective test" in which the subject is exposed to a deliberately ambigious stimulus, asked a question like what could this be? and then their response to it is very carefully studied. The belief (largely discredited) being that the subject will read into the stimulus all sorts of revealing stuff about their inner tensions, repressed fears and so forth.

Just recently, a sort of projective test for programmers was executed via reddit. This old thing was the ambiguous stimulus, and the response revealed a lot about the inner tensions of the programming community--or the reddit-editing part of it, anyway.

It seems to me that no-one in Rickert's story comes away from looking like a hero, not Charles, not Alan and definitely not their managers. Although all for different reasons. This modulo the fact that both Alan and Charles delivered (that's how you can tell it's a work of fiction). It might make for an interesting interview to put the story in front of a candidate (especially one for a role involving any project or line management) and have them discuss it. More interesting to me that some interview techniques, anyway.

So, that was 1985. Twenty years later, where are we as regards discussion of process, its merits and demerits? Well, we nowadays get this sort of thing, and its aftermath.

Cruising?

So, a client is looking to set up a new continuous integration (CI) environment. For historical reasons the default choice is CruiseControl but there are a bunch more tools around, including the very interesting new Build-o-Matic.

Build-o-Matic exists because it turns out to be quicker and easier to build a CI server that does exactly what you want, from scratch, in Python, than to configure the XML-infested Cruise to do something only quite a lot like what you want.

Still, Cruise is fully enterprisey, which doesn't just mean overly complicated, bloated, opaque, and difficult to use--it also means unlikely to trigger the corporate immune system. Which mainly means that the folks who manage the build machine will tolerate it. There's a whole other discussion to be had about organisations that expect developers to be productive but won't give then control of their environment.

Anyway, It'd be nice to suggest B-o-M to my client, but I have a strong suspicion that it would take longer than the lifetime of the project to get B-o-M approved for use (Ivan knows that this is an issue and is working to make it better) What to do, what to do....

Well, how about this: this is a green-field project, and although there's a web front-end coming down the line all the work being done now is real programming with actual POJO's so we can keep the build short. In fact, the build script is going to be a pretty opinionated thing and quite capable of failing your pre-check-in local build if it takes too long (or exhibits any one of a quite long list of other characteristics). So, what will we lose if we have a cron job that checks out head and builds it every now and again (say, every build-time * 3), and a little web page that watches that job and reports what the cron job is upto?

Is this a complete CI solution? No. But then why would we need one of those?

Lessons from life

On a recent business trip to Germany I visited a vineyard. Seems as if climate change in Europe means that the Rheinland vintners can now make reasonable reds, but the (Merlot, in this case) grapes still struggle in the cooler weather. So, in September time the winemaker goes out and removes, by hand, any unripe fruit before the harvest is made and the wine begun.

The polyglot colleague who kindly translated the winemaker's story put it that they "sacrifice quantity for quality". Of course, I shrugged, they reduce scope.

Winemakers have an immovable "delivery date", the time after which the fruit will be past its best. And since, in the case of the Fleischer family, they believe that their business can only flourish is their product is excellent what they do is prefer to deliver less if that maintains the value of what is delivered. Why is this obvious to a winemaker, but not to so many programmers?

Tidbits from Agile 2006

These are some items from Agile 2006 which interested me enough that I wrote them down.

Peter Coffee's keynote

• it's a wonder that spreadsheets don't have features built in to deal with uncertainty
• tooling to improve programmer efficiency makes software so much cheaper to write that much more software is worth writing: and so generates jobs for programmers, not the converse
• 3M apparently has a target for revenues generated by products less than three years old
• the value of a codebase is not dependent on the effort that went into building it
• nervousness is the enemy of innovation

An OpenSpace session

• Agile development manifests the Kolb learning cycle
• if you wanted to cerify a team as Agile, maybe you'd only really need to certify the facilitators of the retrospectives
• maybe a retrospective (or similar explicit learning activity) is about deliberately moving from unconcious competence to concious incompetence
• ...and maybe also has something to do with Heidegger's distinction(pdf) of ready-to-hand from present-at-hand. Something that I've had cause to ponder before in its relation to software.

Ole Jepsen's session "Agile Meets Offshore"

• perhaps counter-intuitively, distributed teams tend to have shorter iterations than colocated ones

Laurent Bossavit and Emmanuel Gaillot's session "Tool Words and Weapon Words"

• You are likely in a difficult spot when people start saying things which can be given an "objective" spin but are really claiming the property on the left for themselves so as to imply the property on the right of you:
  
• natural vs artificial
• important vs childish
• life vs death

Owen Rogers' Agile Estimation tutorial

• have the "customer" provide estimates, as well as the developers: when the two are wildly different there's something to learn
• the size of a story as esitmated by the developers is independent of its business value
• the stories that happen to be in a given iteration are a random sample from the backlog

Johanna Rothman's hiring tutorial

• using puzzle questions at interview discriminates against candidates who aren't white upper-middle class suburban American males.
• some white etc. folks find this idea rather upsetting

Ward's Ten Favourite Wiki Pages

• arguing for of refactoring is arguing for your own lack of ability
• if you don't know calculus you aren't equipped to "embrace change"
  

"Agile Architecture" at Agile 2006

Last week I was at the Agile 2006 conference, in Minneapolis and I see from the records that some of you were too: hello again to anyone I met there.

My session was in the first slot after the opening keynote on Monday morning, and I got the impression that some of the folks there hadn't fully grasped what the nature of a Discovery Session was, some other presenters found the same, I think. I believe that the origanisers might do more to help attendees understand better what's likely to be expected of them at such sessions for next year: they aren't tutorials. Of course, my session was a re-presentation of an XP Day session, which like Spa sessions (the other sort I'm used to presenting) tend to be exploratory, open-ended and interactive in a way that the typical Agile session seems not to be.

Outputs

As soon as I get unpacked from moving house I'll have the London reuslts here to compare with, but a selection of the Minneapolis results are here.

The session was bookended by very quick brainstorm to try an capture any change in the attendees thinking after the discussions and work with the lego.

Before

TDD-not fit for mainframe volatile develop platform framework shared direction drives communication developers like complexity correct today testable malleable adaptable think big, start small

standard organic correct today testable design isn't dead! nimble easy to change business engagement high test coverage flexible easy to change dangerous

And a couple of longer items:

• Have an idea of what possibilities your customers may need to explore
• Not for conventional projects, developers know too much business and not only code by success criteria
• provide guide rails for teams to make decisions within

Indeed, developers do like complexity. Given half a chance, they will put wings on a train!

After

experiment results the right solution for the right problem do only what's needed to satisfy requirements testable not holy ground willingness to restart different is OK ROI

cost of implementation time to implement brilliant think big, start small responsive correct today (still) locally optimal flexible (within reason) mutant

It looks to me as if the aims of the session were met.

Some conference wiki also bears some comments on the session.

Complexity and Test-first 2

The story came from here.

Of course, I'm not the first to notice that software (or, OO software, anyway) has the scale free property. These folks have done a bunch of interesting work on the scale-free nature of the object graph in running systems, and also in various static properties of code. I might just, though, be the first person to observe that the distributions are quantitatively different for different design methodologies.

What the Wellington group are trying to do is disprove what they call the Lego Hypothesis, that all software can be built out of lots of small interchangeable components. Instead, they claim, because of the scale-free nature of software, the biggest components in bigger systems will be bigger than the biggest ones in smaller systems. I'd read some of this research before, but had forgotten that--conciously, anyway--until reminded of it by Kent Beck. Thanks, Kent.

The story continues here.

Complexity and Test-first 1

The story began here.

Interesting to note that a similar discussion, albeit brief, sparked up on the XP egroup. I posted a link to my previous blog post there, but it seemed to get lost in the flood. Oh well. It will be interesting to compare my results with those in the Muller paper.

Anyway, I've managed to find the time to look at a very small sample of Java codebases, find their distribution of complexity, fit it to a Pareto distribution and take a look at the slope of the best fit straight line. Here's the outcome, codebases ordered by published unit tests per unit total cyclomatic complexity (where applicable):

codebase	#tests/ total CC	slope
jasml 0.10	0	1.18
logica smpp library	0	1.39
itext 1.4.1	0	1.96
jfreechart 1.0.1	0.02	2.43
junit 3.8.2	0.14	2.47
ust (proprietary)	0.35	2.79

A few points present themselves: each of the code bases with no tests published has a substantially lower slope (and so substantially greater representation of more complex methods) than any of those with; of those with published tests, number of tests per "unit" of complexity is positively correlated (at about 0.96, very good for "social science", reasonable for "hard science", but this is "computer science", so who knows?) with higher slope and so a preference for simpler methods.

 The story continues here

No graphs? Check the date

Hello. If you've found that some pages here have not been rendering properly it's likely that you were looking at them at the end of June of beinning of July 2006. The problem was that I moved house, thus changing what PSTN exchange my ADSL connection was carried by, and for some reason my ISP couldn't deal with that without taking my whole online presence through them to out of service--I only noticed this problem when my email account stopped working. I apologise on their behalf.

Complexity and Test-first 0

It recently occurred to me to run a Cyclomatic Complexity tool over a codebase I know to have been largely written test-first/code/refactor (and often test-driven). The code is Java, so I used this Eclispe plugin to measure the per-method complexity.

The distribution of complexity per method looked like this:

Note that the count of methods is shown on a logarithmic scale.

There were 19,917 methods in this codebase. The mean complexity per method was 1.46, and the median complexity is 4.5 Not shown on the chart are a few outliers: one with complexity 91, one with 47, and one with 35.

The Extrema

The 91 complexity method turned out to be the doGet of a servlet providing a simple web service, crammed full of if (method.equals("getLocales")) printLocales(out, ids); type dispatching code. Similarly, the 47 complexity method is mainly concerned with calling out to a third-party API and finding which of a small number of equivalence sets a value from a large domain of return codes falls into. It does this by means of a huge switch with many, many fall-throughs. So in these cases we see a disconnect between the arithmatic of cyclomatic complexity and any notion of the code being complicated, in the sense of difficult to undetrstand (and therefore maintain).

Down at the other end of the scale the tens of thousands of complexity 1 and 2 methods are mostly of these sorts:

• constructors
• bean methods
• forwarders

In between are some more interesting methods, which I will deal with in a later post (which is why this is post #0), comparing the amount of testing that was done to write the method test-first with the amount that the Cyclomatic complexity measure would suggest would be required after writing the method.

Interpretation

In one disucssion by McCabe there is the suggestion that 10 is an important threshold for cyclomatic complexity. The SEI treatment gives 11 as the lower bound of the "simple program, without much risk" class. They don't state quite what a "program" is in this sense, I guess they mean "subroutine" in the structured programming sense. As we can see, methods with complexity >= 10 are vanishingly uncommon in the codebase I examined. Evidence that the team working on this code are keeping per-method complexity under good control.

Now, while we can be confident that these methods are overwhelmingly individually very simple the codebase itself is non-the-less considered (by the folks that work on it) highly complex. It does a lot of different things, and interacts with many external systems, and although it is all written in Java it contains several very different technology stacks. So the distribution of cyclomatic complexity isn't the whole story. The total complexity of the whole codebase is about 20,000. It has far fewer that 20,000 test methods (about 7000, in fact), as a simpleminded application of cyclomatic complexity suggests would be required. Although, if you leave out the complexity 1 methods, the total complexity is only 9,000 or so, which paints a different picture.

I don't want to get embroiled in a disucssion of whether or not cyclomatic complexity even makes sense for object-oriented code, not least because the code inside a Java method isn't particularly object-oriented. I do want to examine a mainsteam notion of complexity and testing (the rule is that the complexity equals the smallest number of test cases requied to cover all paths through the code) in the light of code written test-first. Once I've found a suitable candidate method and tests to explore, I'll let you know.

Wait a minute though...

Something about the shape of that chart caught my eye. It looks very much as if it has something close to a power-law distribution. This shows up better on a log-log rendering:

If we switch to a cumulative count of methods we can extract a pretty convincing Pareto distribution:

It so happens that inside the same repository is a bunch of Java that I'm confident was not written test-first (and certianly not test-driven). That doesn't mean that it's bad code, just written differently. What does its complexity chart look like?

Looks like fairly good scale-free beahviour, but let's look at a fitted Pareto distribution:

Not as good a fit. But perhaps more significantly, the slope is about half the slope of the test-first code, 1.39 vs 2.79

Hypotheses

We can imagine all sorts of reasons for these differences (not least, the non-test-first code is an order of magnitude smaller than the test first, which might account for the lower R-squared) but I'm interested in further investigating the tentative hypotheses that the test-code-refactor cycle results in code that has a distribution of complexity closer to Pareto, and with a steeper slope, than traditional approaches.

If that were true, how might we use it? Well, if the complexity distribution is Pareto/Zipf then there is no "preferred" complexity for the codebase. I'd imagine that if there were a preferred complexity then that might point to a (perhaps widely dispersed) collection of methods that could be candidates for refactoring. Question: do the well-known code smells [pdf] cause such a thing?

I'd further guess that the higher level of dupliction expected in an inadequately refactored codebase (not DRY enough, not OAOO enough) would cause a bit of a hump in the complexity distribution. A sag in the distribution I have less of an inution about, although it might point to something along these lines. You'll have to dig around in Prof Salingaros's work to see what I'm getting at, but it's worth it.

Also, if the slope of the Pareto distribution is low then there will be proportionatly more more complex methods in the codebase than less complex ones--that can't be good, can it?

Well, if it wasn't for the day job I'd be looking into this in ernest. But as it is, I'm going to tinker about with this some more as and when I get the time. I'll report back when I do. If anyone else out there does any measurements like this I'd be fascinated to hear about it.

The story continues here.

Staff to Your Methodology

Bill Sempf posted an interesting hypothesis reagrding programmer ability and choice of methodology. Sound enough notion, but I'm not sure I agree with the details, but replying did give me a chance to tell my Royce story.

Doing yourself no favours

There was a time a couple of years ago when I would often travel between Penrith and London on the West Coast Main Line. The WCML is always having a lot of engineering work done on it, so these journey were often held up or interrupted and took place then on the filthy, slow unreliable old train sets that were all the decrepit permanent way could bear at the time.

At Euston there was a display, put on by Railtrack, showing some of the track relaying work going on. Not only did this video loop show off the smart yellow machine that lifts the old track, removes old ties, lays new ballast, new ties and new track (while a bunch of platelayers stand around leaning on their shovels watching it) but the video was speeded up. In the video, the machine whizzed along. To see Railtrack showing off this technology did nothing to improve the mood of those travellers who had been delayed by works.

Why are you talking about trains?

I was reminded of this when I came to Don Box's exposition of "code==data " in the new C# He shows some C# along with allegedly equivalent Scheme, like this. Here's the C#

Expression<Func<int, int>> e = a => a + 1;

and the Scheme

(define e (quote (lambda (a) (+ a 1))))

Well, people who like this sort of thing will find that this is the sort of thing that they like.

Don't you especially love the way that C#'s type system, being at the "sour spot" for such things, requires the programmer to tell the compiler that the code she's writing implements a mapping from int to int? When in fact the implementation given would work for a range of types. It's a bit of a puzzle why this should be required. The way to avoid programmers having to do this is well known, and some of the leaders in that style of working even already work for Microsoft.

The syntax is (to me at least) ugly, verbose and unclear. So, I somehow doubt that putting these pieces of code next to one another is going to make anyone fall in love with the expressive power of C# Added to which, this just is not in general what anyone who knows Lisp means by "code==data ". They mean macros.

Doing Design

In an extract from the new edition ("2.0", indeed) of Software Conflict Robert L. Glass revisits his prior discussion of "design" in software. He focuses on the undoubted fact that design is a cognitive process. It's perhaps a sad reflection on the state of the industry at the time he was writing that this should have been a noteworthy observation.

Where to begin? How to proceed?

Anyway, it seems that Glass undertook a commendable self-assessment of his stance on design, found it wanting and turned to the research of Bill Curtis and Elliot Soloway. As reported by Glass, after a series of "protocal studies" came up with roughly this description of what programmers do when they design:

1. Construct a mental model of a proposed solution to the problem
2. Mentally execute the model to see if it solves the problem
3. It doesn't, so see which parts of the problem the model fails on and enhance it
4. Repeat until good enough

Strangely, Glass calls this a "well defined process" and looks to the implications of this "new concept of design as a mental process."

By the way, Google wants "protocal" to be a typo for "protocol" but apparently it isn't. "Protocal" seems to mean "talking about what you're doing while you do it."

The problem with the design process as described above is that it has by itself nothing to say about where these mental models come from, or how they are compared or improved. Glass quotes Charles Simonyi on this, thus:

The first step in programming is imagining. Just making it crystal clear in my mind what is going to happen.

Which all, as a description of the design "process", seems only little more useful than the Feynman Problem Solving algorithm (*). Especially since one of Glass's goals in studying this matter was to find a better way to teach design.

Tools for the mind?

In the 2006 update to this essay Glass reports on the outcome of the Curtis/Soloway research: nothing. Apparently, they gave up. Glass states:

Because the nature of design, they discovered, is so intensely cognitive, happening inside the mind at mind speed, the researchers could conceive of no useful tools to help in that process!

What!? No useful tools to help with a process "inside the mind". Come now! I'm not sure I believe that this is what two smart guys like that concluded, unless by tool they meant some clanking monstrosity that would, you know, run on a computer.

Well, programmers aren't the only people who do design, nor who introspect upon their own design activity. In 1964 (about 20 years before the research Glass is talking about) Christopher Alexander produced his doctoral thesis, Notes on the Synthesis of Form, which describes how the "imagining" takes place within a culture and a tradition (the design patterns movement inspired by Alexander seeks to make these things explicit for the society of software builders, and others) in both self-conscious and un-selfconcious modes, but more importantly explains how to traverse the loop from less good to more so. Laurent Bossavit gives a good overview of these ideas. Alexander even went so far as to write some computer programs to help architects and planners use these ideas. That would be a "tool". Oh dear.

As Prof. Sir Tony Hoare has put it

If only we could learn the right lessons from the successes of the past, we would not need to learn from our failures.

(*) The Feynman Problem Solving Algorithm

1. Write down the problem
2. Think very hard
3. Write down the solution

Agile Architecture lessons from SEI

My colleague Tim and I addressed the matter of "agile architecture" at the XPDay conference last year, in a way that raised a few eyebrows. We'd be doing it again elsewhere this year, but the other conferences that we submitted the session to didn't have the imagination or guts to let us put the thing on I guess.

Architecture, by definition

Anyway, recently this article from the SEI ("Leading the world to a software-enriched society") came to my attention. The SEI offers this definition of "software architecture"

The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them.

This strikes me as a rather poor definition, since it means that, for example, I can extract some architecture from this program:
#include <stdio.h>

int main (int argc, char** argv){
    printf("Hello World\n");
    return 0;
}

This is source code to be compiled into a binary component intended for deployment into a container. There are many free and commercial implementations of this container, and there's a really great book that explains all about how the container instantiates components like this, wires them up to the other components they rely on (that #include <stdio.h> thing there is a bit of declarative configuration information that tells the container that this component is going to use some services from the stdio component) There's an entrypoint into the component that the container will recognise, and the component will indicate to the container using a standard value that it has completed its work successfully (whether it has or not, but when did you ever see anyone check the return value of printf?)

Thus, my program, er, I mean component here has a structure composed of software elements, these have externally visible properties, and there are relationships among them. Obviously, then, this system is just dripping with architecture. Almost as much as this paragraph is with sarcasm.

This is not what anyone means by "architecture", but based on the SEI's definition it's hard to see why not. (Once upon a time, though, the facilities used by my component were a matter of live architectural discourse--but that sort of change over time is for another posting)

Distinctively Architectural

Now, this sort of problem happens all the time with definitions, which is why definitions suck for serious thinking about the world. The authors of the SEI article kind-of dodge coming right out and saying that, but they do instead adopt a much smarter stance and start to talk about what distinctions can usefully be drawn between "architecture", "design" and the rest.

Speaking of drawing, they note in passing that it is a source of confusion to use a notation primarily intended to capture design ideas (UML) to capture architectural ideas. Why do we keep shooting ourselves in the foot like this? Especially since, as an industry, we have plenty of alternatives that we could use. But I digress.

Well, the authors present, in fine consultant style, a 2 x 2 taxonomy (apparently it is the "underlying dynamic structure of 2 x 2 modeling that brings richness, depth and a uniquely transformational power to this simple form". Whatever) And of course, being architects, they call this taxonomy an "ontology". They might not have noticed that it's a 2 x 2 model, though, because they only give three terms. Using their distinctions intensional vs extensional and local vs non-local, the diagram they don't provide would look like this:

	Intensional	Extensional
non-local	architecture	???
local	design	implementation

Hmm, how fascinating! What is it that address non-local extensional thoughts? (Hey, maybe that's the uniquely transformational power doing its thing ;)

So, architecture is to do with what global constraints there are on how the parts of the system are chosen and assembled. I might be wrong but I'm guessing that anyone who writes for the SEI thinks that in general architectures come first and are somehow realised or filled in during design and implementation. That fits well with the idea that architectural ideas are extensional--they don't say how to build the system but they do say what general properties any system built to that architecture will have, what declarative constraints they all satisfy.

An Agile view Emerges

But what about the Agile setting? Well that's a broad church, so what specifically about the XP setting? In XP we first build a spike (very thin, goes from one end to the other) to learn about the essence of the problem/solution. Then we build out more thin vertical slices of functionality, refactoring as we go. Where's the architecture? If it is non-local then it can only become evident when there are enough of these slices in place for there to be anything that isn't local. And with test-driven development we very much proceed in an extensional fashion, filling iexplicitct cases one at a time, and discover the generalities through refactoring.

So the architecture (non-local, intensional) can only arise late in the game. It can only emerge.

Hypothesis for further investigation: the non-local extensional thing might just be the system metaphor, which is stated in quite concrete, explicit, albeit general, terms and is intended to apply to the whole system. And is stated early. If so, maybe the model presented in the SEI article helps explain why there are so many folks want the system metaphor to be instead of architecture and why they get disappointed when that doesn't work.

Lessons from Life

Motorcycles, post-conditions and idempotent operations

Whenever during my programming work I have to think about pre- and post-conditions, or idempotent operations, I always think back to the Compulsory Basic Training course I had to pass to get my motorcycle licence.

The point came when the instructor was explaining the indicators (aka turn signals). On most bikes these are operated by a switch on the left-hand grip. It has two degrees of freedom: sliding to left or right, or being pressed inwards. All of these are intermittent contact operations and a spring returns the button to its home position (in the middle, popped out) when it is released. Sliding the button to the left or the right activates the indicators on that side, pressing the button in deactivates the indicators.

Now, when riding a bike, unlike when driving a car, there isn't the tick-tick-tick of the flasher to let you know that the indicators are on (well, there is, but you can't hear it over the wind noise and beat of the engine through your helmet), and neither is there a switch on the steering to deactivate the indicators automatically after a manoevre. And while it's very dangerous to ride along with a misleading indication going, so too is it to take your eyes off the road and peer around the bike to see if the indicators are still going. So, said the instructor, if you aren't sure if you cancelled the indicators after the last turn, don't bother to look to find out if they're still on--just press the button.

You have to be a programmer to get this confused

The Simpson's Building in Picadilly (now home to Waterston's flagship store) is, in its public areas, roughly this shape:

Where the area that in my diagram is the horizontal cross-bar meets the vertial section there are a pair of fire-shutters. Closer up, it looks like this:

On the wall between the apertures for the shutters is a small notice (indicated by the thicker line) stating that "the area between these shutters must be kept clear". But what is that area? I thought at first that it was the area between the shutter on the left of the notice and the shutter on the right of it, until I turned around and saw the large sales desk (shown in brown). On closer inspection this desk has two gaps in it, to allow the shutters to close. I've shown a matching pair of shutters behind the desk, which seems right but I couldn't absolutely swear to it. Anyway, the little island of desk in the middle occupies the space between where the left and right shutters would be when closed: my reading can't be correct.

So then it ocurred to me that the "these shutters" to which the sign refers are each of the shutters on either side of the notice and its respective shutter behind the desk. That's confusing. Confuses me, anyway.

I once saw a very good documentary about fire safety, starring a New York fire marshal. At one point he pulled out his copy of the fire safety code for buildings in NY, NY and said "this is like a holy book, everything it says in here was put there because of a time when a lot of people died." Which is an interesting take on religion, but that's not what I wanted to talk about. This notice about the fire shutters, in a busy semi-public space is important. Understanding it correctly could well be a matter of life or death. The fire shutters are a life-critical system, and yet an important part of their operation is woefully under specified. The notice could have some more precise explanation of just what the "these shutters" the area between which it is that should be kept clear are, and that would help, but really the problem is moving prematurely from specification to implementation.

What is required is that "these shutters must always be able to be closed completely". The person who drafted the motice has thought about a likely scenario that could prevent this from being true and prohibited that. There's also an attempt to capture a requirement about an action (closing the shutters must always complete) by a static constraint (the area between the shutters is clear).

I suspect that normal people wouln't give the instructions on the sign a second thought. Actually, most normal people probably wouldn't notice it at all.

Test-first and flow

Frank Sommers has proposed a conflict between test-first and "flow". To answer the question he asks at the end of the article "Do you really write your tests first?" I find it easiest to quote Jason Yip from the ensuing discussion on artima: "Yes I do... except when I don't and I pay for it every time..." These days I even think differently about the two activities: test-first is programming. Anything else may involve the creation of code, but really is exploratory noodling about--which has its value but the results should be discarded.

I wonder where Frank learned his test-first? I suspect that he hasn't ever sat down and paired for a significant time with a master of the art. It is a skill and can't be picked up just from reading about it and having a bit of a go. Also, test-first is only a transitional practice to test-driven, which is where the big wins come from.

In any case, I find Frank's appeal to the theory of flow to explain why he doesn't enjoy test-first programming rather bizarre. He quotes Csikszentmihalyi on some of the characteristics of flow:

• The activity must present just the right amount of challenge: If the activity is too hard, we become frustrated; if too easy, boredom ensues.

• There must be clear and unambiguous goals in mind.

• There needs to be clear-cut and immediate feedback about the activity's success.

• Finally, our focus during the activity must center on the present - the activity itself.

And suggests that they are in conflict somehow with the practice of test-first. I find the exact opposite, and I'm not alone in that. I'm going to talk more about TDD than merely test-first, because I don't practice the one without the other any more.

I find that test-driven development is the way that I turn programming into an activity with the right amount of challenge. I'm just not smart enough to program any other way. I did somehow manage to get paid to program in the days before I adopted TDD, but I no longer understand how that can possibly have happened. YMMV.

In writing a TDD test I am explicitly setting an clear an unabiguous goal for the next few minutes programming.

The red bar -> green bar -> refactor cycle provides very clear-cut and very immediate feedback on my programming activity.

And, as Frank rightly points out, test-first works best on a very small scale. In fact, TDD encourages you to program in very small steps (and, perhaps counterintuitively, the smaller the steps the faster you make progress). So I find that writing the tests first encourages me to center on the present.

Fascinating that two people can have such utterly opposed experience with the same technique.

Agile film-making?

The possibility of parallels between these disciplines came
up on the XP egroup recently, which remineded me of this wiki posting I made a few years ago. Seems the idea is gaining
more currency.

Job (and other) Trends Revealed

Over at Quoderat someone called iain made a comment on this posting:

Search for commoditized skills like Java, SQL, C++ and you find x,000 jobs advertised. Ideally of course you'd have a longitudinal view on this; you'd have data on the number of jobs offering requiring this and that skill across time and understand the rise and ebb of particular skills.

Well, thanks to the kind folks at job search aggregator indeed.com we can do exactly that.

Their site can display the occurrencece of ads with particular search terms in them over roughly the past year. You have to be careful with false positives, though. The origin of the name of the programming language Forth is from a (forced) mis-spelling of "fourth" as in "generation computer", but "forth" is still a word. It appears frequently in ads for positions in which someone will have to go back and it. I was amazed at the popularity of this niche language, until I figured that one out.

Dynamic Languages Duke it Out...

Anyway, take a look at this:



Don't worry about the extreme tinyness of the numbers. When indeed.com say "all jobs", they mean all jobs. That Python accounts for 0.15% of them is salutary reminder that there's a huge old world of work out there where what programming language you prefer isn't even a comprehensible question.

So, this looks like good news for Pythonistas, and very encouraging news for Rubyists. iain would suggest that this is a sign that the "elite" (whomever they are) should be thinking about moving on from Python. A dubious proposition.

Speaking of elites, that Lisp curve is worth drilling down to:



Unless I wildly miss my mark, that's the 2005 Y Combinator Summer Founders Program wildly distorting an entire job market. Let's not be rude about Lisp's 0.005% of the job scene baseline. (Actually, I suspect that most Lisp jobs never get as far as being publicly advertised, so tight-knit is that community).

...and then get Put in Their Place

But let's put our little posse of dynamic languages into context:



Ouch!

That "java" appears in as much as one fiftieth of "all jobs" is cause for sober reflection. Quickly followed by a stiff drink.

In Other news...

This is absolutely fascinating:



Oh yes, that's something I've waited to see. Mid October 2005, a date (range) to remember: "agile" overtook "RUP". Notice, though, how the "agile" and "RUP" curves seem to fall into step afterwards. This is a potential trend worth watching. As is the seemingly imminent overtaking of XP by Scrum.

Increasingly often now there are lead or management type jobs being advertised with some "Agile methods or RUP an advantage" style requirements. Or even, "Agile methods such as RUP". Hmmm. Well, there are those that will claim that RUP done right is Agile. But I digress.

Thinking about closely correlated search terms, how about these two head-to-head competitors:



That's close to the point of being spooky.

Mull this over

This one pretty much speaks for itself. Couldn't be more zeitgeisty:

Some thoughts about you...

And a big السلام عليك to my readers in Arabia.

One of the great things about teh interweb is how you can set one part of it to watch another.

From this I know that, over the last month, only 7.5% of visitors here used IE. Thus I'm fairly sanguine about the fact that the template doesn't render well for them. It's a shame, and I'm sorry. But I'm not going to fix it, either. You folks should do like the 83% who use Firefox.

To the 80% of readers who are first-timers, I say welcome. And to the 20% of repeat readers, I'm gratified that you found the site interesting enough to return to.