Learning from Architects

From dog–houses to skyscrapers, the discipline of the building architect has long been a rich source of metaphor for system architects. I don't disagree, indeed in one of my contributions to the 97 Things Every Software Architect Should Know I recommended that software architects should learn from architects of buildings. So, you can imagine that my eye was caught by Matthew Frederick's 101 Things I Learned in Architecture School. This small book contains the eponymous count of hints, tips and tricks from a qualified architect to architecture students.

This is a good resource for those of us who would learn from architects of buildings, as it is advice to would–practitioners from a practitioner and as such relates to what building architects actually do, rather than what someone who isn't a practitioner thinks they must, surely, do. This latter is a common failure mode of software professionals seeking inspiration from other disciplines—all the way back to the very first Software Engineering conferences of the late 1960's, in which an hallucinatory notion of what "engineers" do was foisted upon us. But I digress.

Some of the 101 are very low level and very specific (eg Number 90 "Roll your drawings for transport or storage with the image side facing out"), others are much broader and seem to me to have relevance for any sort of design work.

Number 15 tells us that "A parti is the central idea or concept of a building." Wikipedia tells us that parti is from the French prendre parti "to make a decision". The parti captures, presents and summarises the highest level decision that has been made about the organising principle of an entire building or building project, and examples are given where the parti expresses all that in one, highly abstract, diagram.

A parti sounds to me a lot like a system metaphor.

Number 100 tells us that the parti should have a name, such as "half–eaten donut" or "meeting of strangers". Could the nominal (or de facto [*]) architect of the system that you are working on draw such a diagram? Would it tell anyone anything if they did? Could they name the parti, the very highest level design decision from which the rest of the system design flows?

Number 28 tells us that a good designer isn't afraid to throw away a good idea. Notice: a good idea. An idea can be good and not fit with the parti, in which case it has no place in the design. We are advised to "save [...] good but ill–fitting ideas for another time and project—and with the knowledge that they might not work then, either." When was the last time you (or your team) threw out a good idea?

Number 46 tells us to "Create architectural richness through informed simplicity or an interaction of simples rather than through unnecessarily busy agglomerations". Frederick warns particularly against "busying up a project with doodads because it is boring without them; agglomerating many unrelated elements without concern for their unity because they are interesting in themselves." What interesting doodads does your current project have?

One reason to follow the guidance of Number 46 is given in Number 51, which observes that "Beauty is due more to the harmonious relationships among the elements of a composition than to the elements themselves". One achieves this beauty through a design process, or system.

Number 77 cautions that "No design system is or should be perfect. Designers are often hampered by a well–intentioned by erroneous belief that a good design solution is perfectly systematic [...] but nonconforming oddities can be enriching, humanising aspects of your project." 77 also observes that "exceptions to the rule are often more interesting that the rules themselves."

Number 81 notes that "Properly gaining control of the design process tends to feel like one is losing control of the design process" 81 advises that the designer should "accept uncertainty. Recognise as normal the feeling of lostness that attends to much of the process. Don't seek to relieve your anxiety by marrying yourself prematurely to a design solution; design divorces are never pretty" No, they never are.

Number 99 can help. It says "Just do something. [...] don't wait for clarity to arrive before beginning to draw. Drawing is not simply a way of depicting a design solution; it is itself a way of learning about the problem you are trying to solve." I think that much the same can be said for coding. Are the design procedures in your team aligned with these principles?

[*] Even the most self–organised, most cross–functional, most Agile, most collectively–code–owning software development team will have one individual who knows most about (and likely has most influence over) the architecture of the system. You can pretend otherwise, or you can take advantage of it.

Plaudit

Apparently, this is the 141st most "top" blog for developers (as of Q2 2009). That's probably pretty far out along some long tail of topness. Ah well.

The metric used captures something like degree of interest of the rest of the blogosphere. So, thanks for your interest.

Let's talk about feelings

I seem to recall that back in the days when he was prone to wild outbursts of public self-examination (this even before blogging and twitter) John Cleese gave an interview in part about his early experiences with therapy, with a "talking" cure. His therapist would begin, "How do you feel?" and John would say "Well, I think..." and his therapist would interrupt "No. How do you feel?" and John would say "Well, I think..." and his therapist would interrupt "No. How do you feel?" And so on. You can see the problem. And he couldn't, I suppose. Which was the problem.

More and more these days I want to ask people how they feel about their code. Here's part of why.

Ivan Moore and "the other" Mike Hill have this conference session that they do called Programming in the Small [pdf]. I love it. They put little examples of code in front of folks and invite them to refactor, just a little bit. And then reflect on the refactoring. One of the things they've noticed that programers tend to do under these conditions is futz around with the code before getting down to actually improving the design.

Mike and Steve Freeman do a similar session called "Programming without Getters" This is in the so-called "dojo" format, where a revolving pair of programmers is invited to take some fairly typical "enterprise" class code and refactor it to remove the getters. There are those of us that believe that there can be something quite special about OO code with that property, but the session didn't really get there. That's because the pairs couldn't bring themselves to do the refactoring as asked, they couldn't even start to remove any getters, until they'd futzed around with the code first. And since a new person rolls into the pair every five minutes or so, it's pretty much a non-stop futz-fest.

Linda Rising saw a talk I give about this design metric that I've been playing with over the last couple of years (I have a day job, so progress has been slow). She was struck by my observation that folks who've tried this metric on their own code have reported that refactoring which made them happier with the code also increases the value of the metric. Linda wanted to know if they really said "happier". They really do.

It turns out that Linda did some research back in the day on a design metric of her own. During this work she had noticed that in general, programers like to futz around with code before they get down to work on it. If I recall correctly, she asked Dick Gabriel about this and he said that "programmers do that", along with some allusion to that metaphorical aphorism about the flavour of soup and pissing in it. I'm sure a lot of that goes on. But what Linda (again, this is as well as I recall) further noticed was that they tended to do this much less with code that scored well on her metric. And they described themselves as being more comfortable working on this code than on code that had a low score.

Now, I've nothing against metrics in principle (after all, I seem to be in the middle of inventing one) but I'm rather dubious about all this dashbordery and piechartism that's going on these days, all this getting of the CI server to dish up trends of metrics and blah, blah, blah. It's nice to have the numbers, it's nice to see a healthy trend, but.......

How do you feel about your code? Does it make you happy? Is it comfortable?


XP Day London 09: Call for Sessions

Submissions are now open for programmed sessions at XpDay London 2009, to be held 7th and 8th December 2009. http://www.xpday.org/

You are invited to propose a session for the first day of the conference. We are particularly interested in the following
  • Experience reports—share your stories of challenge and success with Agile and Lean techniques. Experience reports will be intensively shepherded by experienced practitioners.
  • Hands-on technical sessions—share techniques and practices in practical sessions: workshops, tutorials, simulations
  • Practitioners' advances in the art—share the techniques of expert Agile and Lean practitioners, work with them to move the craft forward.
The second day of the conference will be an OpenSpace session with topics selected at the end of the first day. Programmed sessions are most suitable for topics requiring some set up or extensive preparation.

To submit a session, please go to http://xpday-london.editme.com/XpDay2009Submissions

Submissions will be accepted until Friday 14th August.

Flow: are two dimensions enough?

Inspired by a comment by Joseph at the recent Agile Coach's Gathering I've been experimenting with the use of this model in mid-year reviews for my team.

One of the guys observed that it seems to have a missing dimension. It seems that it's possible to have a case whereby an individual is working on a solidly challenging problem, well up the y-axis, and they have the high level of skill to meet that challenge, well along the x-axis...and when all's said and done they'd really rather be doing something a bit more enjoyable.

Missing axis: fun.

Bridging the Communication Gap

Call it automated acceptance testing, functional test driven development, checked examples or what you will—the use of automatic validation is one of the most effective tools in the Agile developer's kit.

It's a large and involved field, touching on almost every aspect of development. Handy, then, that Gojko Adzic has published a very comprehensive guide to the use of automated acceptance tests in contemporary Agile development practice, Bridging the Communication Gap: Specification by example and agile acceptance testing.

This book is a much-needed checkpoint in the on-going adventure to discover (and re-discover) how to write software effectively. Gojko is a very energetic enthusiast for these ideas, and a very experienced practitioner of them. His knowledge and expertise is present on every page.

A strong theme runs through the book—that the reason we capture examples of required behaviour and automate validation of them is to improve communication. Examples turn out to be a very powerful way to understand a problem domain and to explore a solution suitable for that domain. There turn out to be fascinating reasons for why this is true, but Gojko quite reasonably focusses on practical advice.

The main body of the book tells a story, a story of understanding, finding, and using examples to create shared understanding across a team. Gojko gives very concrete advice in a series of short chapters and explains how to do this. How to organise a workshop to find examples, how to find good examples, how to use tools to automate validation, how to use the resulting tests to guide development. Each chapter ends with a handy bullet list of key points. Together with other material on the best use for developers to make of such checked examples, and how to fit example discovery and capture into a typical Agile development process Bridging the Communication Gap provides as close to a vade mecum for newcomers to the discipline of functional test driven development as we are likely to see.

Gojko draws informative parallels with other techniques more or less strongly aligned with the Agile development world. This places the practice of Agile acceptance testing in context, and as a team-wide activity, reinforcing the cross-functional nature of the tool. Always the emphasis is on helping the various stakeholders in a development project communicate better.

There is a survey of tools available for this kind of work, which I might wish were slightly broader in scope and a little more detailed, but it does give a good overview of the market leaders. "Market leaders" in the weakest sense, since it turns out that the best tools for this kind of work are all FOSS: big-ticket corporate testing tools really aren't in this game.

Various points regarding writing and using tests are illustrated with (of course) illuminating examples. Also described are limitations of these techniques and some pitfalls to watch out for, something that more promoters of development techniques should provide.

The book is self-published and my copy was printed by Lightening Source. Books produced this way are getting better all the time, but are still not presented at the level of quality one would expect from a commercial publishing house. The pages seem very full and with the choice of font made the text a very dark colour which I don't find easy to read. The section and sub-section headings are sometimes over long and are not laid out well, a combination that I found made the book less easy to navigate than it might have been.

I will be using with book with clients and recommending it to them for future reference. A boon to the community.

The Quality of non-declining Velocity

I'm supposing that anyone reading this blog will be familiar with the stuff-left-to-do vs time chart used by many Agile teams to track the progress and (with due care an attention) predict the outcome of a development episode:In his Zurich Lean/Scrum/Agile show keynote Ken Schwaber presented an interesting slant on this chart (apparently he got this from Ron Jeffries). The argument is that if you have poor internal quality—in particular if your "definition of done" is not strong enough and does not require you to keep your code in tip-top condition—then there will be a hidden accumulation of stuff-left-to-do. So your chart is in effect more like this:
This is interestingly different from the technique that some use to show scope added during an episode, with a stepped baseline. Here, additional work is accumulating, invisibly, inside your code. Work that you will have to do at some point to get a releasable increment. This has the effect of making a burn-down line more shallow than it appears, suggesting that there will be either unanticipated under-delivery of scope or (worse yet) a need to slip delivery.
If the causes of poor internal quality are not rectified, then this effect will repeat, and over successive episodes the team in question will get slower and slower (or deliver less and less)
Talking about this with Karl Scotland and Joseph Pelrine in the bar afterwards we tossed around the idea that this shows internal quality (traditionally a hard thing to measure) to be something like the first derivative of project velocity with respect to time.

And now to stretch the metaphor to breaking point. For that to make sense quantitatively it seems as if what we're really saying is that if the internal quality Q is less than some threshold Qcv (the quality of non-decreasing velocity) then the velocity V will decrease over time:
∂V/∂t∝ Q-Qcv
Well, it seems reasonable that this will hold for low quality, when Q - Qcv is negative. But what about when Q - Qcv is positive? Is it possible to take a team that is writing code at the level of quality required for non-decreasing velocity—that is, not accumulating hidden extra work—and then increase velocity by increasing quality?

I think it is.

I think that a team can push really hard on internal quality and have it turn out that there is less work to do than they thought to get finished.

And maybe that's obvious—and maybe it isn't—but certainly I now feel as if I have a much better handle on how to explain to someone why (as the Software Craftsmanship folks say) the only way to go fast is to go well.

The Vendor/Client Relationship in Real Life

What would happen[youtube] if clients tried to deploy the kinds of arguments they use with "professional services" suppliers in other situations...

Epic user interface fail of Homeric proportions

A little while ago I was riding the now sadly degenerate East Coast Main Line service in a Mk IV coach and noticed a bit of a curfuffle in the vestibule (I love it that British railway carriages are still referred to as having "vestibules"). Paying attention I discovered that an elderly lady had required assistance from the train staff with the door of the toilet. This seemed a little odd, so after a suitably discreet interval I went to investigate. One has to make one's own enertainment on the train.

What I found was a flabbergasting cascade of fail.

This advice is welcome:
DSC00106
Judging by the design and finish, this is the original signage.

But how to close and lock the door?
DSC00104
Ok, slightly non-obvious. Also, lacking some rather crucial information, it will turn out.

Seems as if more people than the lady I saw have had problems with the door, since there was this later, auxiliary sign:
DSC00107
Apologies for the poor quality. The text at the bottom reads "If the 'lock' button is not illuminated, the toilet doot is NOT locked" It might very well be closed, you see, but not locked. The original signage has braille attached, not so this vital little nugget of information (as I recall).

It seems that not even this prompt has quite been doing the job, as this third sign had also been added:
DSC00105
The visually impared (and those on urgent business) are now in serious trouble.

Being of an equiring mind, and finding the complexity of this control system too hard to believe, I did a few experiments and made an important discovery not covered by any of the above instructions but of no little importance I think.

If you press, or, let us say, accidentally nudge, the 'lock' button while the door is closed and locked the door (which is powered in the interest of the mobility impared, a good thing in itself) both unlocks and opens.

Now, how hard could this be? What am I missing from this design?

Still two buttons, their respective behaviour being:
  1. If the door is open, close and lock it. If the door is closed and locked, no action.
  2. If the door is closed and locked, open it. If the door is open, no action.

I'll be popping over to Switzerland in a couple of weeks, a place where they still take trains (and much else) seriously. And shall on the trip from Flughafen Zurich to the Haputbahnhof be paying close attention to the toilet doors.

Second Opinions

Got an opinion piece up in E&T magazine regarding second opinions.

Many thanks to all who responded to my earlier posts and enquiries about that. I'll be posting digests of some stories soon.

Identity as a Process

Returning to a topic I've thought about before: identity is a problem.

I expect that most of you to be familiar with the "my grandfather's axe/boat/knife..." problem. In short, how does the identity of a composite object vary (or not) as the identities of the components vary? One proposed solution is the so–called perdurantist approach which hinges on the observation (at once both banal and deeply challenging) that what we think of as objects in the world are really structures with extent in three spacial and one temporal dimension (pace Kaluza-Klein type arguments). We don't seem to be very good at that sort of thinking. Note that perdurantism seems still to be talking about fixing the boundaries of a thing in order to identify it. I think that's missing a trick.

This [pdf] (via Michael Feathers) is a treatment of that trick I've not seen before.

In that paper the biological concept of autpoiesis (and the complex of ideas around it) is used to analyse the working of the glider pattern in Conway's Game of Life. I read the paper as telling us that identity of a glider is extent of the continuation of the process which at any given time looks to us like a glider.

Now, how to apply this understanding elsewhere...

QCon Panel: A Great Leap Forward or Exposed Artery?

A QCon panel disucssion re "Transparency" is now up. It features Kent Beck, Chris Matts, John Nolan and myself, with Steve Freeman in the chair.

The Treppenwitz was pretty strong on this one. What I'm thinking now is that instead of all that ghastly droning on I did, I should have said "the truth shall make you free" . The only thing to watch out for is that being free, although far superior to the alternative, might not be particularly comfortable.

But it is still far superior.

Software Craftsmanship 2010

Apparently there will be one. Excellent! For all my growing distaste for the This Movement and That Manifesto and what all that's springing up around the parts of the industry that I see most closely, a place that happens to be of the same name as one where people can come together and share an enjoyment of good code and coding is a fine thing.

Jason says that he's "banning talks, presentations or any other kind of session that doesn't involve real live coding" I appreciate the gesture, but I'm not sure that's quite right. I found that Ade's session on mapping personal practices particularly valuable—although it could have benefited from another hour or so. None–the–less a programmers' conference with programming as its core topic illustrated through programming is a too–rare thing and Jason is to be commended for making one happen.

Both of the sessions that I ran there involved real coding by attnedees and even though I wasn't coding myself I learned plenty from the reactions of those who were, such as Gojko's on the TDD session. I've not come across a better forum elsewhere for that kind of discussion on that sort of scale than the SC conference, so having another one seems like a very good thing.

Fallacy as tool

In the comments to this post regarding the level of debate regarding climate change in certain circles Pithlord has this to say:
Most fallacies aren’t really fallacies when you reinterpret them as Bayesian reasons to give an idea more credence rather than iron-clad syllogisms. Without [...] the “ad hominem fallacy” [...] you’d give all your money to Nigerian spammers.
That's a very nice formulation of the idea that the "fallacies" of logic are amongst the tools of rehtoric. This is an notion that has fasinated me ever since I first read Zen and the Art of Motorcyle Maintainance. The rhetorical apporach may not demand agreement (in the way that a sound and valid argument would) but it does tend to persuade. And the answers given by fallacious arguments are not neccessarily wrong, just uncertain.

Many folks in the IT industry seem to want to bludgeon their interlocutors into agreement with something that looks a lot like a proof (mea culpa). As I get older the gentle art of persuasion seems more and more attractive.

Seduced by the drama?

Have you ever watched those shows on the Discovery channel (or similar) where the huge construction project goes a bit wrong? If it were more mainstream programming the story would pretty quickly stop being about the stuff and start being about the people, but since it is intended for 14-year old boys of every age these programmes don't quite go down that route. Big yellow machines, yum! But they do always have that drama to them. Drama comes form conflict and in these shows the conflict is between what the plan says should happen and the conditions in the world. A smooth and straightforward project would be less than gripping.

There are lots of those, but they wouldn't make great television. I find that many of these shows still don't make especially great television anyway because what actually happens is that, for example, a smooth technocrat (often German, the best sort) arrives, points out a few alternatives, gets everything back on track and off we go to a successful delivery. That's how it mostly is in the grown-up world of proper stuff. Risks materialise as issues (as it is expected they will, from time to time) and are dealt with in a calm and orderly way. There's the occasional stopppage, the odd bout of overtime, the best crane driver in the world has to be lured out of retirement for this one last job or whatever it may be. But there's no food fight. Food fight? Bear with me.

Scott Berkun has an essay out called "Ugly Teams Win", a part of the forthcoming Beautiful Teams. He presents an...interesting model:
[...] when things get tough, it's the ugly teams that win. People from ugly teams expect things to go wrong and show up anyway.
Well. He passes through some interesting observations about the, what shall we say? challenging personalities of several stellar individuals in several fields. You might be very happy to have a Picasso in your house, but to have had Picasso himself would be a different matter. Fine. Of course, a gang of stellar performers is a very different thing from a well-performing team.

However, Berkun's point is well made that the best people to have on your project for the good of the project might not appear to be the best people generally, in all sorts of ways. Building effective teams is a tricky business. He goes further, though:
The only use of beauty applied to teams that makes sense is the Japanese concept of wabi-sabi. Roughly, wabi-sabi means there is a special beauty found in things that have been used.
I'm not sure that it does, although things that have been used often end up wabi-sabi. Here's a description of wabi-sabi as I've come to understand it
Wabi refers to that which is humble, simple, normal, and healthy, while sabi refers to elegant detachment and the rustic maturity that comes to something as it grows old. It is seen in the quiet loneliness of a garden in which the stones have become covered with moss or an old twig fence that seems to grow naturally from the ground. In the tearoom it is seen in the rusty tea kettle (sabi literally means rusty). The total effect of wabi and sabi is not gloominess or shabbiness, however, but one of peace and tranquillity
In summary "wabi-sabi refers to the delicate balance between the pleasure we get from things and the pleasure we get from freedom from things"

One interesting aspect of this is that things made from natural materials (wood, stone, leather) may acquire charm with wear, but things made form synthetic materials seem not to. Objects can become wabi-sabi through use, wear, or the simple passage of time and the natural processes that they take part in. Berkun wants that to apply to teams:
In this sense, the ugly teams I described at the beginning of this chapter, the underdog, the misfit, represent the wabi-sabi teams
That seems like a huge leap to me, especially when we find out what he means for a team to be used. Here's how he describes the early days of a project, the "Channels" functionality of IE 4:
The deals we made forced legal contracts into the hands of the development team: the use of data [...] had many restrictions and we had to follow them, despite the fact that few doing the design work had seen them before they were signed. Like the day the Titanic set sail with thousands of defective rivets, our fate was sealed well before the screaming began. Despite months of work, the [...] team failed to deliver. The demos were embarrassing. The answers to basic questions were worse.
"Our fate was sealed well before the screaming began" Oooh-kay. And this is the steady-state:
somewhere in our fourth reorg, under our third general manager and with our fifth project manager for Channels, the gallows humor began. It is here that the seeds of team wabi-sabi are sown. Pushed so far beyond what any of us expected, our sense of humor shifted into black-death Beckett mode. It began when we were facing yet another ridiculous, idiotic, self-destructive decision where all options were comically bad. "Feel the love," someone would say.
At least one of Berkun and I have completely and utterly failed to understand what wabi-sabi means.

But that's the least of the issues I have with this. It looks to me as if this team is not being used in the way that an old shoe was used (and so gained its comfort, charm, personality and identity). This team is being abused. Here's how they ended up:
Late in the project, I became the sixth, and last, program manager for Channels. My job was to get something out quickly for the final beta release, and do what damage control I could before it went out the door in the final release. When we pulled it off and found a mostly positive response from the world, we had the craziest ship party I'd ever seen. It wasn't the champagne, or the venue, or even how many people showed up. It was how little of the many tables of food was eaten: in just a few minutes, most of it had been lovingly thrown at teammates and managers.
And there's the food-fight. Don't be distracted by the hysterical high-jinks (as bad a sign as they are). Note that they've had six programme managers by this point. The end-point of the project is described in terms of damage control. Does that sound like the wabi-sabi of the moss-covered stone lantern in the quiet garden? Does it in fact sound like an in any way attractive or desirable outcome? Berkun certainly seems to want to call this a success. He says "The few who remained to work on Internet Explorer 5.0 had a special bond". No doubt they did, no doubt they did.

Here's where this story starts to turn my stomach a little. You see, after going through this dreadful experience "the few that remained" went on to make "Internet Explorer 5.0 [...] the best project team I'd ever work on, and one of the best software releases in Microsoft's history" Maybe so. But at what cost? The majority that didn't remain (it's hard not to imagine them being considered washouts, dropouts, failures), how did they feel about being placed in this outrageous position on IE4? And what are we to infer about teams toughing it out through pre-doomed projects?

One of the few cogent points to have emerged form the recent spasm of interest in so-called Software Craftsmanship is the idea that a "craftsman" has a line that they will not cross, things that they will not do. I tend to agree. I think that the industry would be in globally better shape if more people were prepared to locally say "no" to destructive madness of, well, of exactly the kind reported here for IE4 Channels. I don't consider the members of that team heroic for having made it through and bonded and all that stuff. Well, certainly not "the few". I'm inferring that some folks gave up on this project, walked away from the screaming and got on with something less harmful to themselves. Those are the folks I want to celebrate. And at every scale.

I'm very disturbed that a story such as this one is going to make it into a book about "beautiful teams", even if the point of it is supposed to be the subsequent success of the IE 5 team after their traumatic bonding experience.

This story celebrates failure. And it celebrates a particularly seductive kind of failure, one with which the IT industry is riddled. It celebrates a macho bullshit kind of failure that looks like success to stupid, evil people. It celebrates a kind of failure that too many programmers have come to (secretly) enjoy, and that too many businesses have come to expect that their programmers will (secretly) enjoy and therefore put up with.

In the grown-up world of proper stuff stupid, doomed, destructive projects get cancelled. And that is a successful outcome. We should do more of that.




Life in APL, programming in a live environment

This gorgeous screencast of an APL session shows an implementation of Conway's Game of Life being derived in APL.

It's a very delicious demonstration of what can be achieved in 1) a "live" computational environment (rather than a mere language 'n' platform) with 2) a language that works by evaluating expressions (rather than merely marshalling operations) and 3) already knows how to show you what complicated values look like (because it understands its work to be symbolic not operational).

Ponder what the Dyalog folks (no affiliation) are showing you there, ponder just how far towards that same goal you'd get in whatever programming system you use at work in 7 minutes 47 seconds (even if you'd done as much rehearsal as I'm sure they did) and then ponder the state of our industry.

Then have a stiff drink.

Agile Coaches Gathering

I'm going to the Agile Coaches Gathering at (and partly in aid of) Bletchley Park in May. I commend the event to you.

The Lives of Others, the Lives of Ourselves

I've kept to a rule with this blog, that the posts on it should be of broadly technical and mostly professional nature. I'm going to break that rule now.

Some time ago I watched the film Das Leben der Anderen, a bitter-sweet offering from Florian Henckel von Donnersmarck. It's a love story, of several kinds, and a thriller and little bit of (and little bit of a response to) an Ostalgie comedy. In one scene Hauptmann Wiesler, a Stasi interrogator and surveillance officer, teaches a class of prospective secret policemen. He puts great emphasis on the importance of the handling of the seat cover after an interview. This was a real thing: during interrogation a cloth would be used absorb the personal odours of the subject and then sealed in a jar. When, not if, the Stasi needed to find you the jar would be unsealed and used to give tracker dogs your scent.

I recall hearing about this in a John Peel piece many years ago. He went to interview some punks in the former DDR and they spoke of this and many other horrors of living in an authoritarian surveillance society. And this sort of thing was considered a genuine horror. A fine example of the reason why it was worth the grinding fear of the Cold War, to avoid living in that sort of society.

Between the time of watching that film and writing this piece I got into a conversation with a Daily Mail type of fellow, who was complaining that "the UK is becoming a complete police state. Like[sic], on the level of Nazi Germany". This is well known to be a bad way to argue. My response was that it wasn't really, not even on the level of Communist Germany, go watch Das Leben der Anderen to see what a real police state looked like.

Today I'm thinking that I might owe that guy a little bit of an apology. This report [pdf] from the Rowntree Reform Trust makes dismaying reading:
  • A quarter of the public-sector databases reviewed are almost certainly illegal under human rights or data protection law
  • Fewer than 15% of the public databases assessed in this report are effective, proportionate and necessary, with a proper legal basis for any privacy intrusions
  • The benefits claimed for data sharing are often illusory.
Well, joy.

Top of the list of problematical databases is the National DNA Database. A little background: England and Wales (Scotland and Northern Ireland have their own arrangements) are Common Law countries and like other such used to have a distinction between summary offences, misdemeanours and felonies. We don't any more (much as we don't have Grand Juries, either) we only have summary vs indictable offences. But that only applies if you are brought to trial.

We used also to have a distinction between arrestable offences (effectively, felonies) and non-arrestable offences. Over the years so many more offences have been created that this distinction was seen as unhelpful and at first it was blurred and eventually, in the Serious Organised Crime and Police Act 2005, abolished.

Under section 110 of that act a constable may arrest, without warrant, anyone he or she "has reasonable grounds for suspecting to be about to commit an offence" (any offence at all, remember, however minor) if this is "necessary". For example "to enable the name of the person in question to be ascertained" (and for various other reasons listed in the Act). What this amounts to is a summary power of arrest, of anyone, at any time. This was spectacularly under-reported at the time. And here's the punchline: everyone who is arrested is obliged to give a DNA sample and that goes on the National DNA Database.

And it stays there. It stays there if you are released without charge. It stays there even if you are charged, go to trial and are acquitted. This was already known to be in breach of the European Convention on Human Rights. In that unanimous ruling the European Court of Human Rights made a number of interesting statements:
The Court was struck by the blanket and indiscriminate nature of the power of retention in England and Wales. In particular, the data in question could be retained irrespective of the nature or gravity of the offence with which the individual was originally suspected or of the age of the suspected offender; the retention was not time-limited; and there existed only limited possibilities for an acquitted individual to have the data removed from the nationwide database or to have the materials destroyed.
And indeed it turns out to be non-trivial to get the DNA of innocent persons removed from the database.

Also,
The Court noted that England, Wales and Northern Ireland appeared to be the only jurisdictions within the Council of Europe to allow the indefinite retention of fingerprint and DNA material of any person of any age suspected of any recordable offence.
and
The Court expressed a particular concern at the risk of stigmatisation, stemming from the fact that persons in the position of the applicants, who had not been convicted of any offence and were entitled to the presumption of innocence, were treated in the same way as convicted persons.
So we are, here in the UK, faced with a police system that in this regard at least makes no distinction between the innocent and the guilty. And on a grand scale. The full version[pdf] of the RRT report states that
Over half a million innocent people (people not convicted, reprimanded, given a final warning or cautioned, and with no proceedings pending against them) – including over 39,000 children – are now on the database
and to no good effect
doubling the number of people on the database from about 2m to about 4m has not increased the proportion of crimes solved using DNA, which remains steady at about 1 in 300. Indeed, in 2007 the number actually fell slightly
I'm no longer sure that I know how this is much different from the Stasi bottling seat covers.

Except maybe that we do still have a chance to do something about it. A shame that we must appeal to the European institutions for protection from our own executive. Time to join Liberty. Hope I'm not too late.

Slipping through our fingers?

It was close. So close.

What's the really exiting thing about the Agile development movement? Is it getting to ship working software? That's pretty good. Is it not having to lie about status all the time? That's great too! Is it being able (and allowed, and required) to take true responsibility for our work? Wonderful stuff! But I don't think these are the best part.

I think that the best part is that we were starting to normalise the idea that programmers are valuable knowledge workers who collaborate with other valuable workers in the business as peers. Big chunks of the industry had to (re)learn how to do all that shipping and telling the truth and stuff in order to get to that point, but that's only the means. The end is to be dealt with by our paymasters as if we make a genuine contribution to our businesses. And even that is only a starting point for the really interesting stuff.

And now along come folks shouting about Lean. A lot of them are Agile folks looking (quite properly) for what to do next that's better. And the message seems to be: programmers are operators of a technical workflow, and nothing else. They pull to-do items from a queue (which someone else manages) and work them through various states of a purely technical nature until they can be handed off into production (where someone else creates value using them). Again and again and again forever. And it is claimed that if they are organised this way then they will be more efficient and shipping those to-do items than if organzied in other ways. This is almost certainly true.

In which case it is going to be hard to avoid being shoe-horned into that mode once the development managers wake up to it. Back down into the engine room. Back to being, rather more literally than before, a cog in a machine.

The Agile approach to developing software (or, something a lot like it) is gaining more and more interest because, along with the stuff about getting to look up at the stars it is actually a more productive way to develop than a lot of the incumbent approaches. If Lean is more productive again but doesn't even have that social stuff as a hidden agenda, I'm not sure that we as inhabitants of this corner of the industry will turn out to have done ourselves much of a favour.

Fauxtrovert

It took me a long time to overcome my distaste of blogs. I'm still not a huge fan but writing peripatetic axiom does seem to be better than useless. After a certain amount of prodding I've started to dabble with twitter.

I'm not finding it easy. One problem is that a lot of the time I'm doing things that I'm not supposed to tell anyone else about (because they are commercially sensitive) and a lot of the rest of the time I'm doing things that I just can't believe anyone would be interested to know about. That second part would seem to be a part of the introvert type.

As with blogs in the early days ("I like beer", "isn't my cat cute" etc.) the art of twittering all the miscellaneous stuff of ones' life seem fairly pointless in a way I've found difficult to explain. But now redditor shenpen has expressed it very well for me. The twitter stance would appear often to be not introversion, and not extroversion, but fauxtroversion