Let over Lambda

Let over Lambda is a new self–published book on lisp by Doug Hoyte. I'm not quite sure what to make of it, overall.

It's great to see a book full of straight-ahead programming these days rather than mere system composition. It's especially great to see an extended work dealing with programming in the small. It's great fun to see someone who really likes programming as an activity in its own right exhibit their enjoyment. It's a great pleasure to see assembly language appear in a programming book of the 21st century. I find the curious combination of lisp being at once both a very high level language suited to symbolic programming and being very close to the metal most stimulating. That's a pair of properties that the programming language design mainstream seems to have abandoned. Java, for instance, doesn't feel particularly close even to the JVM.

It's especially great to see a lisp afficionado standing up for vi.

Assembler arises in a couple of spots where the impact of macros, the parsimony of lisp's internal representations and the intelligence of the lisp compiler combine to collapse quite sophisticated source code down to startlingly few opcodes. Which is all very fascinating. So much so that I was inspired to resurrect the SBCL installation on my Mac and go refresh my memory of how cute the disassemble function is. However, it feels to me as if an opportunity has been lost to take that just a bit further and come up with some Programming Perls–like startling observations about performance.

The book builds up to a very interesting exercise in implementing Forth. It's very nicely done and a great illustration of how easy it is to implement one interesting language given another. Lisp/Prolog used to be the canonical pair, I think. This illustration makes a good case for lisp/forth being roughly as illuminating.

Along the way there are several not–quite–consistant claims about what the book is for and the big build up to the alleged principle of “duality of syntax” is a very long run for a very short slide. Again, it feels as if an opportunity to do somehting really startling has been lost. There's a sort of plonking “here it is” presentation of this and other material. It's often good and interesting material, but needs a little bit more development in places.

It's perhaps not so great to have the, what shall I call it? unfettered enthusiasm of the author for lisp, macros and all that they imply coming at you un–moderated. I don't think that a commercial editor would have allowed quite so much polemic to make it onto the page. There's a bit too much direct quotation of Paul Graham material (“Blub”, “secret weapon”, you know the sort of stuff) that makes it quite clear that there are on the one hand people who get it, and on the other dullards. This is made very explicit on the back cover blurb:

Only the top percentile of programmers use lisp and if you can understand this book you are in the top percentile of lisp programmers.

Hmm. I have a strong feeling that I understand most of what's in the book and also that I'm not in the top of the top. Whatever that means. I'm not even a “lisp programmer” in any very serious sense of the term. Faced with a little light programming to do then in the absence of any other constraints I'm likely to reach for Scheme—and that brings me to another item that a commercial editor probably wouldn't have let through.

You might imagine that the differing trade–offs made in Scheme and Common Lisp are something that reasonable people could agree to disagree about. Hoyte wants his reader to understand very clearly that this is not so: the choices made in Scheme are wrong (emphasis in the original) and those made in CL are right (emphasis also in the original). The first one of these assertions was amusing enough. The second, not so much. And they just keep on coming. Hoyte is far too young to be settling scores from some X3J13 puch–up, wich would be embarrasing enough, so it all ends up looking a bit adolescent to me.

One last thing...at least in the print–on–demand form I've got from Lightning Source UK the book looks absolutely ghastly. “Made with lisp” says the front matter. Lisp with a lot of help from TeX and that's really not good enough for 2009, not without a lot more tuning than has gone into this. And Lightning Source (or whomever did the camera-ready copy) have originated the work at too low a resolution. That and the lazy choice of CMR combined with the glossy toner makes the actual print a less than comfortable read. Self–publishig has a long way to go yet.

Genius at work

Over the years there has been a lot of propaganda claiming that Google both creates the only working environment in which double-super-genius types can function at peak effectiveness and then goes out of its way to hire only double-super-geniuses to work in it. How, then, does this happen:

We periodically receive updates to that list [of malevolent sites] and received one such update to release on the site this morning. Unfortunately (and here's the human error), the URL of '/' was mistakenly checked in as a value to the file and '/' expands to all URLs.

This sort of thing happens because of bad processes. It doesn't matter how smart folks are, if they work in a bad way, poor results will ensue.

Of course, I don't know exactly what occurred, but I'll bet you a pound that one lone double-super-genius is left to make changes to this file by themselves. 

Update: Jonathan Wolter raises some questions about testing such changes.

Self: the movie

My recent posting on Self provoked some interest on reddit and one keen redditor found the old Self movie. I first learned about Self when one of the giant pulsing brains from Manchester university came down to a BCS meeting in London (When did the BCS stop doing things that interesting? Why?) and showed us this movie. Great days.

Dynamic vs static: once more with feeling

In what feels to me like a voyage through a time-warp to the beginnings of my programming career in the mid–90's, Jason Gorman has revived the old static vs dynamic typing debate. 

Oh, how it all comes flooding back: "strong[sic] typing is for weak minds", "static type systems catch the kind of bug that managers understand" etc. etc. etc.

Jason's concerns seem to have been raised by someting to do with this sort of thing (see the part on the var keyword) although he seems to muddle up a language having a dynamic type system with it being dynamic. These aspects are closely related, but are not quite the same (as that post explains reasonably well). It's picking nits of this variety that keeps us all in work. 

Anyway, I very much agree with Jason that this resurgence of interest in dynamicish, scripty lanaguages is driven largely by fashion, and that claims made about it should be closely examined. I'm not so sure about the rest of his argument. Not least because I'm not sure that what he complains about: 

Proponents of such languages cite the relative flexibility of dynamic typing compared to statically-typed languages like Java and C++. Type safety, they argue, can be achieved through unit testing.

is actually being said by anyone. Type safety through unit testing? Really? Maybe someone is, and I haven't seen it. I'd be interested to if anyone has a link.

Personally, I do tend towards dynamic languages and away form manifest static type systems. My preference away from manifest static typing is that (in the words of, IIRC Kent Back) they make me say things that I don't know are true. On the other hand I've been dabbling a bit with Haskell, which has a very strong, very expressive static type system and offers the promise (through type inferencing) to not require all those pesky declarations. That has been both educational and fun. Unfortunately, as Nat pointed out in another context, that lovely promise might not be delivered upon:

[in Haskell] If you don't write explicit type constraints you can end up with a type inference error somewhere in your code. Where? The only way to find out is to incrementally add explicit type constraints (using binary chop, for example) to narrow down where the error is. It's not much different, and no easier, than using printf for debugging C code. 

If this is the best case of static typing, then we have a problem.

Meanwhile, let's consider the distinction between systems programming and application programming. Try googling around the various attempts to nail down that distinction. I don't find any of them terribly satisfactory. For me, the crucial distinction is that the system programmer must allow for any possibly use of their code, whereas an application programmer does not.

This means that in systems land a function declared like f :: int -> int must, unles very carefully specified otherwise, be known to do the right thing at every element of the whole of the cartesian product of the int type with itself. But in application land we might, for example, know that those ints are really the numbers of the days in the week, so we only really need the function to do the right thing over {0,1,2,3,4,5,6}² Demonstrating those two different kinds of correctness require different techniques, I think.

Of course, using int when you mean {0,1,2,3,4,5,6} is a smell. And here is the deodorant.

I want to join these two things up. I want to make a connection between the kind of correctness that systems code needs to have and the way that static typing is might help us with that versus the kind of correctness that application code needs to have and the way that unit testing might help us with that. But I'm not there yet. Watch this space.

Self

So, Self continues to be developed and has a very nice new site, here. Self is like Smalltalk "only more so". 

These days Self runs like a dream (and blindingly fast) on the Mac. Back in the day Self would only run on Sparc hardware, so I bought a Sun workstation for the express purpose of being able to play with Self. A wise investment. If a point of view is worth 80 IQ points then learning Self is a much better bet than all that brain training and smart drinks. 

I you want to understand to powerful (and fun and (partly, therefore) enticing) a system can be if you trust your users to manipulate objects directly, take a look at Self. If you want to understand just how far the notion of Integrating and Development Environment can be pushed (and how much fun that turns out to be) take a look at Self. If you want to understand why some folks consider Javascript to be such a huge missed opportunity, take a look at Self.

If you aren't part of the solution...

At XtC the other night Mark McKergow presented the Solutions Focus approach to...stuff. 

"Solution" here seems to mean those actions observed to take you in the direction of a more pleasing state of the world and is contrasted with the more traditional "problem" which would be the reasons for the mis–match between what you have now and what you want. The suggestion is to focus on the former and overlook the latter. While we could identify some pretty nasty failure modes for this approach it also has a lot going for it. In particular, this line of thinking reenforces the too–little practised technique of recording what went well during your retrospectives and taking an action to do more of that.

Meanwhile two items from the discussion caught my attention.

Gary Player said (as reported by David Anderson) that when a good player makes a great stroke he swings again to fix the memory of what the good stroke felt like, whereas a poor player swings again after a bad stroke to try to identify what went wrong. Given what the learning curve tells us about repetition and consistency this suddenly made Solutions Focus make a lot more sense to me. 

It was observed that a lot of the examples given of the effectiveness of Solutions Focus were concerned with its application to psychotherapy. Asked why this should be Mark opined that therapy is easier than management consulting. Nice.

Examples, Exemplars, Gauges, Tests

At XpDay London 2008 I gave a lightening talk that turned into an open space session about examples an why they work so well. Some folks asked for some more information and some is available in this presentation [pdf] from Agile 2008, and also this old post.

There are some gaps in the presentation where we did exercises. The first is to ask why Jaffa Cakes are cakes and not biscuits. The others encouraged attendees to explore firstly some of the limitations of definitions and then some of the power of examples.

Enjoy.

(And sorry it took me so long to get around to posting this)

An unreasonably high bar?

Some time ago Adam Goucher posted this response to my 5 questions interview with Michael Hunter. There's a few points in there that I want to come back to, but right now the one that's at the front of my mind is this:

Count tests to get a useless number; I can write a million tests that provide useless information but still shows 7 figures in the count.

Well yes, you could. But why would you? We seem to have a hankering in the industry for techniques that would give good results even when badly applied by malicious idiots. That seems unreasonable. And also pointless: I don't believe that the industry is populated by malicious idiots. On the other hand, the kind of answer one gets depends a lot on how a question is asked. 

There is (I read somewhere recently) a principle in economics that one cannot use one number as both a measure of and a target for the same thing and expect anything sensible to happen. [Allan tells me that this is Goodhart's Law --kb] In our world this is the route to the gaming of metrics. I also don't believe that gaming works by folks consciously sitting down and conspiring to fabricate results. I do believe that if we measure, say, test coverage at every check-in and publish it on our whizzy CI server dashboard thingy and have a trend line of coverage over time and we talk a lot about higher coverage being better, or even that test coverage has something to do with "quality" [that would be the "surrogate measure" part of Goodhart's Law --kb] then it is in fact the response of a smart and well intentioned team member to write more tests to get the number up. Even if those tests turn out not to be much use for anything else.

I think (certainly I hope so) that my recommendation to measure scope by counting tests doesn't fall into that trap. Don't write the tests so that you can measure scope. But observe that you can if you write the tests the right way. Of which I shall have a bit more to say later.

Soft-Craft

So, on Tuesday night the programme for Software Craftsmanship was worked out. Looks pretty good, I think: there's plenty of the doing sessions that Jason wanted with a reasonable counterbalance of more reflective activities.

I was quite amused by the metaphor collision that's taking place, though. It seems that some people find that the best way to move forward the craft of software development is to head down the dojo for some sparring. I'm far form convinced that either of those metaphors is much help by itself. To see them embedded in one another makes my head spin.

I look forward to the day when we feel secure enough as an industry to have a programming conference where people's ability to do programming is improved. 

Interview form Agile 2008

At Agile 2008 I was interviewed by Amr Elssamadisy for InfoQ, and the video is now up. 

Skill vs Mastery

Since it's the holidays I get to tourist around London at bit in a way that I tend not to much through the rest of the year. Today I took in the Rothko show at Tate Modern. It's the first time in a long time (maybe ever, or at least since Rothko gave them away) that so many of the colour field paintings have been in the same place at the same time. 

Although these paintings are (at first sight) relatively featureless the viewer is intended to get right up close to them, to feel as if they are in the painting, to be overwhelmed. Although they are not figurative or representational they do carry a lot of content, a lot of emotional impact. There were people sitting in the room where the Seagram murals are, completely transfixed by the paintings. People have been known to burst into tears at the sight of these works.

This sort of modern art is perhaps one of the easiest for the "my four-year-old could do better" school of art critic to avoid thinking about. Of course, their four-year-old couldn't. One room of the exhibition goes some way towards explaining why not. These paintings are on (very) close examination revealed to be extremely complicated objects. Rothko put a lot of technique into these paintings: a lot of careful choice of materials, a lot of careful surface treatments, a lot of careful brush work.  In the language of the curators most of them aren't a "painting" at all. They are mixed media on canvas, so complicated is the structure.

But none of this shows up when one stands before the works, only the emotional effect comes through. That's mastery. As with painting, as Steve says over here with reference to music (and code), it's all about the expressiveness.

Craftsmanship

If you read this blog you might be the kind of person who will be interested in the Software Craftsmanship conference being planned for next year in London. More proposals are needed to round out the programme, so get your thinking caps on.

XP Day London 08

Xp Day is done for another year.

I think that the incorporation of a large Open Space aspect worked, with some notes taken for doing it better in future. The programmed sessions seemed to be well received too.

Notes from many session are going up on the wiki, you might like to look, comment or contribute.

Deployment?

I'm sitting here in Zurich with some colleagues working on some guidelines for teams who are starting to use Agile in an evironment formely exclusively RUP-based. We have a stack of Agile books, all by big-name authors, from which we are farming references for them to use. Not one of these books has anything (so far as we can find) to say about deployment, actually getting the software into the hands of the user. The very word is not in the index of any of them. Oops.

"just work"

There's an idea in  the management world that some work is "just work". I once had a holiday job in a factory that involved a good deal of one kind of "just work"—unpack a box of bottles of nose spray originally destined for one market and re–label them for another, put them into a different box. Well defined tasks, repeated many, many times. At the limit "just work" comes close to the physicist's idea of work: it's what happens when a force displaces its point of application.

Over the last century or so "just work" has been the topic of intensive study. Techniques for organising the execution of "just work" with the goal of maximum output are very well developed. But what is "just work" for programmers? The question arose in the comments attached to this discussion of distributed development. It also relates to some thoughts I've been having recently about the rush to adopt Lean practices in some quarters.
 

Andres says that 

In knowledge work, “just work” is learning. [his emphasis]

If that's true then I think we have to say that a lot of contemporary development activity isn't knowledge work. Wiring up GUI controls? Transducing between HTTP requests and SQL queries? Mashing up web services? You have to learn how to do these things at all, it's true. But what do you learn while doing it once you know how? What learning takes place that is the work? 

I'm reluctant to accept "finding out how to hack your way around the deficiencies of the current framework" as a form of knowledge work. Meanwhile, this maybe does give me a way to think about a curious claim (see comments #3 and #4 on that posting) from the Lean camp:

A pair team has to produce twice as much output at a given level of quality as two solo programmers in order to be worthwhile.

What would we have to believe, in order to think that this claim were true? I think that we would have to believe that the paired programmers are engaged in quite a strong sense of "just work". What is it that a pair of programmers necessarily do half as much of because they are pairing? I can't think of anything but typing. Are keystrokes really what it is that programmers do that adds value? At a slightly higher level of sophistication we might say (as Corey in fact does) that the measure of value added is function points. That's a much better measure than typing (which implies code)

Measuring programming progress by lines of code is like measuring aircraft building progress by weight.—Bill Gates

but still gives me slight pause. Function points are still a measure from the technical solution space, not the business problem space. One of the things that makes me a bit nervous about the rush to Lean is the de–emphasis on conversations with the user/customer (which Leaners have a habit of dismissing as wasteful "planning") in favour of getting tons of code written as fast as possible. 

What I really want to see is getting users empowered to make money (or whatever it is they value) as fast as possible. That's knowledge work. And that requries learning.

Can you tell your POJO's from your NOJO's?

Ivan describes a nice distinction between Plain Old Java Objects in general and Non Object-oriented Java Objects in particular.

Certain (web, in particular) frameworks lead on in the direction of NOJO's. It begins with the DTO (just one kind of NOJO) and spreads from there. Is that good or bad? Can't say in the completely abstract, but if your current framework does lead you that way have you given thought to what design choices and architectural stances your are being lead to? Do you like them?

Another take on interviewing

Very nice posting here from Simon Johnson regarding the selection of candidates for programming jobs.

His blog doesn't allow comments, so I'll follow up here: The only real criticism I'd have of Simon's approach is that the list of desirable qualities in a  candidate is backwards. That is, it should read 

1. The person in front of you will complement your pre-existing team
2. The person in front of you can actually write a program
3. The person in front of you is smart

If the candidate won't complement your team it is of no interest whether or not they can program. Even if they are the semi–mythical 100x productive programmer (hint: those folks are not applying for your position). 

If they can't write a program (of the kind that your company finds valuable) it is of no interest how smart they are. Folk wisdom in many parts of the industry suggests that enough smart can outweigh any other shortfall in a programmer. This is only true in very, very limited and highly specific circumstances (hint: you are not in the business of writing hard real–time life–critical massively–parallel seven–nines embedded control software for the black hole at the centre of the galaxy).

If you can get along with them, and if they can code, smarter is preferable to dumber. Your only problem now is to find some way to recognise "smarter", which turns out to be very, very hard indeed.

Registration open for XP Day London 'O8

Registration for XPDay is now open. XPDay sells out each year, so please book now. We operate a waiting list once the conference is full but only a handful of places subsequently become available.

The full cost of the conference is £350 but the first 50 people to book get a special “Early Bird” price of £275.

The few programmed sessions are set, and we encourage registered attendees to think about the topics they'd like to see discussed in the Open Space sessions. Attendees have the option to prepare a 5 minute "lightening talk" to promote their preferred topics, and these will go into the Open Space topic selection..

I look forward to seeing you there.

Programming Interview Questions

There's a cottage industry devoted to devising questions to ask folks applying for "programming" jobs, to collating such questions and then posting answers to them on the internet. I was reminded of this recently when an announcement of a new Google tool, moderator, came along. The example issue being moderated was "Interview Questions for Software Engineers". At the time of writing the winner (with 4 upvotes and 0 down votes) is my own proposal:

Would you please sit down here at this workstation and write some code with me?

If you are reading this then you have almost certainly been through many an interview for a job that will require some programming to be done. Think back and ask yourself how many times you were offered such a job without ever having been in the same room as a computer and an interviewer.

In my own experience (I've had 9 jobs involving programming, and many more interviews) it's been quite rare. One has to wonder why that is. I suspect that having a candidate do some programming is considered very expensive—and yet hiring the wrong candidate is unbelievably expensive. Hence the questions, which seem to be aimed at finding out if someone will be able able to program without watching them do any.

The second most popular question on the moderator site right as I write is

Given a singly linked list, determine whether it contains a loop or not

I can see what's being aimed at here, and yet it feels as if, in the stress of an interview setting, all the answer given will tell you is whether or not the candidate has seen the textbook answer before. How does knowing that help?

By the way, this is a systems programming question. The vast majority of working programmers don't come anywhere near any such class of problem. They do applications programming. There seems to be a sort of folk wisdom in a large part of the industry that having the skills (and, see above, mere knowledge) of a systems programmer will suite them well to being an application programmer. It's far from clear to me that this is true: the two areas of programming are quite distinct and require different (albeit overlapping) skill sets. I wonder how many interviewers use questions like this to create the impression (maybe even to themselves) that their organisation is involved in sexy, impressive systems work, whether or not that's true?

A while ago we did an informal survey of folks who work for Zuhlke in the UK, asking what had made them apply to us and what had made them subsequently accept an offer. We (the managers) were surprised by how many said that the very recruitment process itself had encouraged them to accept an offer.

This is what we do: Line mangers (such as myself) read CVs. There is an HR department, they don't come anywhere near recruitment. CVs that seem to describe a credible technical career go to the Chairman of the company. The candidates he likes the sound of are invited in for a first interview (of around an hour) to see if they are firstly, someone we'd be prepared to put in front of a client under our brand and secondly, someone we would enjoy working alongside. There's also a short written test of programming knowledge.

Those who make it through that are invited back for an interview with a small panel of our consultants who explore their technical knowledge in depth. First the candidate is asked to give a technical presentation on an interesting piece of work they did recently: stand at a white-board, explain the architecture, the technology choices, how it worked, how the project worked, what they did etc.

Then we set to finding out what they know. We have a checklist of technologies that we've used in projects and the questions take the form "what do you know about $technology?" No tricks, no puzzles. We do a breadth first pass to eliminate the things that the candidate doesn't know (and the process rewards optimistic answers at this point with a lot of embarrassment later on) and then we drill down into each area until at least one of the candidate's or the interviewers' knowledge is exhausted. This goes on for several hours.

Those who make it through that, and people have been known to walk out, get invited back to do some pair programming. That's the point (and not before) at which we decide if we'd like to make an offer.

And yes, this is all unbelievably expensive (I mean, the Chairman does first interviews?) And very, very effective. Our false positive rate is essentially zero and best of all, clients love our engineers and ask for them back again and again. Which makes it all worth while. I can't think of a cheaper way that would actually work.

PS: yes we are hiring. If you'd like a go a this, drop me a line.

97 Things Every Software Architect Should Know

The 97 things project is approaching a milestone: having 97 things which, in the opinion of one self–selecting group (which includes myself), every software architect should know. The intention is that the "best" 97 things go into a book to be published by O'Reilly, but that the forum will continue to be active, to gather new so–called "axioms" and discussion of same. 

What do you think every software architect should know?