Optimism

At the Tuesday Club the other week there was a bit of chat about what seems to be a foundational assumption of Agile: that in fact we can successfully execute software development projects, so we should run them in a way that brings about the greatest benefit.

This in contrast to what appears to be the (undeclared) foundational assumption of many other approaches: development project are so hard that we are most likely to fail, so we'd better find a way to do that as cheaply as possible.

This carried across into a panel discussion at the recent Unicom conference, where one of the attendees came up with this summary description of Agile:
Collaborative optimism to solve the right problems (and only the right problems) in an incremental way
I quite like that "collaborative optimism" bit.

Agile Adoption Patterns

I've just done a first pass through Amr Elssamadisy's cracking new book Agile Adoption Patterns. It's a timely work—enough teams have adopted enough Agile practices enough times that there should be patterns by now. And here they are.

There's a lot to like about the book. I especially appreciate the consistent message throughout that we do these practices because of their tie to business value. Not for truth and beauty, not for right versus wrong, but because they make our customers happy through the means of more money sooner, more often. There is an explicit mapping from various enhancements to business value onto clusters of the patterns. There is also mappings by business or process smell (of which there is a good, short catalogue presented). For each kind of business benefit the patterns within each cluster are logically related and given a partial order with respect to how well they help with that goal.

Separately, the clusters of patterns are described themselves as patterns, which is a nice organising principle. Agile Adoption Patterns is quite explicitly a handbook for finding out why your organisation might want to adopt Agile and then building a strategy to make that happen. A damn handy thing to have.

Scattered through the pattern catalogue there are little diagrams relating the patterns. I might wish for more of these. 

The patterns themselves contain a "Sketch" section, in which a surprisingly large cast of characters play out little vignettes. Dave Developer comes to realise the value of stand–up meetings because he sees Scott ScrumMaster [sic] removing his blockers, and so forth. I know that this style of thing is very fashionable these days, but it makes me cringe. Not only that, but I have a terrible memory for names, so keeping these fourteen characters lined up in my head is a strain.

The patterns have the usual "Therefore" section with the things–to–do in it, followed by and "Adoption" section telling you how to get there. They are all quite meaty, concrete and honest.  And best of all is the "But" section telling you common mis–steps, gotchas, misunderstandings and how to recognise same. This is outstanding! A common shorthand description of what a pattern is (in fact, it's in this book too) is words to the effect of a "solution to a problem in a context" Which is sort–of true a far as it goes but has its problems, mostly around the term ''solution". 

Oh, how we love "solutions" in the IT industry. A more accurate but less compelling description what patterns are is that they are mappings form one context to another, in which certain conflicting forces are more agreeably disposed (or resolved) in the latter than in the former. These "But" sections describe common scenarios (I've seen a lot of them, and so have you, I'll bet: Continuous Integration but continuously broken build, anyone?) that you could have and still make some sort of Jesuitical argument to be in agreement with the "Therefore", but actually are very wide of the mark. The next time I write any patterns, they will have a "But" section.

The highest level organisation of the patterns is into those concerned primarily with Feedback, technical practices, supporting practices and the cluster patterns. I was most interested to note that "Coach" is a supporting practice, but the role of ScrumMaster is assumed throughout.

I am one of the latter, but I'd rather be one of the former. Not so much of an option these days. Well, that's the power of marketing for you.

Meanwhile, one of the nicest things about having patterns is having a shared vocabulary. I think that Agile Adoption Patterns will become the standard reference for talking about Agile adoption, which will make my professional life easier and that of may others. A service to the community.

"Opportunities for Improvement"

The pattern form used is has nine sections and I'm not sure it helps to have each section shown in the table of contents. I feel the lack of a ready–to–hand one or two page list of all the patterns by name. Twelve pages of TOC for 320 pages of content seems a bit excessive. The same syndrome shows up in the index. If all the parts of a pattern are within a four page range do they really all need their own individual index entry? This is really a shame as the book otherwise benefits from the several other complementary ways to navigate it.

There is a stunningly egregious formatting botch on page 207. I'm sure this volume will go into multiple printings and I hope that someone at A–W gets that fixed.

The Non–functional in a Functional world?

This post shows an implementation of map/reduce in Haskell. It's a one–liner (if you ignore all the other lines) and begins with this claim:
the type says it all

p_map_reduce :: ([a] -> b) -> (b -> b -> b) -> [a] -> b

The type says a lot, that's for sure. I says that map/reduce results in a value of type b when given a list of values of type a, a function that gives one b given two b's and a function that gives one b given a list of a's. That is a lot of information and it's impressive that it (at least) can be stated on one line. Is it "all", though?

Even allowing for some inter–tubes posting hyperbole "all" is a very strong claim. A Haskell programmer should know that. If one already knows what map/reduce does then the type tells you a lot about how to use this implementation of it. And if you don't, maybe not so much.

But that's not what caught my attention. The point of map/reduce is to recruit large pools of computational resource to operate on very large data sets efficiently. The (14 line, in fact—which is still quite impressive) implementation uses Haskell's Control.Parallel features to chop the list of a's into 16 chunks. That's not in the type. The intention is to have enough lumps of work to 
bump all your cores to 100% usage, and linearly increase overall performance
which it may well do on a large enough data set, if I have fewer than seventeen cores. Currently, I do. There are nine cores in my flat right now. If my other laptop were here, it'd be eleven. But if I'm planning to use map/reduce in the first place, maybe I'm going to recruit all the machines in the office, which would currently be worth over twenty cores. This map/reduce isn't going to pin all of them.

Would Haskell's type system support giving p-map-reduce a dependent type with the number of worker processes in it? I don't know enough about it to tell. And why would we care? Well, that number 16 is a very significant part of the implementation of that function. 

The only reason we'd bother to use map/reduce is because of its non–functional characteristics (as the properties of software that make it actually useful are called) and these are no–where to be seen in its type as a function. 

Computer Weekly IT Blog Awards

The shortlist for the Computer Weekly IT Blog Awards 2008 is just out.

This blog was nominated (most kind, thanks), as were those of my colleagues Ben and Daryn, but Daryn's has made it onto the shortlist. This is fitting recognition for the fine work he has put into his blog, particularly his series of Ruby/Rails tutorials which are gathering some recongnition in that world. Feel free to go and vote for him!

XPDay London Session Submission Process

If you want to propose a session for XP Day London '08 (and I hope that you do), then here's what you need to do.

We want the conference to be largely self–organizing this year, so the submission process is suitably light weight.

Experience Reports
Please send a very brief outline of your report to submissions2008@xpday.org You will receive a URL to a google document. Make sure that you include an email address corresponding to a google account (or let us know if the address you send from is the one to use). The document will be writeable by you and the committee and all other submitters. Develop your report there. As in past years we invite the community to collaborate together to help maximize the quality of all submissions. After the close of submissions the committee will make a selection of experience reports to be presented at the conference. The selected reports will be shepherded between acceptance and the conference.

Conference Sessions
Most of the duration of the conference (other than experience reports) will be OpenSpace. There will be Lightning Talks early in the day where people can publicize topics that they would like to discuss. We encourage people to engage in this way. There will be optional rehearsals at XtC in the weeks running up to the conference. You do not need to attend these to make a lightening talk at the conference. If you would like to have such a rehearsal, send a title and a google account email address to submissions2008@xpday.org You will be allocated to a rehearsal slot. We invite groups outside London to schedule rehearsals too, and are happy to help with scheduling those. There will be a google calendar with the slots shown.

Some kinds of session can benefit from some preparation of materials and some logistics on the day. For example, a workshop involving some sort of server, or extensive materials. There will be a limited number of time slots during the conference for such sessions. Please submit a brief outline to submissions2008@xpday.org indicting an email address associated with a google account. You will be sent the URL of a google document within which to develop your proposal in collaboration with other submitters. After the close of submissions these proposals will be assessed by the committee and suitable sessions will be selected for the program.

We have decided to extend the submission period, so submissions for experience reports, rehearsals and programmed sessions now closes on Friday August 8th 2008

This year we want to de–emphasize sessions introducing Agile or Scrum or XP or TDD or... and promote topics of current interest to practitioners. We want the conference to be a forum in which the state of the art is advanced. That doesn't mean that only experts are welcome, or welcome to present. Experts have their failure modes, simple questions from journeymen often reveal the essence. 

All are welcome, so get your thinking caps on!

TDD, Mocks and Design

Mike Feathers has posted an exploration of some ideas about and misconceptions of TDD. I wish that more people were familiar this story that he mentions:
John Nolan, the CTO of a startup named Connextra [...] gave his developers a challenge: write OO code with no getters. Whenever possible, tell another object to do something rather than ask. In the process of doing this, they noticed that their code became supple and easy to change.
That's right: no getters. Well, Steve Freeman was amongst those developers and the rest is history. Tim Mackinnon tells another part of the story. I think that there's actually a little bit missing from Michael's decription. I'll get to it at the end.

A World Without Getters

Suppose that we want to print a value that some object can provide. Rather than writing something like statement.append(account.getTransactions()) instead we would write something more like account.appendTransactionTo(statement) We can test this easily by passing in a mocked statement that expects to have a call like append(transaction) made. Code written this way does turn out to be more flexible, easier to maintain and also, I submit, easier to read and understand. (Partly because) This style lends itself well to the use of Intention Revealing Names.

This is the real essence of TDD with Mocks. It happens to be true that we can use mocks to stub out databases or web services or what all else, but we shouldn't. Not doing that leads us to write code for each sub-domain within our application in terms of very narrow, very specific interfaces with other sub-domains and to write transducers that sit at the boundaries of those domains. This is a good thing. At the largest scale, with functional tests, it leads to hexagonal architecture. And that can apply equally well recursively down to the level of individual objects. 

The next time someone tries to tell you that an application has a top and a bottom and a one-dimensional stack of layers in between like pancakes, try exploring with them the idea that  what systems really have is an inside and a outside and a nest of layers like an onion. It works remarkable wonders.

If we've decided that we don't mock infrastructure, and we have these transducers at domain boundaries, then we write the tests in terms of the problem domain and get a good OO design. Nice.

The World We Actually Live In

Let's suppose that we work in a mainstream IT shop, doing in-house development. Chances are that someone will have decided (without thinking too hard about it) that the world of facts that our system works with will live in a relational database. It also means that someone (else) will have decided that there will be a object-relational mapping layer, based on the inference that since we are working in Java(C#) which is deemed by Sun(Microsoft) to be an object-oriented language then we are doing object-oriented programming. As we shall see, this inference is a little shaky.

Well, a popular approach to this is to introduce a Data Access Object as a facade onto wherever the data actually lives. The full-blown DAO pattern is a hefty old thing, but note the "transfer object" which the data source (inside the DAO) uses to pass values to and receive values from the business object that's using the DAO. These things are basically structs, their job is to carry a set of named values. And if the data source is hooked up to an RDBMS then they more-or-less represent a row in a table. And note that the business object is different from the transfer object. The write-up that I've linked to is pretty generic, but the inference seems to be invited that the business object is a big old thing with lots of logic inside it.

A lot of the mechanics of this are rolled up into nice frameworks and tools such as Hibernate. Now, don't get me wrong in what follows: Hibernate is great stuff. I do struggle a bit with how it tends to be used, though. Hibernate shunts data in and out of your system using transfer objects, which are (lets say) Java Beans festooned with getters and setters. That's fine. The trouble begins with the business objects. 

Irresponsible and Out of Control

In this world another popular approach is, whether it's named as such or not, whether it's explicitly recognized or not, robustness analysis. A design found by robustness analysis (as I've seen it in the wild, which may well not be be what's intended, see comments on ICONIX) is built out of  "controllers", big old lumps of logic, and "entities", bags of named values. (And a few other bits and  bobs) Can you see where this is going? There are rules for robustness analysis and one of them is that entities are not allowed to interact directly, but a controller may have many entities that it uses together. 

Can you imagine what the code inside the update method on the GenerateStatementController (along with its Statement and Account entities) might look like?
Hmmm.

Classy Behaviour

Whenever I've taught robustness analysis I've always contrasted it with Class Responsibility Collaboration, a superficially similar technique that produces radically different results. The lesson has been that RA-style controllers always, but always, hide valuable domain concepts.

It's seductively easy to bash in a controller for a use case and then bolt on a few passive entities that it can use without really considering the essence of the domain. What you end up with is the moral equivalent of stored procedures and tables. That's not necessarily wrong, and it's not even necessarily bad depending on the circumstances. But it is completely missing the point of the last thirty-odd years worth of advances in development technique. One almost might as well be building the system in PRO*C

Anyway, with CRC all of the objects we find are assumed to have the capability of knowing things and doing stuff. In RA we assume that objects either know stuff or do stuff. And how's a know-nothing stuff-doer get the information to carry out its work? Why, it uses a passive knower, an entity which (ta-daaah!) pops ready made out of a DAO in the form of a transfer object.

And actually that is bad.

Old Skool

Back in the day the masters of structured programming[pdf] worried a lot about various coupling modes that can occur between two components in a system. One of these is "Stamp Coupling". We are invited to think of the "stamp" or template from which instances of a struct are created. Stamp coupling is considered (in the structured design world) one of the least bad kinds of coupling. Some coupling is inevitable, or else your system won't work, so one would like to choose the least bad ones, and (as of 1997) stamp coupling was a recommended choice.

OK, so the thing about stamp coupling is that it implicitly couples together all the client modules of a struct. If one of them changes in a way that requires the shape of the struct to change then all the clients are impacted, even if they don't use the changed or new or deleted field. That actually doesn't sound so great, but if you're bashing out PL/1 it's probably about the best you can do. Stamp coupling is second best, with only "data"  coupling as preferable: the direct passing of atomic values as arguments. Atomic data, eh? We'll come back to that.

However, the second worst kind of coupling that the gurus identified was "common coupling" What that originally meant was something like a COMMON block in Fortran, or global variables  in C, or pretty much everything in a COBOL program: just a pile of values that all modules/processes/what have you can go an monkey with. Oops! Isn't that what  a transfer object that comes straight out of a (single, system-wide) database ends up being? This is not looking so good now.

What about those atomic data values? What was meant back in the day was what we would now call native types: int, char, that sort of thing. The point being that these are safe because it's profoundly unlikely that some other application programmer is going to kybosh your programming effort by changing the layout of int
And the trouble with structs is that they can. And the trouble with transfer objects covered in getters and setters is that they can, too. But what if there were none...

Putting Your Head in a Bag Doesn't Make you Hidden

David Parnas helped us all out a lot when in 1972 he made some comments[pdf] on the criteria to be used in decomposing systems into modules
Every module [...] is characterized by its knowledge of a design decision which it hides from all others. Its interface or definition was chosen to reveal as little as possible about its inner workings.
Unfortunately, this design principle of information hiding has become fatally confused with the implementation technique of encapsulation. 

If the design of a class involves a member private int count then encapsulating that behind a getter public int getCount() hides nothing. When (not if) count gets renamed, changed to a big integer class, or whatever, all the client classes need to know about it. 

I hope you can see that if we didn't have any getters on our objects then this whole story unwinds and a nasty set of design problems evaporate before our eyes. 

What was the point of all that, again?

John's simple sounding request: write code with no getters (and thoroughly test it, quite important that bit) is a miraculously clever one. He is a clever guy, but even so that's good. 

Eliminating getters leads developers down a route that we have known is beneficial for thirty years, without really paying much attention. And the idea has become embedded in the first really new development technique to come along for a long time: TDD. What we need to do now is make sure that mocking doesn't get blurred in the same was as a lot of these other ideas have been. 

First step: stop talking about mocking out infrastructure, start talking about mocks as a design tool.

Image

Rick Minerich asks the question "Why are our programs still represented by flat files?" To which my answer is "not all of them are." 

Judging by the tag cloud on his blog, Rick works mainly in the Microsoft end of the enterprisey world. If VisualStudio is the main front end to your system then I can see why the question would arise. Of course, in other parts of the IT world programs stopped being in flat files a long time ago. Or even never were. But you have to go looking for that world. As luck would have it, Gilad Bracha has just recently explained some of the very pleasant aspects of that world. But so many programmers either don't know it exists, or once shown it refuse to enter.