Complexity and Test-first 3

The story came from here.

At XPDay last year I ran a session exploring some of the ideas about complexity and TDD that I've discussed here before. Attendees at the session got a very early copy of a tool, "measure", which calculates some of the statistics I'm intersted in. From that came some good feedback which I've incorporated into a new version of the tool, avilable here. Note to XPDay attendees: this is a much more informative bit of software than the one you have, I suggest you upgrade.

The session will be repeated at Spa this year.

New Tool Features

Measure now shows the parameters for both a Zipf and Pareto distribution modelling how the complexity is found in the codebase. It also shows the R-squared value for the regression through the data points. This was a particular request from the XPDay session and, with due care and attention, can be used to get some idea of how good a fit the distributions are.

Measure also now accepts a -h flag and will display some information about how to interpret the results.

New Results

Using this updated tool, I examined some more codebases (and reëxamined some from before). This table table shows the Zipf distribution paramters. It's getting to be quite difficult to find Java code these days that doens't come with tests, so some of the codebases with an N shown there are ones from SourceForge that haven't had a commit made for several years.

Codebase	Slope	Intercept	R^2	Automatedunit tests?
Jasml 0.1	0.95	3.52	0.73	N
Jruby 0.9.2	1.47	6.27	0.76	Y
Smallsql 0.16	1.54	5.87	0.80	Y
Sunflow 0.06.3	1.59	5.67	0.90	N
m-e-c scehdule α3-10	1.69	4.94	0.92	N
NanoXML 2.2.1	1.77	5.26	0.94	N
Syncbuilder1999	1.84	5.93	0.9	N
Itext 1.4.8	1.88	7.99	0.91	N
Xcool 0.1	1.93	4.60	0.84	N
Ant 1.7.0	2.25	8.90	0.93	Y
Jfreechart 1.0.3	2.30	8.58	0.94	Y
MarsProject 2.79	2.33	7.9	0.96	Y
Log4j 1.2.14	2.43	7.34	0.96	Y
Junit 3.8.1	2.49	5.99	0.94	Y
Jmock 1.1.0	2.79	6.38	0.96	Y
Spring 2.0.1	2.78	9.70	0.96	Y

So, the lower slope and higher incercept of JRuby was a surprise. But note also that the R-squared is quite low. This causes me to refine my thinking about these (candidate) metrics. A low R-squared means that the linear regression through the (log-log) points describing the complexity distribution is not good. This suggests that the actual distribution then is not modelled well by a Zipf type of relation. Maybe there's something about language implementations that produces this? It's worth checking against, say, Jython.

Meanwhile, those codebases with a high R-squared (say, 0.9 or above) seem to confirm my hypothesis that (now I must say "in the case of codebases with very strongly Zipf distributions of complexity) having tests produces a steeper slope to the (log-log) linear regression throught the distribution. And it looks as if a slope of 2.0 is the breakpoint. Still lots more to do though.

Further Investigation

I'd be particularly interested from anyone who finds a codebase with an R-squared of 0.9 or more and tests, but a slope less than 2.0

The discovery of codeases whos complexity distribution is not well modelled by Zipf is not a surprise. The fact that such codebases do not match the tests => steeper slope hypothesis is very interesting and suggests strongly that the next feature ot add to measure is a chart of the actual complexity distribution to eyball.

Note to publishers of code

This sort of research would be impossible without your kind generosity in publishing your code, and I thank your for that. If your code appears here, and has been given a low intercept, this should not be interpreted as indicating that your code is of low quality, merely that it does not exibit particularly strongly the property that I'm investigating.

The Story continues here.

Thanks to Vin for the table formatting fix