All evidence is anecdotal

It’s a commonplace for people to disparage the direct experience of practitioners and to favor the indirect observations of researchers. This has always struck me as a rather strange attitude. A person who does a thing every day, and whose livelihood depends on doing it well, surely knows what has worked and what has not worked well in practice, at least in his/her own experience. Someone who takes the work seriously will naturally remain open-minded about learning new techniques and methods throughout his/her career, building up a deep and reality-based understanding of the work over time. People working in different domains will have had different experiences and will have solved different problems, so if you listen to several experienced individuals representing different industry sectors you will end up with a pretty good cross-section of practical knowledge.

A practitioner who explains what he/she has found useful in the field is giving you an anecdotal report of his/her experience. Is a researcher giving you any better information when he/she publishes the findings of a study? Isn’t the researcher giving you, for all practical purposes, an anecdotal report of his/her experiences in preparing a research paper? Both forms of evidence are anecdotal, are they not? Would you really dismiss the opinions of a 35-year veteran of software development in favor of a report prepared by a couple of graduate students who had never written a line of code in anger, and whose main purpose in doing the study had been to learn how to “do a study?” Are the hundred-odd real-world projects the veteran has completed really less informative than the two or three sets of apples-and-oranges observations the graduate students made in their study? Anecdotal evidence is still anecdotal, even after someone publishes it. The question is whose anecdotes carry more weight.

At the Agile 2007 conference in Washington DC, I scheduled my activities so that I could attend some of the talks on the research track. I felt there was too little cross-pollination between practitioners and researchers, and I was curious to know what was happening in research. One presentation I attended reported the results of the study, “A Longitudinal Study of the Use of a Test-Driven Development Practice in Industry”, conducted by Laurie Williams of North Carolina State University, and Julio Cesar Sanchez and E. Michael Maximilian of IBM. I knew that Dr Williams had run one of the very few credible studies of pair programming, back in 2000 at the University of Utah, so I had high hopes that this study would provide some useful insights about TDD. The IBMers were professional researchers with good knowledge of the field, and not graduate students who were just learning how to do studies. Max was the presenter.

The researchers noted a correlation between the use of TDD and the control of cyclomatic complexity in the resulting code. This observation is hardly news to anyone who writes software for a living. In our informal discussion after the presentation, I was quite astonished at how pleased the researchers were with themselves; they kept saying they had “discovered” the correlation, and that no one had ever noticed it before. It was a breakthrough! I pointed out that this was already well known to practitioners; it was one of the reasons we choose TDD as a preferred software development practice. If TDD didn’t give us results like that, we wouldn’t bother to do it. We didn’t wait for a study to tell us so; we knew it to be so based on direct experience. It is a result of incremental refactoring, which is done as part of the basic red-green-refactor cycle. Cyclomatic complexity is just one sort of design debt that we manage through the TDD cycle. They said, well, no one has published the observation before. They excitedly suggested that I write a paper about it.

I felt genuinely puzzled. The publication of a paper does not cause reality to exist. Reality exists first, and then researchers eventually notice it and write it down. The first one to write something down gets to name it after him/herself. Practitioners don’t stop delivering value to customers so that they can write papers. Practitioners don’t wait until someone publishes a paper to “prove” the value of a technique before they try it and decide for themselves whether the technique helps them. If they did, then researchers would never have anything to observe. The discussion drove home the fact a wide gulf exists between software development practitioners and the research community.

Another example is the study, “Investigating the Usefulness of Pair-Programming in a Mature Agile Team,” carried out by Irina Coman, Alberto Sillitti, and Giancarlo Succi at the Free University of Bozen-Bolzano, Italy, and presented at the XP 2008 conference in Limerick, Ireland. The paper is published in the Proceedings of that conference, ISBN 978-3-540-68254-7, Springer-Verlag, pp. 127-136. The study was based on observations of a team of 16 experienced developers at an Italian company over a period of three months. As a way to avoid subjective reporting of when pair programming was taking place, the researchers used a software tool that recorded the times and the focus window at one-second intervals, if someone was actively using the workstation. They discarded the data from 2 of the developers, as they were newcomers who joined the team after the period of study had begun.

There are several problems with this study. Firstly, the researchers only observed one team and only for a limited time. Secondly, the data collection tool could not really tell whether appropriate pair programming practices were in use. Only a qualified human observer would be able to tell whether a pair was engaged in real collaborative work, or if they just happened to be sitting near one another. Thirdly, although the team claimed to be a “mature agile team,” they did not in fact use pair programming in a rigorous, disciplined way. They used it “on an as-needed basis.” In my experience (anecdotal evidence), this is a euphemism for “we’re great programmers and we don’t need to pair in order to produce good code.” This is not the definition of “mature agile team.” They actually did pair programming (or whatever passed for pair programming) less than half the time. This raises questions about whether the observations can tell us anything about the relative value of paired work versus solo work. Thirdly, the researchers were not experienced, professional software developers (past or present), able to relate on a deep level to the activities they were studying. They were young graduate students, learning how to conduct studies, how to apply statistical analysis techniques to their observations, how to format and submit a research paper for publication, and how to deliver a talk in front of an audience. Those are all good things to do, of course. They just don’t result in a study that business people can depend on to understand the value of the programming technique in question.

These are just two examples among many.

One of the criticisms people express about anecdotal information is that any given report reflects just one set of observations interpreted from just one perspective. A formal study, they argue, compiles information obtained from many sources, many observations, many situations. It normalizes the results, removes statistical outliers, and determines general conclusions that may be drawn based on objective criteria. Really? The pair programming study mentioned above reflects just one set of observations. It is based on observations of a single team. It draws conclusions from just one perspective. It represents the anecdotal report of its three authors. The TDD study mentioned previously also only reports on observations of a single project.

A formal paper published in a respected journal seems more impressive than a comment made by a practitioner over beers after work. I have to wonder, though, whether a formal study is any more meaningful than the practitioner’s comments, if even as meaningful.

It’s all anecdotal, when you come right down to it. Whose anecdotes do you trust?