My friends know me to be skeptical of “studies” about software development techniques. The main reason for my skepticism is that such studies are rarely undertaken by people who have any understanding of what they are studying. They combine data points from different sets of observations as a way to try and accumulate sufficient information to make charts and trend lines, but often the data points aren’t consistent enough for the aggregated data to be meaningful. I wonder whether many studies of programming techniques are based on enough observations to be meaningful, and whether the researchers’ analysis of the results is based on any direct knowledge of the subject at hand.
Tag: studies
Correlation between high WIP and defects
Empirically (or "anecdotally," if you’re academically-inclined), practitioners know that high work-in-process (WIP) levels tend to lead to a variety of problems. Thanks to Rally Software, we have some hard data to help us make that case with clients. Continue reading Correlation between high WIP and defects
Does pair programming work?
The fact this question continues to come up time and again after all these years prompted me to wonder why the matter hasn’t been settled by now. Thousands of people have tried their hand at pairing in a wide range of circumstances. Some swear by the practice and feel as if something is missing when they must work solo. Others are convinced pairing is pure waste and cannot possibly yield good results. Both opinions are informed by real-world experience. What specific differences in these situations resulted in such radically different outcomes?
I know how to tie my shoes
Recently there have been numerous discussions online about the difficulty of convincing people to try unfamiliar software development techniques that technical coaches and mentors consider useful. The same discussions have been taking place for many years, with no progress. Why is there no answer?
All evidence is anecdotal
It’s a commonplace for people to disparage the direct experience of practitioners and to favor the indirect observations of researchers. This has always struck me as a rather strange attitude. A person who does a thing every day, and whose livelihood depends on doing it well, surely knows what has worked and what has not worked well in practice, at least in his/her own experience. Someone who takes the work seriously will naturally remain open-minded about learning new techniques and methods throughout his/her career, building up a deep and reality-based understanding of the work over time. People working in different domains will have had different experiences and will have solved different problems, so if you listen to several experienced individuals representing different industry sectors you will end up with a pretty good cross-section of practical knowledge.
A practitioner who explains what he/she has found useful in the field is giving you an anecdotal report of his/her experience. Is a researcher giving you any better information when he/she publishes the findings of a study? Isn’t the researcher giving you, for all practical purposes, an anecdotal report of his/her experiences in preparing a research paper? Both forms of evidence are anecdotal, are they not? Would you really dismiss the opinions of a 35-year veteran of software development in favor of a report prepared by a couple of graduate students who had never written a line of code in anger, and whose main purpose in doing the study had been to learn how to “do a study?” Are the hundred-odd real-world projects the veteran has completed really less informative than the two or three sets of apples-and-oranges observations the graduate students made in their study? Anecdotal evidence is still anecdotal, even after someone publishes it. The question is whose anecdotes carry more weight.
Continue reading All evidence is anecdotal