It’s a commonplace for people to disparage the direct experience of practitioners and to favor the indirect observations of researchers. This has always struck me as a rather strange attitude. A person who does a thing every day, and whose livelihood depends on doing it well, surely knows what has worked and what has not worked well in practice, at least in his/her own experience. Someone who takes the work seriously will naturally remain open-minded about learning new techniques and methods throughout his/her career, building up a deep and reality-based understanding of the work over time. People working in different domains will have had different experiences and will have solved different problems, so if you listen to several experienced individuals representing different industry sectors you will end up with a pretty good cross-section of practical knowledge.
A practitioner who explains what he/she has found useful in the field is giving you an anecdotal report of his/her experience. Is a researcher giving you any better information when he/she publishes the findings of a study? Isn’t the researcher giving you, for all practical purposes, an anecdotal report of his/her experiences in preparing a research paper? Both forms of evidence are anecdotal, are they not? Would you really dismiss the opinions of a 35-year veteran of software development in favor of a report prepared by a couple of graduate students who had never written a line of code in anger, and whose main purpose in doing the study had been to learn how to “do a study?” Are the hundred-odd real-world projects the veteran has completed really less informative than the two or three sets of apples-and-oranges observations the graduate students made in their study? Anecdotal evidence is still anecdotal, even after someone publishes it. The question is whose anecdotes carry more weight.
Continue reading All evidence is anecdotal