With the benefit (and bias) of hindsight

This post could be subtitled, "A self-indulgent look at past writings."

In April of 2012, Paul Bowler (@spbowler) found some articles from my old site on the Wayback Machine and tweeted about it. I replied to him, "Thanks for finding that! Some of the articles make me smile; learned a thing or 2 since then. Others I’m glad to have recovered." That prompted him to ask, "Which articles should I avoid?" That’s not a question to be answered in 140 charcters.

The short answer is that people should read whatever they think will be interesting, and then use their own brains to arrive at independent conclusions. I’m not one of those people who insists that others provide published references to support everything they say or write. If anything, I’d rather people did their own thinking, and used published references as references rather than as a substitute for their own brains. I’m not so sure published references are all that special, anyway.

Now for the long answer.

I re-read the articles as a sort of self-check, keeping in mind the idea of hindsight bias. We sometimes tend to assume we always knew what we know now. Of course, we didn’t. We make choices and express opinions based on what we know at the moment, on the situation we’re in at the moment, and on the person we are at the moment. If we are learning from our experiences, then our mindset will evolve and we might not make the same choices or hold the same opinions as we did at various times in the past. Here was a chance for me to re-read myself as if I were reading someone else. Would I agree or disagree with that person? Would I think him stupid or naive? Would my hindsight provide 20-20 vision, or would it be filtered through the lens of my present-day knowledge?

One of the earliest of the articles was published on June 22, 2005. Selling Agile describes a pattern of bottom-up "agile" adoption that had worked well for us at the company where I was employed at the time. The story is based on our first "agile" project, which we undertook in 2002 on a skunkworks basis on behalf of a line of business manager who was fed up with the cost and delay of dealing with the IT department.

The first thing I noticed is that the author wrote agile with a capital A. It was Agile this and Agile that. He was an Agile Evangelist; a True Believer. Why was that so, and why does the same person now write "agile" in lower case and in quotation marks, and avoid the A word when talking about organizational improvement?

The author’s experience up to 2001 was that the IT industry had become progressively more bureaucratic through the 1980s and 1990s. Industry-wide organizational dysfunction had reached such a state that he was actively investigating franchise opportunities, and planning to start a small business unconnected with IT. Some possibilities included a venetian blinds store; a mail and package storefront of the sort that provides street addresses for post office boxes and services to pick up and drop off UPS and FedEx packages; a combination video game, board game, and trading card game store and activity center; a storage organization and interior design service. Anything but the status quo IT industry.

In 2002, something happened that kept the author in the IT field. In the course of investigating ways to improve the IT function at the corporation, the small group of investigators of which the author was a part came across a document on the web entitled, The Agile Manifesto. It seemed to them that it expressed the same fundamental values and principles that they were seeking in their organization. They followed up and learned how to apply the principles from one of the leaders in the field, ThoughtWorks.

They learned well. Beginning with a small-sized project in 2002, they grew an internal "agile" practice to a size of 60-70 people. Not so impressive, maybe, considering the IT department comprised 1,300 people out of a total employee population of 33,000; but still, it was a pretty good start. By the middle of 2005, when the article was written, they had delivered quite a few projects ranging in size from very small to fairly large, out-performing the IT department’s estimated ROI for the same LOB requests by an average of 9x, with the single worst case delivering 4x and the single best 27x the estimate provided by the official IT department. The group was on a roll.

Naturally, the author assumed the approach they had taken was the best approach: Drive improvement from the bottom up and just ignore or go around IT management. Work directly for the LOBs, because business people already understand agile thinking. They have to, because they are responsible for profit centers, while the IT department is a cost center and is managed with cost center thinking — the diametric opposite of agile thinking.

What the author did not know then, and knows all too well today, is that the agile practice at that company was just about to reach its peak at the time he wrote the article. In 2006, IT management decided that the agile initiative had proved itself, and they decided to take "control" of it. They declared it had been their idea all along, and they proceeded to destroy everything about it that resembled the values and principles of the agile movement. By the end of 2006, all but 4 of the people who had been part of the agile practice had left the company. By 2008, most employees had no memory that any such thing had ever happened. In 2009, the company was bought out by a competitor.

By 2012, the author had developed into a person who understands that a bottom-up skunkworks approach may yield some benefit on a limited scale and for a limited time, but for long-lasting and meaningful improvement we have to engage stakeholders at all levels in the organization, and we have to throttle the rate of change so that all the moving parts of the organization can continue to interoperate smoothly during the transition. He also knows it isn’t about "evangelism," but about delivery effectiveness. "Agile" is a source of useful ideas for improving delivery effectiveness, but not the only one.

Another article from later that same year, RUP and XP, strikes me today as complete nonsense. My thinking on that subject (also related to the 2008 article The Iron Triangle is not a dinner bell), has since evolved into a model that considers software delivery processes along three axes: Iron Triangle management, process model, and delivery mode. Should I condemn the me of 2005 for failing to leap ahead in time to a more evolved perspective on processes, or would doing so be an example of hindsight bias?

As of June 1, 2005, the old me was an advocate of "technical stories" to augment user stories. Since that time, I’ve learned it is possible to include non-functional requirements and quality attributes in business-focused user stories. It’s a question of how the stories are crafted. I now see the use of "technical stories" as a process smell that points to opportunities for improvement.

Even so, the basic idea that we shouldn’t hide any work from stakeholders is valid. That’s what I think the former me was really trying to get to. It’s just that the mechanism of "technical stories" isn’t the way to do it.

The 2009 article Managing Non-Functional Requirements in Agile Software Development, seems a much more sensible treatment of non-functional requirements than the 2005 article. That may only be because 2009 is closer to 2012 than 2005 is, so the 2009 me is more similar to the 2012 me than the 2005 me is. The articles represent a progression of thought; to judge them all by a 2012 yardstick would surely be a case of hindsight bias.

Circa 2006, I considered the staggered iterative waterfall anti-pattern a very serious process smell, and I said so in my characteristically verbose way in Slouching towards waterfall. I recall debates with other coaches as recently as 2009 when I defended the same position.

As of 2012, I don’t see it the same way at all. From a Lean perspective, the same pattern can be seen as an effective and pragmatic way to apply the concept of rolling wave planning that fits in nicely with continuous-flow process models such as Kanban.

The pattern does break time-boxed iterative process models such as Scrum, since a potentially-shippable solution increment is not necessarily delivered in each iteration. I no longer think that is necessarily wrong or bad; it depends on the circumstances. Evangelists of first-generation "agile" concepts, like the me of 2005, assume the time-boxed iterative model is the single best model for all circumstances. The problem with the staggered iterative waterfall pattern is simply that it doesn’t conform to the time-boxed iterative model. The judgment circa 2005 did not consider whether it could be effective; just whether it conformed.

I think a few of the old articles are still pretty good. A lesson in enablement from US Joint Forces Command dates from 2005, and I think it’s still very relevant to IT work today. I stand by my 2005 view of the relationship between Six Sigma and agile software development. Yet, I use Six Sigma tools in my work, when they add value. I still see organizations falling into the trap of insitutionalizing work-arounds to problems instead of identifying and addressing the root causes, as described in Pain-killers can be addictive. And the Butterflies and blizzards piece shows an early stage of my journey toward Systems Thinking, which is now central to my approach to organizational change.

I hope that if I re-read my writing from 2012 five years from now, I will find at least half of it to be complete nonsense. That would be a sure sign of ongoing learning.