Where are all the Agile success stories?

Daniel Mezick confronts the elephant in the “agile” room in his post, Deviation from the Norm: “If current approaches actually worked well, then by now, thousands of organizations would have reached a state of self-sustaining, “freestanding” agility. Clearly, that is not the case.”

Pondering the question, several possible reasons for this result (or lack of) occurred to me. These are speculative and based on my own experience and observations.

These reasons pertain to agile proponents working as change agents in organizations.

1. Evangelism. Agile coaches and consultants “believe in” agile. This is reflected in their language: “adoption,” “buy-in,” “transformation,” “overcoming resistance,” etc. This is the language of sales. They know what they want clients to do before they ever set foot in the door, before they ask any questions. They approach clients with the goal of causing them to “adopt” something the coach or consultant believes in, rather than with the goal of helping the client identify and solve their problems and achieve their goals. Externally-defined and -imposed goals do not stick because the client does not own them. People more-or-less follow along while the consultant is on site because it is the Next Big Thing their management has instructed them to do. Last year there was a different Next Big Thing, and next year there will be another.

2. Form over substance. Agile evangelists talk about values and principles over methods and practices, yet they often focus on the adoption of specific process frameworks and technical practices rather than on software quality and delivery of value. They carry out assessments to determine “how agile” an organization is, rather than how effective it is at delivering value. They track the use of specific practices like test-driven development and continuous integration, rather than factors related to value delivery like time-to-market and business alignment. They focus on seating arrangements rather than on genuine collaboration, and on busy walls rather than useful information radiators.

3. Unrealistic change model. Many coaches and consultants believe in the viral model of change adoption. They believe that if a single team or a small group of teams can demonstrate improvement through agile methods, others in the organization will be inspired to change the way they work, as well. In large organizations, the forces holding the status quo in place are so numerous, so subtle, so complex, so pervasive, and so powerful that small-scale attempts to demonstrate a better way are quickly snuffed out. Agile evangelists talk about complex adaptive systems, yet seem not to recognize that their clients’ organizations are complex adaptive systems; or perhaps they do not understand that a complex adaptive system will not shift into a different state as a result of a point change like introducing agile methods to a team.

4. Insufficient guidance. In many cases, coaching support for unfamiliar ways of thinking, unfamiliar processes, and unfamiliar practices stops after an initial introduction. The assumption appears to be that people will “wake up” and become inspired to drive forward with continuous improvement, guided by a deep understanding of “agile” values and principles that they internalized fully during a training class and a few weeks of part-time coaching. A further assumption appears to be that the transformed team will muscle its way through challenging, entrenched organizational, managerial, psychological, and technical barriers they are ill-equipped to handle, week after week, month after month, with no ongoing support, while under pressure from management, stakeholders, and peers to stop fooling around and get back to business as usual.

5. Level of engagement. Companies engage management consultants – not agile coaches – at the C level to effect organizational transformation. They engage agile coaches and consultants within the IT department to introduce agile processes and practices. Usually, these engagements are effectively invisible and inconsequential from the perspective of C level management. The agile coaches and consultants do not interact with organizations at the level of management necessary to effect genuine organizational transformation. While many agile consultants would like to engage at a higher level so that they might have a chance to influence large-scale organizational improvement, most of them lack the experience or credentials to advise the senior management of a large enterprise.

6. Excessive language. Agile coaches and consultants often use extreme language to describe the problems they want to solve and the results of their efforts to solve them. They use the word “organization” to refer to a small subset of an organization, such as a software development group or even just a single software development team and its immediate stakeholders; they use the word “failure” when they only mean that things could be better, or that people can learn from the outcomes they achieve; they use the word “transformation” when they only mean some degree of change at a small scale (probably temporary). Thus, the language one hears and reads from “agile” sources sounds bigger than the reality, and reduces credibility.

These reasons pertain to observed patterns of behavior in large organizations.

7. Misunderstanding how change happens. More often than not, management does not understand the J-curve effect when change is introduced. They expect (and often demand) that delivery performance continue at the current level from the very beginning of a change initiative. It is normal for performance to drop initially, and then to climb above the previous level (assuming the change was in fact an improvement). Most organizations do not allow time for this to occur before they give up on the proposed change. They see the initial drop in performance and panic. In some cases, they abandon the entire improvement initiative when the first thing they try doesn’t work out well. Sometimes the managers involved in the agile transformation understand the J-curve effect, but they are overruled by senior management and/or business stakeholders. (Point #6 above might exacerbate this problem.)

8. Non-systemic thinking. Often, the assumption is that delivery performance can be improved dramatically by introducing a few simple changes in technical practices or by wrapping current methods in a nominally-agile process framework (or at least the easy parts of such a framework). The numerous inhibitors – dependencies on non-agile working groups with long lead times for critical services, inadequate test environments or test data management, functional silos with separate management hierarchies, third-party software packages whose architecture prevents incremental delivery, recruitment methods that fail to identify suitable candidates, the use of low-cost offshore services for development or testing, “rank-and-yank” employee performance evaluation schemes, lack of engagement in IT initiatives by business stakeholders, a focus on resource utilization over throughput – are left untouched.

9. Management turnover. Only a vanishingly small proportion of “agile” change initiatives at large companies have lasted more than a year after the departure of the consultants who introduced it. I’ve observed a pattern that may partially explain this. An “agile” initiative is usually undertaken by a manager who is new to the company, and who has a mandate to improve delivery performance. He/she got the job by convincing senior management that “agile” is the way to accomplish the goal. Using the transformation program as a springboard, he/she moves on to a new career opportunity just as the initiative is gaining traction, while it still looks good. The next manager to join the organization has a very different view of how to achieve good delivery performance, and besides that he/she wants to put his/her personal mark on the job by erasing whatever his/her predecessor has done. “Goodbye, agile;” or perhaps, “See you later, agile,” as this manager will also move on in a year or two.