Some thoughts on scaling agile software delivery

Premise: Agile principles depend on biologically hard-wired limits on the effective size of collaborative human groups. For this reason, agile methods as such do not scale to “enterprise” levels.

A theoretical notion

Human groups that support the hunter-gatherer lifestyle of our ancestors range in size from about 5 to about 150 individuals; the sizes of “bands” and “tribes.” This is the way humans are evolved to function. Research in neuroscience suggests there are natural limits to the number of people an individual can “know” and care about. Agile values and principles are based on the level of collaboration that occurs naturally in small groups, compatible with human nature.

Large political organizations and advanced technology are very recent changes in comparison with the time scale of human evolution. Humans are not evolved to function naturally at scale. They can do so only when artificial mechanisms are in place to “manage” their activities.

As a group grows beyond about 150 people, individuals become anonymous and emotionally disconnected from one another. Things like “crime” and “injustice” become possible, and they increase in direct proportion to group size. Formal mechanisms to coordinate group work become necessary. Less-formal mechanisms are workable only for groups whose size is consistent with homo sapiens’ nature as a collaborative hunter-gatherer species that operates in small groups.

Traditional methods of managing large-scale enterprises descend directly from organizational methods that were developed to manage very large groups of humans, starting with the agricultural revolution approximately 8,000 to 10,000 years ago. These methods were developed to enable large groups of people to function in an organized way. Scientists think this became necessary when desertification reduced the availability of food in northern Africa, creating a need for people to cultivate and store food. Political systems came into existence to manage the large groups of people needed to carry out this work. Contemporary organizational structures and management methods are essentially the same as the ones our ancestors created in the era of the agricultural revolution.

Anecdotal observations

A self-contained, collocated team of 2 to 4 people does not benefit from any formal process, including a lightweight process.

Lightweight processes like Scrum, XP, Crystal Clear, Naked Planning, and Mob Programming work best with a team of 7 plus-or-minus 2 members, provided the team is self-contained (has no dependencies on other teams or external resources). Collocation makes things better, but the team can be distributed or dispersed, provided they use appropriate technologies for collaboration.

Agile teams that reach a size of 12-15 people split themselves along the natural “seams” in the work, whether we want them to split or not. The sub-teams then evolve mechanisms to interface with one another, with or without explicit acknowledgement that the sub-teams exist, and with or without any official way to manage their interaction. Left unattended, this phenomenon can cause problems in software delivery.

Work groups larger than about 15 people cannot sustain a lightweight method like the ones mentioned above. They can manage their work using a flow-based process such as Kanban up to a group size of around 40 people, but they do not function as an integrated, cohesive “team” in the full sense of the word. They are more accurately described as a “work group.” First-generation agile methods do not address this, except for the problematic mechanism called “Scrum of-Scrums” in the Scrum framework.

Frameworks for scaling agile methods to support enterprise IT, such as SAFe, DAD, and various home-grown frameworks, have served well for working groups of up to about 150 people. As of 2014 these frameworks have not been applied to larger groups, so we don’t know what the size limit may be. My guess is that 150 will prove to be a practical upper limit beyond which the work flow will become effectively the same as with traditional processes. I speculate this is so because 150-ish seems to be a biological limit hard-wired into the human brain. A group of more than 150 people has no chance of operating as a true “team,” and their work has to be coordinated artificially (that is, learned behavior vs. evolved behavior) and indirectly (that is, via intermediaries to coordinate the work, such as “managers” to enforce heavyweight processes).

As the size of the work group increases, the level of “true” agility they can sustain declines. Agile values and principles disappear above a group size of about 15. Agile processes and practices can survive at the level of individual development teams that are part of a larger structure, but they do not operate across the larger structure itself. As the proportion of time spent in process management tasks grows relative to the time spent in value-add work, teams lose progressively more and more agility. At some size threshold, an “agile” organization is the same as a traditional one, except for superficial elements such as jargon, job titles, furniture arrangement, and wall decorations.

Scaling agile

Many people believe the values and principles that form the basis of the agile school of thought are applicable to any domain and can be scaled to any size. The idea is expressed very nicely in this statement of principles: http://scaledprinciples.org/.

In practice, agile software development methods are usually implemented mechanically; most people never adopt (or even properly understand) the values and principles. In addition, most people do not mindfully use the values and principles as a guide to craft a process tailored to their needs; they simply treat their chosen process as a set of rules to be followed. While I appreciate the passion of agile purists, I am more interested in the things people actually do in practice. In practice, nearly every “agile” implementation is based on traditional thinking, and agile practices are mixed in with traditional ones. These implementations often result in some improvement, but they do not reach the level of improvement agile enthusiasts hope to see.

To help people obtain value from a mechanical implementation of agile methods, a couple of commercial scaled agile frameworks have become popular in recent years: SAFe (Scaled Agile Framework) and DAD (Disciplined Agile Delivery). Many similar home-grown frameworks have been developed inside various companies. These frameworks “scale” agile by re-introducing traditional governance and management mechanisms to coordinate the work of multiple small teams, to support risk management, and to integrate agile activities with non-agile operations across the enterprise. Canonical first-generation “agile” methods such as Scrum and XP are team-focused and do not address organizational issues explicitly.

It is also possible to “scale” agile by applying a process-improvement approach (Kaizen, Kanban Method, etc.), and letting agile practices fall into place naturally wherever they add value. With this approach, the goal is not specifically to scale agile, but rather to achieve business goals and solve delivery problems. Scaling occurs as a side-effect of pursuing these objectives.

Traditional elements in SAFe include PSI Planning, HIP Sprints, and System Teams. “Agile” methods as such are applied at the level of individual teams; SAFe suggests Scrum at that level, and teams can also use other lightweight methods as appropriate (XP, Crystal Clear, Kanban, etc.). I think this is proper, because agile methods are designed to take advantage of human nature, and humans are evolved to function most effectively in small, collaborative groups.

The definition of DAD is a bit more open about its incorporation of multiple approaches. For example, this introduction http://disciplinedagiledelivery.wordpress.com/introduction-to-dad/ describes DAD as a “hybrid framework” with a focus on end-to-end delivery. “It is a three-phase lifecycle where you incrementally build a consumable solution over time.” This is pretty similar to SAFe and home-grown corporate frameworks, except it is explicit about where “agile” fits into the big picture, rather than claiming everything “good” is to be labled “agile” by definition. In fact, DAD lists SAFe among the optional building blocks you can choose when crafting your delivery process.

Not everyone uses agile scaling frameworks in a mechanical way. Here is one example of an organization that mindfully adapts SAFe to support their needs: http://www.prettyagile.com/2014/02/from-psi-planning-to-unity-day-and-back.html. At that company, people mindfully adapt SAFe and adjust their practices as their needs change and as they mature in the use of agile methods. Initially, they dispensed with the PSI Planning event, and recently introduced a lightweight version of it that they call Unity Day. I can’t help noticing that the size of this operation is close to 150 people, which may be an upper limit to “true” agility.

When you have a self-contained, autonomous team about the same size as a hunter-gatherer band and the team has no external dependencies to deliver its product, the team can self-organize effectively and the agile approach works smoothly and powerfully. It is an error to assume you can obtain the same value by inflating the team to a size that exceeds the natural limits of direct human collaboration. Even the fallback approach of coordinating multiple small teams (e.g., Scrum-of-Scrums in Scrum, or Release Trains in SAFe) incurs process overhead costs that grow geometrically with the size of the operation, and effectively limit the value added by agile methods.