Once “agile” methods had crossed the chasm, organizations in the Late Adopter and Laggard categories began to try and roll out “agile” across their enterprises, or at least across their IT departments.
To support deep-pocketed enterprises that wanted to scale “agile,” consultancies sprouted up offering “frameworks” that could be dropped into place to achieve agility. They are generally based on Scrum and/or Kanban principles, in a way that often lends itself to fun (provided you aren’t the one paying for them).
As the years passed, large enterprises made multiple attempts to implement agile frameworks. With each attempt, they are growing smarter about what will or will not work at scale. But these lessons are excruciatingly expensive.
Despite the general results, consultancies that specialize in scaling agile methods continue to promote the idea. New books have appeared that explain how to scale agile in large enterprises. I checked Amazon books briefly, and got 14 hits for relatively recent books on the subject. But is it even a “subject?”
It occurs to me the basic reason for most of the difficulty is that “agile” as such simply does not scale. If that’s the case, then how can larger enterprises gain the benefits of agility? I think there is a way. It boils down to applying “agile” tactically rather than strategically; except perhaps to say that the strategy is to apply “agile” tactically.
What’s the natural scale of “agile?”
After the publication of the Agile Manifesto in 2001, the word “agile” became a popular term to refer to software development methods based on ideas like iterative development, incremental delivery, lightweight management, direct customer collaboration, and so forth.
I wasn’t involved with it at that time, but in hindsight I see it as a synthesis of ideas and practices that had been growing at the grassroots level of software development for some time. Let’s take a brief look at some “agile” methods to get a sense of how they may apply at the team level and the organizational level.
Extreme Programming
Based on informal conversations with different people over the years, my understanding is that Extreme Programming was a result of a particular project at Chrysler Corporation in the 1990s, to develop a payroll and benefits application. The company had experienced a couple of misfires, and was looking for a new approach. Kent Beck proposed an approach we would recognize as “agile” today.
The responsible executive, CIO Susan Unger, took it to heart. She gave him considerable autonomy to assemble a team and to run the project as he saw fit. The team didn’t start with Extreme Programming; instead, a method by that name emerged from the lessons they learned on that project, and (I suppose) other sources, as well. I apologize if my memory of this is faulty. I wasn’t there personally.
According to Chet Hendrickson (in an informal conversation I might be mis-remembering), the project was a success in that they met the objectives and the system ran in production for a few years. Management decided to replace the system because they believed the labor force could not supply enough Smalltalk programmers to maintain it over time. I don’t know what all the reasons were, but in any case the company reverted to more-conventional payroll software after a few years.
Later, people who loved the idea of “agile” cited this project, known as C3, or the Chrysler Comprehensive Compensation System, as a success that demonstrated the value of the approach, while those who hated the idea of “agile” cited the same project as a failure that demonstrated its folly. The same project. Both groups referred to the project to support feelings about “agile” they already held, before they had ever heard of C3.
Scrum
At roughly the same time, the early 1990s, the process framework known as Scrum was defined. Jeff Sutherland, one of the creators of Scrum, writes about its origins in a 2007 article, Origins of Scrum, on the scrum.inc site. He likes to use the phrase, “hyperproductive state,” which is not my favorite term, but for purposes of understanding the “natural scale” of agile methods the article is informative.
Sutherland states Scrum was intentionally designed as a practical way to handle certain “laws” of software development (from the article):
- Ziv’s law – specifications will never be fully understood
- Humphrey’s law – the user will never know what they want until after the system is in production (maybe not even then)
- Wegner’s lemma – an interactive system can never be fully specified nor can it ever be fully tested. This is the software analogy to Godel’s theorem.
- Langdon’s lemma – software evolves more rapidly as it approaches chaotic regions (taking care not to spill over into chaos)
And he concludes: “Thus any association of predictive or defined processes with Scrum is an exercise in futility.”
That last point is important for understanding some of the challenges very large organizations have faced in applying agile methods, under whatever name.
Crystal Clear
Crystal Clear is a methodology developed by Agile Manifesto author Dr. Alistair Cockburn. It’s described in the book, Crystal Clear: A Human-Powered Methodology for Small Teams, first published in 2005. The method is practical; it’s “distilled” (as the author puts it) from the experiences of actual teams, and informed by Cockburn’s own considerable knowledge of software development methods.
The title of the book mentions small teams, and Cockburn has defined other “clear” methodologies for larger teams, although those have not caught on. The general theme of “small teams” is repeated many times in the text of the book. There is no attempt to force-fit this method at enterprise scale. It’s “human-powered,” and there’s a natural limit to the number of humans who can collaborate closely and function as a “true” team.
Limits of LeSS and SAFe
Of course, I’m not trying to describe these methods comprehensively. The point I want to emphasize for this article is that all these methods are centered on the work of a single team (or in some cases a small group of teams, maybe 4 or 5) that has reasonably good control over all or nearly all aspects of development and delivery, and the opportunity to collaborate directly with real customers or, at least, a key stakeholder who has a vision for the product and the authority to make decisions about the direction of development.
What these methods are explicitly not about is the coordination of tens or dozens or hundreds of teams engaged in a complicated dance of interdependent activities. So I’ll offer a conclusion many agilists will not like: When a business operation has to operate at large scale, it is simply out of scope for agile (unless they take a tactical approach, as described below).
I’m not the first person to see it that way. One of the leading agile frameworks, Large-Scale Scrum (LeSS) defines two models for coordinating multiple teams – LeSS, for groups of around 4 teams, and LeSS Huge for groups of up to 8 teams. As far as I know, there have been no particularly great examples of LeSS Huge to date. Also consider that in large enterprises a group of 8 teams hardly qualifies as “huge.”
I’m not a LeSS insider, but the word “huge” in the context of LeSS strikes me more as a warning than as a description. You won’t really be able to achieve agility if you try to coordinate more than 8 teams as an operating unit.
Consider that 8 small teams amounts to around 70 to 80 people, and in the early days of the Scaled Agile Framework (SAFe) its proponents said it was designed to support an area comprising up to about 150 people. Even at that kind of scale, “agile” is a bit of a stretch. At the scale of a truly large enterprise, it becomes very challenging indeed.
LeSS is a de-scaling framework; a characteristic that distinguishes it from scaling frameworks. If an organization is able to restructure a bit, then Scrum can operate well, with very little “additional” overhead to coordinate multiple teams. But it may not always be feasible for an organization to do so, depending on exactly what they do for a living.
An organization of 20,000 people or more that is structured in a hierarchical way will have many challenges in de-scaling. I don’t know of a successful example.
My own observations, the case studies I’ve read, and conversations I’ve had with people involved with other such initiatives all show that the successes were achieved within limited areas of the corporation, and did not actually “transform” the entire organization. The best results were obtained when the “new way” was implemented in a group that was, in effect, “fenced off” from the rest of the organization, so that it was psychologically and politically “safe” to work in that group. Often, the applications they supported were not mission-critical.
The Rise of Dogmatism
It seems to be a law of nature that once a thing has been created and released into the world, it takes on a life of its own and is no longer under the control of its creator(s). This seems to be true of art, music, literature, architecture, engineering, and anything else. It seems to be true of “agile,” too.
The majority of people who work as agile coaches or change agents today learned about these methods from books and training courses. The original generation is still largely intact and still working, for the most part, but their relative influence over the course of events in the “agile” world has diminished with the emergence of a new generation of practitioners. The new generation first encountered “agile” as a Thing that had a Definition and a Corpus of Literature. They see it as a Set of Rules.
On social media, one sees comments from this generation of agilists to the effect that we must be directly engaged with real customers external to our company, or else we’re “not really agile.” One effect of this dogmatism has been that many individuals working on ostensibly-agile teams in very large organizations feel as if they have “failed” and there is nothing they can do about it. Why? Because software teams in very large organization have no contact with “real customers.”
Furthermore, most software teams in large organizations don’t build complete products on their own. Their work may be one piece of a larger effort that spans many teams in different facilities located in multiple countries. The end product may be something like a vehicle or a network switch or a rocket that can’t be “delivered” in the space of a week or two by any single team of seven-plus-or-minus-two people.
Besides that, who is the “real” customer, and does it make any sense at all for them to be in direct contact with individual members of software development teams at your company?
If you work for a fashion clothing manufacturer, will you (not a fashion designer – just a programmer or software tester) be asked to find out what “real” customers are looking for in the way of fashion? If you work for a company that makes network switches and sells to telecommunications firms, will you (not a lead engineer – just a programmer or software tester) collaborate directly with all the telecommunications firms that might buy the switch whose software your team is writing? Well, if not, then I guess you have no chance of becoming “really agile.”
Teams in very large organizations could benefit from ideas we associated with words like “agile” and “lean,” but when change agents present the ideas in the form of rules to be followed, and deliver unrealistic admonitions about working directly with “real customers,” it tends to have a chilling effect on people. This is one of the reasons large organizations have had limited success with “agile.”
Realities of Large Organizations
Large enterprises have a mission, a strategic plan, and (often) long-term initiatives or programs. They may have to ensure their services are consistent across very large geographical areas while conforming with multiple national laws and regulatory agencies. They may have products that simply cannot be built or supported by any single team or a group of 4 to 8 teams.
Individual software development teams have various roles to play in these matters, but it’s exceedingly rare for any single team to have an overarching view of any given product line or service line, direct access to “real” external customers, autonomy to make significant adjustments to scope, flexibility regarding dates that are driven by product announcements and marketing programs and/or industry regulations, or even freedom to choose “any” development tools they please, due to the need for the enterprise to operate as a cohesive whole, to limit the complexity of production environments, and to control costs.
Many agile coaches encourage teams to think of their work as a series of experiments, and to think of their main goal as to learn from these experiments. There’s value in that, but it has to be applied in the context of what the organization does. In organizations of that size, most of the work items a team takes on are not in the nature of an “experiment.” Any individual team that attempts to operate in that way will create organizational friction, as they are really part of a larger program and they can’t get far out of sync with the other teams that are part of that program.
Is Enterprise-Wide Transformation Realistic?
I get the impression from many agile coaches that they believe literally any organization can be “transformed” to be “fully agile.” I’m not so sure.
Working with smaller companies, I’ve seen profound transformation occur. I’m talking about companies that have a total of between about 50 and 200 people. Some of those people do software-related work. Everyone knows everyone else. The C-level leaders are often present in product strategy meetings, design meetings, and engineering meetings. It’s possible for the group to agree to try an alternative way of working, and put it into effect immediately.
So, why can’t a company the size of JP Morgan Chase, General Motors, or Exxon do the same thing? The reasons are so obvious I can’t think of a way to state them.
In principle, it’s possible for large enterprises to transform themselves. Groups like the Stoos Network and the Beyond Budgeting Institute offer guidance in doing so.
In practice, a couple of factors make this sort of transformation difficult. First, these very large organizations grew up over time. Their structure, culture, and management philosophy are a result of their experiences navigating the competitive market and the world of governmental regulation through many years of operation, reinforced by object lessons taught to them by the market and by regulators.
They way they are currently structured and operated is the result of countless small feedback loops in which their actions were rewarded or punished, and it “works” for them. Beyond using “agile” as a way to try and improve software delivery performance internally, they aren’t interested in deep changes.
There’s another point that I’m pretty comfortable with, but I think a lot of my colleagues in the coaching space don’t like to think about: There is no agile consultant or coach or trainer anywhere in the world who has ever met payroll month after month for a staff of 100,000 or 200,000 employees. Executives quite reasonably question who we are to tell them their organizations are “dysfunctional” and need to become “agile.”
Government Interference in Natural Economics
A second factor inhibiting meaningful transformation is the relationship between very large corporations and national governments. For example, in the United States, despite an espoused belief in the value of capitalism and free market economics, the government will always step in to rescue a large corporation that is having difficulty due to mismanagement or even due to unethical business activities. Always.
Thanks to this “too big to fail” mentality, smaller, more innovative companies are unable to step in to fill in the market demand left unfulfilled by a failed behemoth. The behemoth is simply propped up using taxpayers’ money. That removes any connection between business practices and financial outcomes for corporate leaders, and the resulting lack of accountability creates an incentive for them to repeat the behaviors of the past.
The basic mechanisms of free market economics are not allowed to operate with respect to the behemoths. Executives feel they are completely safe, provided someone in the C suite plays golf with a Senator occasionally. They have no incentive to modernize their organizational structure, management practices, or assumptions about people. The status quo serves them perfectly, and there are no penalties for failure to change.
Impact of Labor Unions
In some countries, labor unions play a large role in economic life. When we attempt to organize people into collaborative working groups or teams, and to think of the team instead of the individual as the “unit of labor,” unions often feel the change is not in the interest of their members, or in the interest of the union itself as a center of political and economic power. After all, if workers normally collaborate and trust each other, and management has been repurposed as facilitators and impediment-removers rather than “bosses,” what role is there for a powerful labor union?
Federated Organizational Structure
One approach that some companies have taken is to restructure from a hierarchical organization to a federated one. This is not directly related to “agile” methods, but it can be an enabler of agility when each part of the federated organization is dedicated to a distinct product line or service line. It’s also compatible with the Beyond Budgeting concept of pushing decision-making authority down and out in the organization.
Federated organizations seem to have been discussed more in the context of social structures than businesses, as mentioned in this summary. In the context of business, Graham Douglas offered some practical advice in his 2014 article, Adopt a federated structure rather than the existing monolithic structure.
But there is considerable misunderstanding about federated organizations. Consider the 2016 article by Rick Goldstein, Are Federated Association Structures Becoming Obsolete?. He is thinking about associations or organizations in general, and not necessarily just for-profit businesses. He cites three “major impacts” of federated organizational structure:
- High senior management costs.
- Separate back-office operations.
- Internal competition.
He writes, “Federated organizations typically maintain separate executive teams.” Well, maybe they do, but who says every organization needs an “executive team?” That strikes me as a carry-over from hierachical organizational thinking. Therefore, I consider it a misapplication of the idea of federation, rather than a natural effect.
If you were to apply “agile” thinking, and possibly a structure such as LeSS, you wouldn’t need an “executive team.” You’d probably want some kind of overarching guidance or steering of the whole federation, but that can be lighter-weight than a 20th-century “executive team.”
Why would an organization carry the cost of running its own back-office functions? It seems to me that takes resources away from mission-critical and market-differentiating activities. There are companies that specialize in back-office services, such as payroll and benefits administration and general accounting. Back-office functions do not provide any competitive advantage, and should not be a resource sink. This management error is not connected with organizational structure, whether hierarchical or federated.
Why would there be internal competition unless more than one of the federated business units were concerned with the same product or service line? It seems to me this would result from a mistake in defining which business units should be established, how they are aligned with value streams or product lines, and how they will work together. It is not a natural result of federation itself.
I’ve observed a general problem with the adoption of “new” ideas in business. The leaders tend to overlay the “new thing” on top of the “old thing” without adjusting their thinking to align with the “new thing.” Then they blame the inevitable friction categorically on the “new thing,” without questioning their own assumptions. This applies to federated organizational structure, lean, agile, and literally any other “new thing.”
Issues with Parallel Organizations
A parallel organization is usually described as a way to effect organizational change in a step-by-step fashion. A “techie” might see it as similar to the “strangler” pattern for shifting a workload from an old system to a new system. Usually, the idea is the parallel organization will be part of the larger organization, and will pave the way for a new way of working that ultimately is folded back into the larger organization.
The typical result is that once the parallel organization is folded back into the parent organization, it is overwhelmed by the constraints, culture, and standard procedures of the parent organization, and everything reverts to the status quo ante.
In the book, Large-Scale Scrum: More with LeSS, the authors present a few guides to help people apply LeSS effectively. One of these uses the idea of a parallel organization. Rather than trying to “transform” a large and complicated organization from within, we stand up a new organization based on the principles we want to see in use. Then we gradually shift work from the old to the new organization, until the work is entirely handled by the new organization in the new way.
I’ve seen a strategy like that used a couple of times when large companies wanted to create work units that were unburdened by the traditional bureaucracy and management assumptions of the parent organization. The strategy side-steps the difficulties of “transforming” a long-established organization in place.
In the successful cases I’ve seen, it did not result in the parent organization’s transformation. The company created a subsidiary located in a different place than the “old” department, hired completely new staff who were already experienced with or at least predisposed toward the new way of working, and shut down the old department once the workload had been shifted to the subsidiary. It was not a technique to transform the parent organization.
Note: The term parallel organization is overloaded in our field. It is sometimes used as I’m using it in this section, to mean standing up an organization whose general business function parallels an existing department, which will ultimately replace that department. The term is also used as a synonym for “flattening” the organizational structure. That is a different sense than I’m using it here.
Issues with Pilot Programs
In many ways, the parallel organization approach resembles the “pilot” approach for adopting agile methods in an organization. A pilot initiative for agile methods is created, a management “bubble” is created around it, a “tiger” team is selected to deliver the pilot project or projects, and the group is largely exempt from enterprise standards and guidelines regarding development tools and other details.
They cherry-pick projects that almost cannot fail, and celebrate their success. Then management wants to overlay this new thing on top of the old.
The typical result is that when the “new thing” is “rolled out” to the larger organization, all the teams are subject to the existing organizational constraints, rules, and standard procedures, and they are no longer free to operate in the “agile” way. Often, most of the people who participated in the pilot leave the company, and within a few months the organization is back where it started – except that it has fewer good people on staff.
Effect of Stack Ranking
Many larger organizations use a performance management system for employees known as stack ranking. Each employee has to be ranked against the other employees within a department or within a team. Managers are required to assign specific proportions of rankings; for instance, 10% “exceed expectations,” 80% “meet expectations,” and 10% “do not meet expectations.” When the pilot teams consist of the very best people in the organization, standard human resources procedures require the company to fire 10% of them.
The direct impact is to reduce the effectiveness of the organization. A temporary indirect impact is the loss of people who quit rather than await the next stack ranking assessment. Lasting indirect impacts are to destroy morale and to encourage people to hold back and not perform at their best, to try and make everyone’s assessment come out the same. When the organization is interested in adopting agile methods or any other contemporary working practices based on collaboration and trust, the stack ranking program promotes individual competition instead.
I’ve seen multiple cases in which the pilot program was going very well, and people who were involved in critical activities suddenly disappeared. They had been abruptly laid off, with no handover of their work, as a result of a stack ranking performance assessment program. No one in the organization picked up that work, as in the immediate aftermath of the annual round of layoffs there were not enough people left to handle “business as usual” work. The new initiative fizzled out.
They had been selected for the pilot program because they were among the top people in the company. Not only did the agile initiative grind to a halt, the organization now had fewer good people than it had previously.
A more insidious effect of stack ranking is the deterioration of the company’s reputation in the labor market. Not only do they release some of their best people, they also make it harder to convince others to join the company. A damaged reputation often lasts longer than the problematic situation as such, and may continue to hamper the company even after they have corrected the problem internally.
Stack ranking is colloquially known as “rank and yank,” for obvious reasons. It is one of the most damaging of 20th-century management practices.
Agile consultants are aware of the impact of stack ranking on fundamental matters such as collaboration, transparency, trust, and agile-compatible performance assessment and career paths. However, very few agile consulting engagements at very large companies involve the whole enterprise; they are almost always carried out within the IT department, with no higher level of engagement than the CIO. Human Resources policies are simply out of scope for the consulting engagement. I and others have been advised quietly by client managers to stop mentioning employee performance assessment during consulting engagements, if we wanted to make any progress with the agile initiative.
Management Turnover
I’d like to connect a few dots here. First, there’s the idea that an “agile transformation” can be broad and shallow or narrow and deep. In this context, a broad and shallow transformation means that agile decorations are overlaid atop existing organizational structure and management processes, and people in the organization play whatever games are necessary to make their numbers look good. Typically, this is done by “implementing” a framework.
A narrow and deep transformation implies people in the organization make the profound and meaningful changes necessary to achieve business agility in a real sense. Agile consultants will tell you this kind of transformation takes years, progresses gradually, and requires constant attention to principles and day-to-day support for the new way of working.
The second dot is a reference to systems thinking. Not to make a dissertation of out it, but the key idea for this section is the observation that people can make choices within the constraints of the system in which they operate. “System” here can mean a national culture, a corporate culture, or something like that. It means people can’t do literally anything they please even if they fully understand the implications of continuing to work in the “old way” while the transformation is in progress.
The third dot to connect is the nature of management or leadership turnover in very large enterprises. Many agilists will argue that a small company is also an “enterprise,” and they are quite right. However, the dynamics – that is, the systemic constraints on available choices – are different in very large enterprises than they are in smaller ones.
Agile transformation programs are almost always initiated from inside the IT department. IT departments are cost centers. Managers in cost centers are judged by the headcount and budget they control. The career path involves increasing the number of people under you on the organization chart, and managing ever-larger annual budgets.
Please note the career path is explicitly not to demonstrate that you can “do more with less.” The system in which you must operate in order to follow this career path does not allow you to make that choice.
Senior management in large corporate IT departments tends to turn over fairly rapidly – in many cases in the range of 10 to 18 months. The outgoing leader takes with them their favorite people. The incoming leader brings with them some of their favorite people. To demonstrate to the executive suite that it was a good idea to hire them, the new leadership dismantles whatever their predecessors had started.
The situation is a little different when the agile transformation is initiated at the C level. In that case, leaders are responsible for revenue, profit, market share, customer satisfaction, and corporate reputation. New leaders will do something splashy to show the board of directors that it was a good idea to hire them. That often includes mass layoffs and restructuring.
In both cases, if an agile transformation was already underway it will be terminated, and if no agile transformation was already underway a broad and shallow initiative will be started. People on a C-level career path are operating within the constraints of an overarching “system” that limits the choices they can make when they are hired into a C-level or IT leadership role. They cannot spend years and years to carry out a narrow and deep – that is, meaningful – agile transformation.
Tactical Application of Agile Methods
So, if we take as given that very large enterprises cannot really “transform” in a meaningful way, and that agile methods by their nature only work at relatively small scale, then how can large organizations take good advantage of agile principles?
Even in the context of a traditional organization, software development and support teams can improve their performance and the quality of their working lives by applying agile and lean ideas to their work. It isn’t necessary to “deploy” or “roll out” something like SAFe enterprise-wide to obtain these benefits. It’s quite practical to apply agile methods within an area that has responsibility for a given product or product line, on the general scale of a LeSS Requirement Area.
We can apply LeSS or team-level Scrum or Kanban or a combination (often called “scrumban”) tactically, where it makes sense, and at a scale where there’s a chance it can add value, without worrying about trying to “transform” the entire enterprise or even the entire IT department.
Besides, forcing everyone to work in the same structure and follow the same procedures doesn’t seem like an “agile” way of doing things. A tactical approach might be both more effective and more consistent with agile values than an enterprise-scale rollout of a framework.
It’s also more practical than trying to transform the whole enterprise, given that the largest enterprises are fundamentally untransformable and agile itself doesn’t scale.
Aside: LeSS and Tactical Agile
It bears mentioning that LeSS specialists I know complain that the idea of “tactical” application of LeSS is not consistent with the overall goals or philosophy of LeSS. It’s a framework for general organizational improvement, and not meant to support a fenced-off working group in isolation from the rest of the company.
I appreciate that and I’m aligned with the philosophy, but in the context of the realities of very large organizations, it simply isn’t going to happen that way. It’s better to cut our losses and adapt our approach early rather than late…even if we really, really like a particular framework or method.
If it is unacceptable to use the name LeSS in this way, then we don’t have to use that name. We do have to face up to reality, however. Organizations up to a certain size will be able to embrace LeSS as its creators envision it. Larger organizations will probably not be able to do so. They tend to employ LeSS as a way of coordinating a group of teams with a “software factory” mentality rather than as an organizational de-scaling approach. They have no intention of changing the organization as such.
Second-Generation Agile Coaches and Muda
I’ve mentioned already that the second and later generations of agilists learned about agile after it was a Thing. They learned buzzwords and practices. The original generation didn’t think of it that way. They were exploring the values and principles that they had observed or experienced as helpful in building and delivering useful software in a practical way. The whole “agile thing” came from the trenches of development, and was not an ivory tower academic exercise. Academic studies of how and why agile techniques seemed to “work” came after the fact.
Today, most agile coaches seem to be interested in adding more and more and more to agile. More analogies, more models, more games, more rituals, more…well, more overhead, when you come right down to it. Agile has become very complicated, and I think a good deal of the complication is artificial and unnecessary.
Some of these coaches focus a great deal of energy on “perfecting” specific practices or rituals defined in one or another branded process model, especially Scrum. It might be useful to remember that all the Scrum events (and similar activities associated with other branded processes) are overhead.
In Lean terms, any work that doesn’t advance value for the customer is muda – non-value-add activity. That doesn’t mean these things aren’t useful. Some of them are necessary. Others are helpful to novice teams, and one would hope such teams will eventually improve to the point they can dispense with some muda while still delivering effectively.
To maximize value-add time, we want to minimize the time and effort spent in doing overhead activities while still achieving the benefits of doing them, and absolutely not to maximize the time and effort dedicated to them, and inflate them into Important Things in their own right.
Speculation: I suspect that some of the agile coaches who focus exclusively or disproportionately on perfecting overhead activities don’t have personal experience with the hands-on work. They expand the relative importance of the overhead activities, because those are the activities they can understand. They feel as if they are helping when they can teach and mentor these activities. They may feel (I’m not sure what they feel, of course) that they are a little lost in the technical details and that they aren’t adding value, so they tend to underplay technical activities. But in the software field, those are the activities that add value for customers. It’s true there are many facets to customer satisfaction, but unless we get around to writing some code sooner or later, there will be no product at all, good or bad.
People like to say the real product is the customer experience, and I think it’s important to keep that in mind; but on a practical level, the software is the “thing” that provides the customer experience. The software is the thing-that-is-made by the teams we coach. Everything else is supportive, but not central. When we lose sight of that, we may tend to spend proportionally too much time on secondary matters and lose focus on that which brings value.
Rediscovering the Fundamentals
So, what if we returned to the fundamentals? What if we stopped nit-picking buzzwords and trying to follow the “rules” at all costs? What if we focused on the original scope of the Agile Manifesto – software development – and stopped trying to force-fit “agile” into anything and everything?
Returning to fundamentals means, in part, setting aside quasi-religious dogma. When we catch ourselves treating some “agile thing” as a rule inscribed on a stone tablet and handed down from on high, let’s try and remember – or find out – how and why the thing originated. What problem did it solve at the time? Do we still have that problem?
Dogma: Stable, Dedicated Teams
There’s a great emphasis in agile on stable, dedicated teams. What does that mean? Why was it deemed important? Is it still important?
I would say stable team and dedicated team are two different ideas, even if most coaches say all those words together as a set.
A stable team is one that remains together over a long period of time, rather than disbanding at the end of their project as used to be the case in the 20th century. It was important in the early days of the agile movement because people were unaccustomed to working together collaboratively. Bruce Tuckman’s Forming, Storming, Norming, Performing model became popular, as people who had always worked alone were put together with others for the first time in their careers.
Is that model still relevant today? My observation is that it is obsolete. Today, many software professionals are quite comfortable working in a collaborative way. They seamlessly join “mob programming” groups or pair up with a colleague they don’t know very well. Introverts learn how to communicate effectively and extroverts learn to dial back their enthusiasm so they won’t put people off. It’s simply expected of people who choose this field of work.
I’ve seen “advanced” or “mature” agile development groups with 20 to 80 people who could self-organize around specific pieces of work seamlessly and without stress. Different individuals would team up to complete different work items. There was no “storming” or any of that.
(By the way, notice how that group size compares with an Agile Release Train in SAFe or a Requirement Area in LeSS. Coincidence?)
Today’s world is not the same as the world in which the stable team concept was revolutionary and game-changing. That is for the best of all possible reasons: It worked. The agile movement changed the world (in a small way – don’t get carried away).
What about the dedicated team concept? It means that any given team is focused on just one initiative at a time. The problem this was meant to solve was so-called “matrix management,” in which individuals were assigned on a percentage basis to multiple initiatives or projects concurrently. That approach was a disaster from Day One and never worked.
One problem with matrix management was predictability in planning. It was impossible to be predictable. Individuals recorded their time against whatever projects they were authorized to update in the time tracking tool. There was no connection between the metrics used for planning and the reality on the ground.
The second problem with matrix management was multitasking, or context-switching overhead. This incurs a much higher cost than 20th-century managers usually recognized.
Predictable planning is still an important business need in most companies, and context-switching overhead is just a fact of nature. So, the dedicated team concept remains relevant today. It is not quasi-religious agile dogma, it’s a practical reality.
Dogma: User Stories
Why do we use User Stories? Is that a requirement of agile work?
Many organizations and teams are hung up on the canonical User Story format: “As a something I want something so that something.” But this is only a suggested starting point for people who have never worked in this way before. It is not a “rule.”
Treating User Stories and the single rigid format as a “rule” leads to workplace stress. Many teams spin their wheels trying to force-fit a piece of work into that format. They believe they are required to do so. Who is telling them they are required to do so?
To alleviate the problem, agile coaches have added more and more and more to User Stories. There are Job Stories, Technical Stories, System Stories, and so forth.
Why so much complexity? Why so much formality? Why so much rigidity? Are complexity, formality, and rigidity part of agile thinking? What happened to solving problems by removing things rather than adding them?
In most cases, I’ve found it useful to consider
- what is the work we need to do?
- who wants it and (briefly) why?
- is it reasonable to get this work done in a short time or do we need to break it up a bit?
- how will we be able to tell when we’re done with the work?
In some cases, additional information may be needed, such as a list of system codes that are relevant to the software we’ll be writing or changing. The information can be codified in any way that makes sense to the team. The level of detail can be whatever is appropriate. There’s no need to cleave to a rigid format and bite our fingernails when a piece of work doesn’t naturally lend itself to that format.
Agilists have added more and more and more even to the simple idea of breaking up a large story into smaller pieces. A quick Internet search finds something north of thirty (30 – yes, that’s a three with a zero after it) different techniques for slicing User Stories. Short-term planning of small work items becomes an Important Thing that takes a long time and considerable analysis. I may be wrong, but I don’t think that was the original idea.
People say a User Story (or at least, a card) is a placeholder for a conversation. That’s great when it makes sense. But in larger enterprises that have thousands of teams, there may be no one to have a conversation with. The team’s work is just a small part of a much larger process. Their “product backlog” isn’t a list of potential features to be prioritized by a product owner; it’s a to-do list, plain and simple.
And that’s okay. If that’s the reality where you work, you haven’t “failed” agile. You can still apply agile values and principles to your work.
Dogma: Relative Sizing
Why do we “size” stories?
Remember the point about predictability in planning. In the Olden Days, people were asked to estimate how long each task would take. This was problematic. For one thing, no one could be sure how much of their time would be dedicated to the task. People were matrixed across multiple initiatives, and they were frequently interrupted to attend meetings or training or to answer questions. They worked alone, so they got stuck on small problems for an hour or two at a time; sometimes even for days at a time.
So, they padded their estimates. That didn’t help. Estimates had wide ranges or low confidence, and often didn’t correspond with reality.
The next thing people tried was to use “ideal time.” They would add a fudge factor (what ThoughtWorks used to call a “load factor”) to their available work hours, and leave that aside as a buffer to absorb unexpected interruptions and context-switching overhead. They might say, for instance, that in a 40 hour week they actually had 20 hours of dedicated work time. Then they estimated how long they thought the task would take to complete, absent any interruptions.
That was better than padding clock-time estimates, but still didn’t provide good predictability for planning.
Someone – I’m not sure who – came up with the idea of comparing each work item with other work items the team had completed in the past. A new work item that seemed similar to a previous one was considered to be about the same “size.” Teams sized work items relative to each other instead of trying to guess or calculate the time that would be needed to complete tasks.
Teams used whatever units of measure they pleased, and whatever scale that made sense to them. Agile coaches turned all this into a detailed and complicated set of rules (of course – more and more instead of less and less). The most common unit of measure was the “story point,” and agile coaches coined a new verb: to point <sigh>. It became dogma that a team had to “point” all their stories before they could begin work <sigh>.
“What is your team working on today?”
“We’re pointing today!”
“Cool! You’re really agile now!”
<sigh>
This tended to produce estimates with less variability and higher confidence than previous methods, but the question remained: Why are the work items of vastly different sizes?
If teams could learn to define work items of roughly the same size, they wouldn’t need to spend any time trying to “size” or estimate their work. Is that practical?
Software development is not assembly-line work; it’s creative work. That means two tasks may well require very different amounts of time. Even so, with practice a team can reduce the variability in the size of their work items.
I recall a company (I think it was Menlo Innovations in Michigan – perhaps a kind reader will correct me) that used a sheet of paper per day for planning software development work. The sheet was marked in quarter-day sections. Teams would fold their “story” description in half or in quarters (or not fold it), and place it on the sheet. Work items filled a quarter day, half day, or full day. When five sheets were filled, planning was done for the week. That was it.
The reason this worked well was that management ensured teams knew how much dedicated work time they would have. Meetings were kept to a minimum and were announced well in advance. Teams weren’t constantly surprised with interruptions.
Technical practices can help with this, too. Consider continuous integration, continuous deployment, and feature toggles, taken together as a related set of practices. Teams can deliver small increments of a solution all the way to production safely and smoothly, on a short time frame. Then there is little need for estimation or sizing.
Which will provide better value to stakeholders by reducing overhead in the delivery process: Perfecting our sizing technique, or improving our story-crafting skills to eliminate the need to size the stories?
When we do have to size stories, must we use planning poker? Who says that is The Way to size work? James Grenning, who came up with planning poker as a way to break an impasse between two engineers locked in a circular debate during a planning session, was surprised when the technique became part of agile dogma.
Sizing stories isn’t a fundamental agile Thing, and still less sizing them in any particular way. It’s a workaround for teams that have extreme variability in the sizes of their work items and/or can’t deliver smoothly for some reason – lots of long-lived branches, low collaboration, insufficient automation, etc. Yet, many agile coaches approach story sizing as a Thing for teams to learn and master. It should be a “thing” they learn to do without. But they have to earn that, by learning to craft work items of roughly similar size, and possibly learning certain supporting technical practices, as well.
Dogma: Daily Stand-up
Despite the amount of material available in books, websites, videos, and at conferences and user group meetings, agile software development still hasn’t penetrated the industry very deeply. Most people know little about it, except that agile teams hold a meeting every single day in which they are required to stand up for fifteen minutes.
Many agile coaches, as well as Scrum Masters, make very sure their teams fill the entire fifteen minutes every day. They maximize the muda and call it “agile.”
What’s the deal about fifteen minutes, anyway?
In the previous millennium, meetings typically took up a minimum of one hour, included many people whose presence was not really necessary, and no individual participant needed to attend for more than five minutes. They spent the rest of the time waiting for their five minutes. Weekly status meetings for software teams dragged on for hours, and yielded no useful outcome.
So, early agile methods defined a way for team members to stay abreast of what was happening with their project, to find out if a piece of work was going to take longer than expected, or if any team mates needed help with something. Fifteen minutes was a very short time for a meeting in those days.
Today, teams that are operating in an agile fashion and working collaboratively don’t need fifteen minutes to touch base every day. Most teams that are well-organized and functional can accomplish the objectives of the daily stand-up in five minutes or less.
What’s the deal about standing up, anyway?
Many people say the team members should stand up so they won’t get too comfortable and make the meeting drag out too long. The story I heard was a little different.
I heard that in early Extreme Programming teams, people sat around a central table in the collaborative work area. When the time came for the daily stand-up, if people remained seated then they also remained focused on their monitors and keyboards. To ensure they would look each other in the eye and listen to each other, they stood up to break eye contact with their monitors and to keep their keyboards out of reach.
What’s the deal about doing this every day, anyway?
Remember that agile software development is about…well, software development. Specifically, application development. The assumption is the team is intensively adding features to a software product. In that type of work, things change rapidly. The product might change in a significant way within hours.
Let’s say a team uses pair programming. One pair is working on an API and a second pair is working on client code that will call the API. If the two diverge, they will create a large problem to be solved later. If they talk every day, they can make small course corrections that are painless, and never end up with a large problem.
If the daily stand-up is muda, how can we do away with it?
When highly collaborative methods are used, such as “mob programming” or “ensemble” work, there is no need for the team to hold a special event every day to share what they’re working on, because the whole team works together most (or all) of the time; everyone already knows what’s going on.
Increasing collaboration is a good way to advance a team beyond the need for the daily stand-up event.
What about other kinds of work?
The nature of the work determines how rapidly things might change, which in turn suggests an appropriate cadence for team members to discuss what they’re doing. One team I coached was visiting 160 sites around the world and making an inventory of network equipment installed at each site. When that was complete, they were going to install or upgrade the equipment at each site.
Their first agile coach had insisted they have a stand-up every day. They were frustrated and bored with that, and felt agile was worthless. After they explained the nature of their work to me, I asked them if they saw any value in staying up to date with what the other people were doing, and learning about any issues early rather than late. They said yes. So I asked them what they thought an appropriate cadence for doing this might be. I was thinking, maybe, twice a week, but I didn’t say that. They suggested three times a week. From then on, they had their “daily” stand-up three times a week, and they were pretty satisfied with that.
The problem with dogmatic agile coaches is they don’t think about the purpose or value of Agile Things. If it’s an Agile Thing, then you do it by rote according to “the book” and you never vary.
A case in point: At a large client, the agile coaches function as a team. Some of them insist we have a daily stand-up, because that’s what we teach the software development teams to do. But coaching work doesn’t result in significant change on a daily basis. It takes longer. There’s nothing useful to say every single day. The dogmatic coaches just can’t seem to see that. All they know is the daily stand-up is an Agile Thing. Period.
What Can Tactical Agile Look Like?
If we combine a few of these ideas, we can visualize tactical agile in a large, traditional enterprise. Of course, my concept of this is conditioned by my own experiences, and your experiences are different. But at least you can get a general sense of it.
At a higher level of planning – often called the “portfolio” level in large enterprises – initiatives are scoped, budgeted, and prioritized. When work is handed off to one of these agile work groups, the work group manages itself to deliver the work. So, we can have traditional management operating in the large and agile operating at the tactical level without friction.
The scope of a tactical working group is roughly the same as a SAFe Agile Release Train (ART) or a LeSS Requirement Area. However, I don’t mean to suggest that they are an ART or a Requirement Area. I’m only thinking generally about how many people can function effectively in this way.
The work group loosely corresponds with concepts like product line or value stream. Don’t worry if the terms don’t align perfectly or precisely. That isn’t important. The point is that this structure enables traditional management and agile delivery to work in concert, with each contributing what it best supports.
The overhead necessary to coordinate the teams within this group is smaller than that needed with SAFe or LeSS, because the individual teams are not “stable” or “persistent” or “long-lived.” Heidi Helfand’s idea of Dynamic Reteaming is simply the normal mode of operation, and not a Thing that needs a Brand.
There’s no need to explain carefully why and how individuals might move from one team to another, as happens in LeSS and also with the Spotify model, because it isn’t unusual. Individuals may team up with different people for each work item they contribute to. It’s normal, and it isn’t disruptive. It’s a non-event.
The result is that it’s normal for people to self-organize around the work. The shape of the work at hand drives the formation of temporary teams from a pool of people who are accustomed to working collaboratively, and who do not suffer from the Tuckman model of team formation.
Because different subsets of people work together on different work items, and work items are of different sizes, the various temporary teams may complete their work at different times. To level that out, we have a steady release cadence. That way, everyone (or most people, anyway) becomes available to work on something new at around the same time, and they have the opportunity to re-form new temporary teams.
Does that cause “resources” to be “idle?” Well, it would, if humans were resources. As humans are humans, they will find useful things to do until the next release point. If I were their manager, they could do anything they pleased. Sometimes a person might take a few days off. Sometimes a person might join another temporary team to help them finish up whatever they were doing. Sometimes a person might spend the time cleaning up old files or doing some refactoring. They’re grown-ups.
The work goes up on a wall. Each item has a short description and a set of cards that represent skill sets relevant to the work item. People review the descriptions and grab the card that represents what they would like to work on.
People are encouraged to choose different kinds of work over time, so they can enjoy variety and extend their skills.
People take turns getting first pick of the available work, so that “senior” people will not dominate the process and claim all the most interesting work items for themselves.
There is no attempt to “normalize” or “standardize” agile practices across the enterprise. Each agile working group is free to self-organize and operate as it chooses. However, there may still be value in standardizing some of the software tools used in the enterprise, for reasons of cost, support, and consistency.
Many organizations that engage in large-scale transformation programs get tangled up in metrics. They try to roll up metrics like “velocity” for executive dashboards. In my view, they don’t need to worry about that. They can track the standard Lean metrics without drilling into the individual tactical agile work groups. They can track cycle time, throughput, and other key metrics pertaining to overall delivery effectiveness without trying to “control” what happens at the tactical level.
Each tactical agile work group is free to determine how they want to operate. This model doesn’t prefer, recommend, or depend on any branded frameworks or methods.
This model takes bits and pieces from companies that have tried various approaches. I don’t know of any organization that has implemented a model like this comprehensively. It might be worth a try.
At the very least, it’s an attempt to break free of entrenched, dogmatic assumptions about what “agile” has to be.