Contemporary software development methods emphasize collaborative work over solo work. I remember this idea gaining momentum from as far back as the 1990s, when Extreme Programming was becoming known, including one of its core practices, Pair Programming.
In retrospect, I can recall many instances when people collaborated in an ad hoc way to solve specific problems, going back to the beginning of my IT career in 1977. I’ve read articles and heard stories about the value of collaborative work in engineering going back at least to the 1950s. There was even a deliberate experiment in 1975, carried out by the US Army, on the value of “two-person teams” in software development (the article does not appear to be online anymore, but there is an excerpt in this old blog post: Does pair programming work?).
In all these cases, from formal studies to informal “war stories” told over coffee or beer, direct collaboration provided immediate and palpable benefits to the people involved. That has been my experience, as well. And yet, it seems as if solo work continues to be the norm.
What is “collaboration?”
The Century Dictionary defines collaboration as “The act of working together; united labor, especially in literary or scientific work.”
According to Wikipedia, “Collaboration is the process of two or more people or organizations working together to complete a task or achieve a goal.”
Dictionary.com says to collaborate is “to work, one with another; cooperate, as on a literary work.”
The Cambridge English Dictionary defines collaboration as “the situation of two or more people working together to create or achieve the same thing.”
The Collins English Dictionary defines it as “the act of working together to produce a piece of work, especially a book or some research.”
According to The Free Dictionary, to collaborate is “to work together, especially in a joint intellectual effort.”
Study.com has an online class on Effective Communication in the Workplace. One of the lessons is “What is Collaboration in the Workplace?” They present a short video that explains the concept in the context of business organizations. They suggest the following key elements of workplace collaboration:
- Thinking and brainstorming ideas to provide solutions
- A strong sense of purpose
- Equal participation
Market penetration?
No doubt all these definitions and examples make sense to you. You’ve probably experienced collaboration in various forms throughout your career, and in your personal life. And when it happened, I’ll bet you found it helpful more often than not.
Why, then, have collaborative methods of work not penetrated the software field more profoundly? I think there are three main reasons.
First, to be effective, collaboration must be intentional and supported deliberately and mindfully in the organization. Ad hoc collaboration in the heat of the moment can save the day – for instance, when a team is working to solve a critical production support issue – but in most situations we can’t rely on ad hoc collaboration as a standard way of working.
I’d like to explore this using Jean Tabaka’s 2006 book, Collaboration Explained: Facilitation Skills for Software Project Leaders as a guide. In addition to her suggestions, we also need to pay attention to the organization’s “human resources” policies. When these are focused on individual performance, it tends to discourage collaboration, to the detriment of the business.
Second, people have different intuitive notions of exactly what “work together” really means in practice. I would say there’s a range or spectrum of collaborative methods. Technically, we can say we’re “collaborating” if we are working in a way that falls anywhere on that spectrum. However, to gain significant benefit from collaboration, we want to try and work closer to the “fully collaborative” end of the spectrum.
Relatively few people in the software field have ever experienced this, and when they think about collaboration it just doesn’t come to mind. I’d like to explore this idea later in the article.
Third, technical schools and universities don’t usually teach software development as a collaborative activity. In most cases, students perform their lab assignments independently. Of course, there are good reasons for this. Instructors need to be able to distinguish the work of individual students in order to see who is progressing well and who needs help. If the work were done as group exercises, it would be hard to see which students need guidance. Unfortunately, there’s also the question of cheating or plagiarism.
But I know from experience that it’s possible to teach a hands-on software-related course that is heavy on lab work and allow students to collaborate freely, and still be able to detect plagiarism and to see which students need additional help. It just doesn’t seem to be the norm in schools today.
Mindful, deliberate collaboration
I’ve worked with organizations whose management declares collaboration to be a high priority for “transforming” the way of working, and yet they do little or nothing to enable collaborative work. In her 2006 book on the subject, Jean Tabaka makes several concrete recommendations to help organizations get value from collaboration. Her book offers specific guidance in each of the following areas. I’m paraphrasing most of these.
- Build collaborative cultures, leaders, and teams
- Prepare yourself and your team to collaborate
- Define clear roles for each participant in promoting collaboration
- Set a collaborative agenda
- Use and master collaboration tools
- Run effective collaborative meetings
- Promote small-group and pair programming collaboration
- Get better information and use it to make better decisions
- Use non-abusive conflict to drive positive outcomes
- Collaborate in planning and estimating work
- Strengthen remote collaboration
- Extend collaboration from individual teams to the whole organization
The state of the art has continued to evolve since 2006, but Tabaka’s book remains relevant today. We now organize teams around products rather than projects. We’ve developed even more-collaborative methods than pair programming. But nearly every recommendation Tabaka makes in the book still applies to organizations and teams in 2020.
If the section above containing definitions describes the “what,” then this book describes the “how.”
It isn’t my intention to reiterate everything in the book here, but I do want to suggest it as a reference and guide. Most organizations I work with, and especially the larger ones, are at a stage in their adoption of collaborative methods where this book would be an ideal fit.
Aren’t we already collaborating?
Short answer: Yes.
How can I answer, simply, “Yes,” without asking any clarifying questions? Because people don’t work entirely in isolation. No matter the organizational structure, definition of roles and job descriptions, or formal development process in use, people communicate with each other somehow. And that’s enough to qualify as a form of “collaboration.”
Let me assert that many, and possibly most activities pertaining to software development benefit from a collaborative approach. What do I mean by “benefit from?” I’m thinking mainly of Lean concepts such as Lead Time and Throughput, as well as “software craftsmanship” concepts such as “clean design,” “functional correctness,” and “code habitability.” I place those terms in quotation marks because people may use different words for the same general ideas.
With that in mind, different degrees of collaboration may result in different degrees of value. Here’s a quick-and-dirty model:
It’s fair to say teams operating in any of the ways shown in the illustration are collaborating to some extent. In my view, those who are collaborating more closely are likely to achieve better outcomes than those who are collaborating less closely.
To illustrate, let me draw an example from a blog post of mine from 2015, Team practices, code reviews, and lead times. The idea in that post is to compare different ways teams may conduct code reviews, and to explore how these alternatives affect Lead Time. My assumptions are:
- Code reviews are useful to software development teams
- Code reviews are not something customers pay for explicitly, and therefore qualify as “waste” according to Lean Thinking
- When we need to do something that’s useful to us, but that is “waste” from the customer’s point of view, we want to do the thing in a way that incurs a minimum of overhead (time and cost) so that we can invest proportionally more of our time in work that adds value from the customer’s point of view
Are these assumptions valid? Let’s consider the value of code reviews.
In a May, 2013 article in Proceedings of the 35th IEEE/ACM International Conference On Software Engineering (ICSE 2013), “Expectations, outcomes, and challenges of modern code review”, Alberto Bacchelli and Christian Bird explain “Our study reveals that while finding defects remains the main motivation for review, reviews are less about defects than expected and instead provide additional benefits such as knowledge transfer, increased team awareness, and creation of alternative solutions to problems.”
In IEEE Transactions on Software Engineering, Vol. 35, No. X, 2009, Mika Mantyla and Casper Lassenius wrote in “What types of defects are really discovered in code reviews?” that “75 percent of defects found during the review do not affect the visible functionality of the software. Instead, these defects improved software evolvability by making it easier to understand and modify.” They also write, “…code reviews find many evolvability defects and, thus, offer additional benefits over execution-based quality assurance methods that cannot detect evolvability defects. We suggest that code reviews may be most valuable for software products with long life cycles as the value of discovering evolvability defects in them is greater than for short life cycle systems.”
You can find many more references that report similar findings. What this boils down to on a practical level is that code reviews help teams control the growth of technical debt and maintain the code base in a habitable state. These factors contribute to controlling the cost of ownership of the software product over its lifetime as well as enabling more-rapid turnaround of software changes with reduced risk of introducing regressions; major business benefits.
If code reviews are so useful, then how can I call them “waste?”
The definition of “waste” in Lean Thinking is narrower than its definition in plain English. We might feel as if anything and everything we do that contributes to the product is valuable, and therefore not waste. When we view the work through a Lean lens, we see that only the specific things the customer wants to pay for represent value. Everything else, even if it’s necessary, even if it’s helpful in an indirect way, is waste by definition.
That doesn’t mean we can just stop doing things the customer doesn’t want to pay for directly. We still have to provide a product that deals with security threats, supports regulatory requirements, is usable and accessible and localized, meets the users’ performance expectations, handles error conditions gracefully, and so forth. Customers expect those things in a software product, but they don’t think they should be paying for them one by one. They’re paying for the value the software brings them, not the mechanics of the underlying functionality or the difficulty the development team may have in implementation.
The business implication is that we want to handle all the necessary things that customers don’t want to pay for in a way that minimizes the cost and time of supporting them. In the case of code reviews, we want their value but we don’t want to spend an excessive amount of time doing them. In their book Lean Thinking, Womack and Jones call this category of waste Type 1 Muda – it’s muda (non value-add activity) that we can’t just discard from our process altogether.
If our development process includes a formal code review step, and the authorized reviewers are rare and often busy with their own tasks, then our process will have a bottleneck and a built-in delay. There’s also the possibility that the findings of a code review will require developers to revisit work they believe was already finished. That causes another form of Lean waste – rework – as well as incurring opportunity cost, as the developers can’t spend the time on new value-add work. If we can gain the value of code reviews with less overhead, we’ll be in a better position.
That’s the way increasing the degree of collaboration in our work helps us. Just using code reviews as an example, let’s take a look at the effect of different levels of collaboration on the efficiency of our delivery flow.
The chart suggests we incur less overhead in terms of time and cost when we use more-collaborative working methods. Teams that routinely use mob programming or ensemble work incur the least overhead to obtain the value of code review. Teams that routinely use pairing are close to that level. As we examine teams that use progressively less robust collaborative methods, we see the cost of code review rising. At the low end of the collaboration spectrum, teams can rightfully claim they “collaborate” to an extent, but they are not enjoying the benefits they could obtain through closer collaboration.
There are two ways close collaboration contributes here: Eliminating wait states when reviewers are not immediately available, and reducing or eliminating back-flows for rework.
I used code reviews as an example here, but the same rule of thumb applies to nearly every practice involved in software development and delivery. What’s the lowest-overhead way to ensure all interested parties have a consistent understanding of user needs? Direct collaboration across all roles in defining requirements. What’s the lowest-overhead way to ensure all necessary conditions are tested? Direct collaboration across the programmer and tester roles in developing test plans and writing executable test suites, as well as whole-team Exploratory Testing sessions. And the list goes on.
Conclusions
Software development is more complicated than it may have been in the past. Sourcegraph polled over 500 software developers in North America and concluded developers are managing about 100 times more code today than they were in 2010. As more technologies and more methods enter the picture – data analysis, machine learning, Internet of Things, microservices, etc. – the range of skills needed to build and support business applications and the infrastructures they depend on is expanding.
That means no individual can hold all these things in their head at the same time, and become a master of every relevant skill. Multiple people have to work together to get the work done. If they operate in functional silos and only communicate indirectly and occasionally, the result will be long lead times, high costs, defects, and unmaintainable code.
Collaboration has always been helpful in our field, and in the face of contemporary developments it has become almost mandatory for us to be able to handle the complexity and scope of the work.
We need to understand that “collaboration” in the context of our work means more than it does in plain English. To obtain the benefits of collaboration, we need to take explicit and mindful steps to enable it and support it in our organizations. To apply collaboration effectively we need to understand its impact on flow, throughput, and lead time.
When collaboration is challenging, we need to understand the organizational constraints that make it so, and strive to achieve the highest level of collaboration that is practical given the realities of our work environment, rather than resorting to the traditional answer to every new idea: “That won’t work here.”