Open-plan offices, in particular, seem an impressively terrible idea. "Open plan layouts create massive distraction, damaging productivity," according to a recent analysis by the U.K.’s Channel 4. See also the related Hacker News commentary, which includes gems like "Most modern office layouts seem to be designed to screw with people’s fight or flight instincts all day," and "I’m a quiet type, and I often need time alone to think and write code and documentation. The ‘rah rah’ social types railroaded us…I am becoming bitter and resentful."
The context of the article was a software development practice known as pair programming. As evidence that pair programming ought to be used with caution, the paragraph conflates the general notion of "open plan layouts" with the context-specific concept of work spaces designed for collaborative product development. The cited study by Channel 4 concerned open plan offices in general. You’ve probably seen an open plan arrangement before: A large space, possibly spanning most of the available floor space, fitted with cubicles having very low walls, or no walls at all between desks.
The study found — well, actually it didn’t find anything, it just quoted an earlier study that someone else had done — "open plan office noise reduces the productivity of knowledge workers (people trying to manipulate words or numbers in their brains, for example to write a report or plan) by a staggering 66%!" So, they’re talking about people who are writing reports or plans. They aren’t talking about cohesive, cross-functional teams that are collaboratively developing a new software product, in a work space that has been designed to support such work. I don’t blame them for feeling distracted!
The remaining references in the paragraph are quotes from a comment stream on Hacker News in which people relate negative experiences working in environments that were not properly structured to support their needs. What this communicates to me is not that collaborative work spaces are categorically a bad idea, but rather that poorly-designed collaborative work spaces are a widespread problem.
People misunderstand how to configure collaborative work spaces for software development teams. I tend to think structure begets function. I think that rule of thumb applies at all levels, from the organizational structure of an enterprise to the physical set-up of work spaces. Work that benefits from a collaborative approach is best done in an appropriate work space. I’ve seen too many examples of poorly-thought-out team work spaces. Articles, comments, and "studies" like these don’t help, because they blame negative outcomes on the concept of a collaborative work space rather than on poor implementation. If you hit your thumb with a hammer, is it the hammer’s fault?
Most books in our field are time-sensitive and become obsolete quickly as technology advances. A few books remain relevant even as technology changes. One of these is Peopleware by Tom DeMarco and Timothy Lister. Chapter 13 of the second edition describes the sort of work environment a collaborative team needs to carry out creative product development work. Far from an open plan, such an environment comprises three types of space: Collaborative, semi-private, and private.
The collaborative area is where team members work together. When people want to work in pairs or small groups, this is where they work. When people want to work individually, but want to be close to their team mates, this is where they work. The team is also protected from interruptions and distractions from outside their world.
If you haven’t worked in this way before, you might wonder why anyone would choose to do this. They are taking advantage of what Alistair Cockburn has called osmotic communication. It’s a simple and powerful way to maintain good communication on a team. The general buzz in the collaborative work space isn’t just distracting "noise" as it would be in a generic open plan office, because it’s all relevant to the shared goals of the team. People learn (quickly, it seems) to tune out the chatter that doesn’t interest them, and to perk up and join in when they overhear something relevant. I understand this may be counter-intuitive, because I was unsure about it too, until I experienced it.
All noise isn’t useful, though. There are times when an individual needs a block of time to work quietly alone, or a small group needs to have a discussion or working session that might be distracting to others in the collaborative space. Semi-private areas equipped with whiteboards, chairs, and anything else the team needs are used for brainstorming and design sessions. Private spaces, sometimes shared hotel-style and sometimes dedicated to individuals, are used for one-on-one conversations and personal phone calls.
You might also hear this arrangement called caves and commons. The caves are the private spaces, and the commons are the other two types of space. Just different terms for the same idea.
It’s a common mistake to provide teams with only the collaborative space, and to neglect the semi-private and private spaces. When that happens, you have the cases that Channel 4 and the Hacker News commenters are talking about.
Many configurations are possible. Here is a Google query that will search for photos of team rooms: https://www.google.com/search?q=agile+team+rooms&tbm=isch. You can see that the idea is quite different from an open plan office.