The what, the how, professionalism, and pragmatism

Warning: Rant
Warning: tl;dr

So, there’s this idea floating around of technical debt. There are some nuanced definitions, but in general it means bad design baked into the code. The “nuance” has to do with why bad design is baked into the code. There are a lot of debates about this, and you might wonder why, as it isn’t intuitively obvious how anyone could argue in favor of baking bad design into their code.

Yet, they do.

Some argue technical debt is a pragmatic acknowledgement that software has to be delivered quickly enough to generate return on investment, or else the cost of building it isn’t repaid. Fair point…if we also assume that developing clean code actually takes more time and effort than building sloppy code. That isn’t a settled question.

Others argue that in today’s fast-paced world of microservices and continuous delivery, it may be cheaper and quicker to replace poorly-designed code than to clean it up. Fair point…in certain contexts, under certain conditions, to a certain extent, with certain caveats, etc. And we still have the assumption that it takes longer to build clean code than sloppy code.

Some argue that technical debt results from a well-reasoned trade-off between time-to-market and craftsmanship. Based on purely rational decision-making, which we know our business stakeholders and managers always apply rigorously, we choose to incur a bit of “debt” temporarily in order to meet a tactical goal. Later, we are always diligent about going back to clean up the mess.

And that funny little assumption that it takes longer to build clean code than sloppy code is still in play, to support the idea that there is a trade-off to be made in the first place. I have a nagging sense that this assumption may be another of those leprechauns of software engineering that I keep hearing about. But I digress.

Finally, some software development teams whine that they want to remediate the technical debt in their code base, but their business stakeholders or project managers won’t let them. The Powers That Be insist the teams hack up sloppy code as fast as possible, forever. There’s no intention of ever repaying the “debt.” There’s no recognition that there is any debt. Some, like Gil Zilberfeld, have begun to question the usefulness of the “debt” metaphor.

And there’s nothing the poor, benighted developers can do about it. They must comply, or lose their jobs.

Now I’m going to change direction suddenly and for no apparent reason.

The day laborer and the engineer

A day laborer goes to the vacant lot where people say there might be work. He waits for someone to drive up in a truck and point him out. Then he climbs on the truck and is taken to a work site. There, he may be handed a shovel and told to dig a hole. Dig the hole just there, just this wide, just this long, just this deep, just this shape. Make the sides straight. Make the bottom flat.

He digs the hole exactly where and how he is told to dig it. Then they say “Thanks for the hole” and give him some money. It doesn’t occur to them that he might be interested in the reason for the hole. He’s only there to dig. He is returned to the place where he was picked up. On the following day, he repeats the process.

Now, maybe he knows a better way to dig holes. Maybe, even, he knows a better way to accomplish the true goal of the project than to dig holes all over the place. But the relationship between an employer and a day laborer is such that he can’t raise the subject. If he questions the instructions, they will simply point to a different man tomorrow.

Besides that, a day laborer is not professionally responsible for his work. If the project does not go well, it will be the fault of the people who told him to dig the hole.

An engineer meets with a prospective client. The client explains that they need a bridge to be built to connect this side of the river with the other side. The client provides information about the purpose of the bridge, what sorts of loads it must support, how it must be integrated with the surrounding environment and infrastructure.

But the client does not tell the engineer how to build the bridge. They do not say to the engineer, “We need a bridge, so dig a hole just there, just this wide, just this long, just this deep, just this shape.” If they did it that way, then the bridge could only be as good as the client’s knowledge of bridge-building. The skills of the engineer would go unused.

Certainly, the client does not tell the engineer, “Build the bridge as quickly and sloppily as you can. We don’t care if it stands up for more than an hour. We only care that it is completed by this arbitrary deadline.” Engineers do not accept assignments like that, because they are professionally responsible for their work.

An engineer who allowed him/herself to be treated like a day laborer would be considered unprofessional.

Now it is time for another abrupt change of direction.

The what and the how

In software development, we like to separate the idea of what to build from the idea of how to build it. The business stakeholders or consumers of the software generally have a pretty good idea of what they want. Software developers generally have a pretty good idea of how to build software. So, by bringing the two together, we have a reasonably good chance of delivering software that is both useful and well-made.

But there’s a small problem.

Business stakeholders have a tendency to dictate the how as well as the what. Doing it that way, the software can only be as good as the stakeholder’s knowledge of software development. The skills of the development team go unused.

To compound the problem, software developers tend to allow themselves to be treated like day laborers. To the stakeholders they say, “Where do you want the hole? How long? How wide? How deep? What shape?” To one another they say, “Can you believe what they’re asking for? Are they crazy?” Then they go ahead and dig the hole exactly where and how they were told to dig it.

When advised to take control of their own work and do the right thing, many software developers say they aren’t allowed to do that. They must ask permission of their stakeholders to do basic things like writing automated checks and refactoring their code.

To me, this sort of excuse is neither more nor less than a decision on the part of software developers to be treated like day laborers instead of professionals.

In the Proceedings of the 1968 NATO conference on software engineering, computer science pioneer Alan Perlis wrote:

“A software system can best be designed if the testing is interlaced with the designing instead of being used after the design. […] Through successive repetitions of this process of interlaced testing and design the model ultimately becomes the software system itself.”

There really is no valid excuse not to use the available tools and methods to assure high-quality code. There hasn’t been any such excuse at least since 1968.

I asked several colleagues who offer advice to software development professionals how they respond when developers insist they don’t have the option to apply good development practices, because their stakeholders or managers won’t let them.

The highly respected developer, author, and trainer J.B. Rainsberger pointed me to an article of his, The eternal struggle between business and programmers, in which he examines the common goals of stakeholders and development teams and how the disconnect over the what and the how can occur.

One of the creators of Extreme Programming and a signatory of the Agile Manifesto, Ron Jeffries, answered this way (quoted with permission):

Bair’s Law: Don’t ask a question unless you want to hear the answer. In particular, dont ask permission for things you need to do.

Grace Hopper: It’s easier to ask for forgiveness than to ask for permission.

Scrum: Product Owner says what to do; Development team says how.

Everyone with a clue: Testing and refactoring make you go net faster, not slower. Testing prevents defects, which takes far less time than fixing them and is far more predictable as well. Refactoring (done in the right places) makes the code easier to modify and therefore allows us to go faster.

Me: I explicitly make clear that refactoring and testing are not business decisions in the sense of belonging to the Product Owner. They are not comparable with the kinds of value-based decisions the PO makes: they are cost-based decisions, so the sorting algorithms all blow up. So we just don’t ask.

Instead, we work clean, which results in moving at the best possible pace. We keep the code clean, and when it gets crufty, as it will, we clean up according to the “Boy Scout Rule”, which is to leave the campground cleaner than we found it. We don’t go hunting down dirty places to clean: we clean the ones we visit.

We work as close to defect free as we can, because there is no case in which anyone wants a defect. We prevent them, by checking our code as we go, so that in every moment, we are quite confident that it does what we’ve been asked to do. When, as they will, our checking falls short, we bear down a bit and write more. In particular, if a defect gets discovered, we improve the checking in the neighborhood of that defect.

We don’t ask permission to do our work well, any more than we ask permission to use if statements or for loops. The mechanic doesn’t ask you what wrench to use: he knows what wrench to use. The programmer doesn’t ask the Product Owner when to test or refactor: he knows when to test and refactor. (And that time is now, and always.)

Proper professional practices are not optional and are not up to business stakeholders to authorize.

Fear of job loss

After hearing that sort of advice from their helpers, many developers resort to saying they will be fired if they “push back” on stakeholders or management regarding the use of proper development practices. It’s the last, weakest excuse they can think of.

A few years ago there was an economic downturn in the United States. I haven’t gone back in search of the exact numbers, but as memory serves we hit a high of around 9.5% average unemployment nationwide. At that time, the unemployment rate in the IT sector was about 2%. Bear in mind when unemployment is in the 4% to 5% range, we are at “full employment.” That’s normal job churn. Those numbers might not be precisely correct, but they’re close enough to make the point.

That means when the economy was at its worst since the Great Depression, software development professionals were still in better shape than other workers during periods of full employment. So, why the fear?

There’s another reason not to fear. Consider what it takes for your employer to replace you. Assuming you’re any good at all as a developer, it will take them a good six months to find, recruit, and ramp up a replacement for you. Do you honestly believe management will skip through the halls of your company merrily firing people left and right for no good reason? They would be crippling their own company.

Finally, there’s the question of who is responsible for your career. Is it your employer, or is it you? If you are truly in a job where management and stakeholders simply do not understand the fundamentals of all this, and where you know you will always be treated as a day laborer instead of as a professional, then why the hell are you still there?