Refactor Anyway

Have you heard or read statements like the following?

“Tests are the wall at your back. You gotta have tests or you don’t know what you’re doing.” ( Sandi Metz video; time index 9:35)
“Not using an IDE with refactor tools like the ones discussed above is a waste of time.” ( Brian Ambielli)

I’ve seen a lot of people paralyzed by this advice. But why? It’s good advice, after all.

I think the problem is advice like this assumes the listener has a certain understanding of what software development work entails, and an ability to synthesize information and apply new techniques in context and with the benefit of substantial experience that includes particular activities and skills.

Absent those conditions, advice like this can scare people. They assume they literally cannot or must not attempt to refactor code unless the code is already well covered by a comprehensive and meaningful suite of executable checks, and they have the privilege of using very specific tools that verify the refactoring is completed safely.

There are many examples of off-putting language. I’ve heard:

“It’s unprofessional not to test-drive your code.”
“Only an amateur uses the mouse with an IDE.”

The choice of words can shut people down. There are countless statements like the ones cited above, in articles, books, blogs, websites, conference talks, videos, and training classes. What most people hear is no more than:

You don’t know what you’re doing.
You’re wasting your time.
You’re unprofessional.
You’re an amateur.

And they stop paying attention.

Of course, that’s not the speaker’s or writer’s intent.

“You don’t know what you’re doing” is shorthand for “refactoring without tests carries risk of modifying system behavior unintentionally”.
“You’re wasting your time” is shorthand for “refactoring support built into IDEs (or provided by an IDE plugin or extension) will save time when working with existing code, especially when the tool performs post-refactoring processing to detect unintended changes”.
“You’re unprofessional” is shorthand for “professionals want to learn and apply the best practices available”.
“You’re an amateur” is shorthand for “using keyboard shortcuts for common IDE operations is often quicker than using the mouse to select menu items”.

The speaker’s or writer’s intent is important, but the impact of words on the listener or reader is more so.

It seems to me the problem is the following combination of elements:

conditional phrasing that implies blame or shame (you’re choosing not to be [desirable thing to be])
negative verb phrase (you’re not [desirable thing to be]; or you’re not doing [desirable thing to do])
second person personal pronoun (the problem is you, not me; that’s you over there, choosing not to be [desirable thing to be], while I’m over here, above you, better than you, judging you)

So, I could say, “If you don’t think about how you express your ideas, no one will care what you have to say.” But that sentence just another example of the same problem, isn’t it?

Maybe I should say, “It’s helpful to try and express our ideas in a way our audience can accept, so they might be willing to consider our advice.”

That formulation has the following combination of elements:

nonjudgmental statement (my idea is [thing])
positive verb phrase ([thing] is good because [expected outcome])
inclusive personal pronoun (we’re in this together)

A minimalist approach

OK, what about refactoring, then? We can find practical advice for working with existing code along these lines:

With an approach like the ones suggested in those articles, developers try to write (or salvage) executable checks that validate system behavior at whatever level it’s possible to exercise the code in a controlled way. Depending on the structure of the code, that may or may not be the “unit” level. In most cases, it isn’t at the “unit” level, even when the phrase “unit tests” is used.

I call those approaches “minimalist” because they avoid remediating existing code at a fine-grained level. They recommend test-driving (or at least writing unit-level checks for) any new code, but they leave existing code alone, for the most part. So, we perform only a minimum of refactoring, to try and achieve some degree of meaningful test coverage.

This is a risk-mitigation strategy. We avoid trying to refactor any more than absolutely necessary to make tests work at whatever level the current design of the code will allow. There are many situations in which this is a sensible approach.

A careful approach

Another approach, described in Michael Feathers’ book, Working Effectively with Legacy Code, is to attack monolithic code by looking for the natural “seams” in the code, and pulling those sections out into separate methods, functions, or modules. Then we can write executable functional checks against the extracted source units.

That becomes a starting point for gradually improving the design of the code through ongoing, incremental refactoring.

People who practice incremental refactoring regularly usually discover that potential design improvements gradually become easier to see over the course of many small changes. There’s a concept from Sandi Metz that I find very useful: Source code has shape. By cultivating a sense for code shape, we can perceive opportunities to make useful improvements in the design through incremental refactoring.

Her article, The Shape at the Bottom of All Things, is a step-by-step walkthrough of a series of refactorings on a piece of Ruby code. I recommend reading this article deeply, and not just scanning it. It describes a practical way to begin teasing out the hidden good design in a piece of existing code whose design is questionable.

Metz suggests some shapes are better than others. I interpret that to mean some shapes offer a clearer path to further design improvement than others. She gives an example of this near the end of the article. It’s helpful to try and cultivate the skill to recognize which shapes are “better”, in this sense.

I call this a “careful” approach as opposed to a “minimalist” approach because we do address design improvement all the way down to the smallest code units. We do it in a mindful, stepwise way, adding executable checks (test cases) as we go.

How can we cultivate this skill? It may seem counterintuitive, but I think a good way to begin is to stop trying so hard. Humans and their evolutionary predecessors have survived for millions of years by developing a natural ability to perceive patterns in their environment. Hiding in the tall grass, just there…a sabre-toothed cat. Peeking out from the underbrush across the field…edible berries. All we really see is the barest hint of those things; our brains automatically fill in the gaps, and we see the cat or the berries clearly in our minds.

Things to avoid and things to seek out present themselves as patterns in our environment. We need to remain open to perceiving them.

When we overthink it, we tend to see patterns that aren’t really there. If you have a minute, more-or-less literally, then listen to this short audio piece from Scientific American, “Brain Seeks Patterns Where None Exist”.

I’m reminded of Geepaw Hill’s idea that there’s a difference between scanning code and reading code. When we read code deeply, we have to concentrate on it and think about what we’re reading. When we scan code, we let the code flow into our visual cortex without trying to understand it on a deep level. That’s when the patterns or shapes in the code make themselves apparent.

We can learn to perceive these shapes and pause our scanning when we think there’s something interesting to read more deeply: A sabre-toothed cat (a code smell hiding in the tall grass, ready to cause a regression as soon as we change the logic); edible berries (a domain concept waiting to be extracted from the underbrush of conditional statements).

An aggressive approach

The opposite of a minimalist approach is an aggressive one in which we don’t hesitate to refactor code, even when we’re reading it for the first time.

A leading software development practitioner, Arlo Belshee, advocates teams adopt the mindset of bug zero. He uses the analogy of inbox zero to explain it. Watch the interview with Agile Amped at AATC2016 to learn a bit about it. For a longer presentation that goes more deeply into the underlying principles and value proposition, see his keynote from AgileIsrael 2017.

When a team makes it a guiding principle to tolerate no bugs of any kind, it leads them to take a much more aggressive approach to refactoring than we usually see in the field. Arlo teaches a couple of closely-related practices to make this approach feasible.

The key practice is called read by refactoring. The idea is that we immediately clean up code “in the small” as we are reading it; even when we are reading it for the first time, and trying to learn our way around an unfamiliar code base.

If that sounds scary, it’s because it is scary. There are a couple of things that make it feasible. One of them is another idea of Arlo’s – naming as a process. He observes that programmers spend more time reading code than writing it. Therefore, it makes sense to optimize code for readability rather than to focus on code entry speed.

Many have observed that one of the hardest problems in programming, if not the hardest, is naming things. It’s rare that we can think of the most descriptive and intention-revealing names immediately. Instead, it’s likely that we can think of better and better names as we learn more and more about the code base, and we incrementally improve its design through refactoring. Arlo has written a series of articles to explain the concept and its application.

The combination of read by refactoring and naming as a process enable us to act on our mindset of bug zero…but only when we are confident we can refactor freely without breaking things. What gives us that level of confidence? Development tools that support “safe” refactoring.

Safe refactoring

In a Twitter exchange some time ago, Arlo told a story of an error that caused some grief for a team he was working with. He said a refactoring had gone wrong and changed the behavior of the application in some small way. Had the team not caught the error right away, the consequences could have been costly for the company.

He made a point of learning how various development tools support refactoring, and which ones had the best support for safe refactoring. He concluded there were two good IDEs from that perspective: Microsoft VisualStudio and JetBrains IntelliJ IDEA. Other IDEs, including those with basic refactorings available as a menu selection, were less safe. Text editors? Forget it!

Advocates of the approach usually express themselves dogmatically on this subject. Most will insist that you can’t work in this way unless you’re using one of those two IDEs. There’s an unfortunate side-effect of their exuberance: It leads other developers to believe they can’t refactor the legacy code bases they support. It seems not all legacy code was written in C# or Java, and not all development environments have an IDE that supports safe refactoring…or any refactoring that we don’t do “by hand”.

Does this mean the read by refactoring approach is infeasible unless we happen to be working with C# or Java? I suggest the approach is (probably) always feasible. Here’s why.

IDEs that offer basic refactoring support are safe enough, even without the post-refactoring validation algorithms used in VisualStudio and IntelliJ IDEA. I wouldn’t say the error in Arlo’s was a refactoring gone wrong. If the modification changed the behavior of the application, then it wasn’t a refactoring at all, by definition.

Tools have no idea what we’re doing. They just do what we tell them to do. We can swing a hammer clumsily and bend the nail. We can swing it true and drive a screw into the wood, spoiling the piece. We can smash our own thumb. The hammer doesn’t know. Getting a better hammer won’t solve the problem.

In the story, the team must have done something in addition to invoking the IDE’s refactoring operation; something that changed behavior. I don’t think the lesson to take from that experience is that we can only dare to refactor when we’re using a particular IDE. If the problem was not caused by the development tool’s refactoring support, then using a different development tool won’t solve it.

If our tools don’t know what we intend to do, and they can only perform the operations we tell them to perform, then whose responsibility is it to ensure refactorings are completed properly?

Yes, you’re right: We are. If it’s our responsibility to ensure refactorings are completed properly, then which development tools are okay for us to use?

Yes, you’re right again: Any tools. Some tools will make the task easier for us than others. But it’s still our job as software developers to take proper care of the code we support.

The full-featured IDEs help with refactoring, but they don’t handle everything for us. Remember the incremental approach. A meaningful improvement in code design might involve a number of small refactorings followed up by minor hand-made edits.

For instance, Extract Method makes a guess about which variables should be local in the extracted method and which should be arguments passed into the method. It doesn’t always guess well. Similarly, we don’t often find precisely duplicated code to extract neatly into a new method in a single go. Instead, we find almost-identical code in multiple places. The automated refactoring operations of the IDE don’t handle that entirely.

So, even with these tools, we’re still left with a certain amount of “manual” work. Using other tools, the amount of manual work is greater. But there’s no magic bullet.

Not all “legacy” code is written in a language for which the two full-featured IDEs support safe refactoring, even if they might support some level of help such as syntax highlighting. There’s a fair bit of legacy code out there written in C/C++, Python, Ruby, PHP, and a very large quantity in COBOL. There are IDEs for these languages, but none that include the level of safe refactoring support that VisualStudio and IntelliJ IDEA offer for C# and Java respectively. Even the other JetBrains IDEs don’t offer that for languages like Ruby, Python, Go, C, etc.

Nonetheless, we must do our jobs. In my view, read by refactoring and related practices are useful even when we don’t have the option to use an ideal development tool. We just have to exercise due diligence as best we can.