Can a new dog learn old tricks?

[This was originally published on my old blog on 18 September, 2007. A friend recovered the post from Time Machine. There were some good comments on the original posting, but those were not saved by Time Machine. Unfortunately, the attitude I observed 5, 15, and 25 years ago hasn’t changed much. Enjoy.]

A few months ago, I used the phrase “the collective experience of the industry” in an online discussion about good practices in software development. I got slammed. We have no “collective experience,” I was told, we just have “a bunch of individual experiences.”

At the time, I thought it was incorrect to say that we have no collective experience. Software developers encounter many of the same situations and solve many of the same problems over and over again, in different domains and in different technical contexts. At the time, it seemed to me that the common aspects of these experiences could form the basis of a corpus of shared professional knowledge.

In the professions, practitioners embrace the collective experience of their peers. They use it to build a foundation for improving their profession and as the core knowledge new practitioners must learn before they are considered fully qualified to practice independently. Doctors, lawyers, engineers, and certified accountants must work under the guidance of senior members of their professions for some period of time after completing their education. They learn what others have already learned so that they will not repeat old mistakes and reinvent solutions that are already known, possibly doing avoidable damage along the way.

If we really have no collective experience as software developers, then we have no basis on which to recognize good practices, no way to identify the warning signs of common pitfalls or risks, no understanding of repeating patterns in our work. If we truly lack a sense of collective experience, then we are all doomed, individually, to repeat the past mistakes of others, even if solutions to those problems have already been discovered.

At the time of the online discussion, I assumed our occupation, software development, was also a profession, and that practitioners viewed themselves and their work in the same way as other professionals view their own. I was wrong. The truth is, we just have “a bunch of individual experiences.” The reason for this is particularly discouraging: Most of us actively avoid learning from the experiences of our peers.

I was reminded of this during a recent presentation by Object Mentor consultant Brett Schuchert at the Oklahoma City JUG. The topic was Java 1.6 support for concurrent programming. But it wasn’t the presentation itself that reminded me we aren’t a real profession, nor even the anecdotes Brett shared, but rather the reaction of the participants to a couple of things.

Brett mentioned the AtomicInteger class, which is part of the java.util.concurrent.atomic package. When he explained that AtomicInteger would use the compare and swap instruction if the underlying hardware supported it, participants were surprised — pleasantly surprised, but surprised nonetheless — that such an instruction existed.

Yet, compare and swap has long been a part of the venerable IBM line of mainframe operating systems. In that environment, high-performance concurrent programming has been commonplace for decades. Multithreaded execution was supported many years before someone coined the word, “multithreaded.” What we would call threads were called subtasks, and concurrent I/O processing was supported by the use of multiple asynchronous channel programs. It isn’t a new trick.

Brett shared an anecdote about a time when he converted a single-threaded, long-running, I/O-bound batch process to use multiple threads for I/O, and explained how the change reduced run time significantly. Again, participants were surprised to learn how much run time can be reduced by doing this. And again, this is an old technique. In the heyday of the mainframe, we used the same technique routinely for very long-running batch jobs. In many cases the run times could be reduced from 20 hours or more to less than 30 minutes. It isn’t a new trick.

Unfortunately, innocence of what has gone before is not a recent phonomenon in our — I wish I could say profession — line of work. A good ten years ago, I attended a training class on the application server that was then called Tengah, now known as WebLogic. I was shocked to learn that in order to change a configuration value of any kind, it was necesary to stop the server. This was a middleware server. All traffic between front-end and back-end systems passed through it. When the middleware is down, everything is down; and at this stage in the product’s evolution, there was no clustering.

This was another common problem that others had solved many years earlier. High-availability systems from Tandem and Stratus were well established, and in the mainframe world products that had to provide high availability to applications, such as VTAM and CICS, had been configurable on the fly for many years. These products were originally written so that they had to be shut down to make changes, but earlier generations of practitioners had learned the lesson and fixed that problem. It wasn’t a new trick.

Why, in the mid-1990s, would perfectly intelligent people create a middleware server that could not be reconfigured on the fly? How could it have occurred to them that it would be acceptable to shut down a middleware server, and by extension every application in the shop, just to change a resource definition? Why would they repeat problems that had been solved twenty years earlier? Why did they wilfully ignore lesons learned by thousands of their peers going back decades? It was a question of attitude. The developers simply assumed that all technology and all people who had gone before were dinosaurs with nothing to offer.

That attitude did not begin this year or ten years ago. I recall similar experiences from 20 years ago, in the era when microcomputers and distributed processing were just beginning to take center stage in place of mainframes and minicomputers and their centralized processing model. That generation of software developers took the same dim view of the past as their descendants in the 1990s and 2000s would do. They ignored the lessons learned up to that point, and worked as if they, personally, were discovering how to write software for the first time ever.

Early adopters of distributed solutions were eager to replace their expensive mainframe contracts with cheap, off-the-shelf hardware (IBM XT level equipment, mainly) running single-user operating systems like DOS. They soon found themselves in the proverbial ditch, unable to complete their nightly batch processing before morning, and unable to support high-volume OLTP applications. Furthermore, they quickly learned that these newer, cheaper systems lacked many of the support features they had taken for granted in the older mainframe environment, such as reliable backup and recovery, security, and robust workload management. The cool new systems didn’t have those things even though they were not new tricks, even in the 1980s.

The fundamental problem through all these years has been attitude — the attitude that nothing useful has ever been done before, and we have to invent everything from whole cloth as we go along; the attitude that whenever a new technology comes along to replace an old technology, it replaces principles of software development as well; the attitude that we have no collective experience as an industry, we just have “a bunch of individual experiences.” If software development wants to be a real profession when it grows up, we need to drop that attitude and start treating our work in the way doctors, lawyers, engineers, and certified accountants treat theirs.