A modest proposal for restructuring enterprise IT

Submitted for your consideration: The simplest way to solve a problem is not to solve it at all, but to define it out of existence. Many of the challenges we face in corporate IT may be caused by traditional organizational structure. If it is true that structure begets function, then we may be able to define some of these challenges out of existence by changing the organizational structure.

Premise 1

Large enterprises usually have difficulty moving en masse to a lightweight approach for software development and systems support. Some individuals and teams embrace lightweight methods while others insist that traditional methods are necessary for their cateogory of work. Most people do not appear to be passionate about either approach; they go along with whatever management says will affect performance reviews and raises.

Premise 2

Since the Agile Manifesto was published in 2001, there has been a concerted effort to transform IT organizations from traditional thinking and methods to “agile” thinking and methods. Although many would argue that “agile” has crossed the Chasm and become a mainstream option for software delivery, others are dissatisfied with the degree of penetration of the approach into IT work generally. This is especially true of “agile” implementations in large organizations. These tend to be “agile in name only” to a great extent. Good results with lightweight methods seem to occur only in isolated pockets in large organizations, and to be temporary.

Premise 3

“Agile” thinking and methods are not for everyone. The “agile” mindset calls for individuals to reason through first principles on their own, then to craft processes based on those principles, and then to follow up by continually paying close attention to the way things happen so they can tweak the process to improve delivery. Most people, it turns out, would rather be told what process to follow and what practices to use in their work, and to have a stable, predictable work environment where they always know what is expected of them. They want to spend their days working in their chosen area of specialization – testing, programming, analysis, management, or a favorite platform or product – and leave process definition to others.

There are no hard-and-fast rules, no one to blame when things go wrong, and no predictable steady state. “Agile” methods demand a high level of technical expertise and self-discipline on the part of team members, as well as a level of transparency and trust that many people find uncomfortable or threatening. These methods also call for close and nearly continuous direct collaboration between business stakeholders and technical staff. In most large organizations, business stakeholders are uninterested in participating with technical projects in this way.

Premise 4

The IT department of a for-profit enterprise generally performs two types of work: (1) Maintenance of a functioning technical infrastructure that can support the capabilities the business requires, and (2) Development and support of market-differentiating applications that help the enterprise gain market share and enhance revenue.

Premise 5

Strategic IT assets tend to be expensive, rarely replaced, shared across the enterprise, and centrally managed. Implementations can be closely tied to specific products to take full advantage of product-specific features, and companies may have partnership relationships with providers as opposed to simple producer-consumer relationships. Supportive systems in the “keep-the-lights-on” category may reasonably be outsourced.

Market-differentiating applications tend to be tactical assets. They need to be easy to build, easy to replace, quick to deploy, vendor-independent, and aligned with business processes rather than with technologies. It is not reasonable to outsource this work.

Strategic and tactical assets call for different organizational structures, personnel selection and incentives, standard procedures, governance mechanisms, funding models, development methods, and change controls.

Premise 6

Lightweight software development and delivery methods tend to be highly valuable for developing market-differentiating applications. Early delivery of high-value components, frequent feedback from the market and from internal stakeholders, and incremental funding help IT departments deliver critical business capabilities with short lead times, close alignment with business needs, and high quality.

On the other hand, the process overhead of frequent feedback and frequent deployment does not yield the same benefit when applied to routine technical infrastructure maintenance work. That sort of work lends itself to traditional methods, because there is low uncertainty, low time-to-market urgency, and (often) a fixed expense budget.

Premise 7

In the context of software development and delivery, sources of “waste” (as defined by the Lean school of thought) tend to be – in order of impact – (1) organizational structure, (2) formal processes, and (3) technical practices and tools. To reduce friction, cost, and time-to-market, we want to address sources of waste in the order of impact. Organizational structure is the highest-impact area. When the structure is suboptimal, it is difficult to achieve gains in the other areas.

Proposed structure

For technical infrastructure and enterprise IT assets: Central IT department. Traditional mindset and methods. Prescriptive process. Standards treated as “rules” and justification required to deviate from them. Technical staff who value stability and predictability will find job satisfaction here.

For market-differentiating softwre: Technical teams included in business units – not in a separate department. Agile mindset and methods. Flexible process. Standards treated as “guidelines” and justification required to override development team choices regarding technical options. Technical staff who enjoy a variety of challenges will find job satisfaction here.

Technical structure: A SOA environment to decouple tactical applications from centrally-managed IT assets. A clean interface between the two. This enables robust central governance without impeding the rapid delivery of market-differentiating applications.

Expected benefits

IT assets that require traditional governance can be managed appropriately while market-differentiating work can be done efficiently and effectively. This provides risk management alongside competitive advantage; two factors that are usually at odds.

Technical staff who have different personal and professional needs can find assignments in the organization that meet their respective needs. This has positive implications for the costs of recruiting and retaining staff.

Removing the separation between application development teams and their business stakeholders eliminates the overhead of “requirements management,” project initiation, and related process overhead. It also makes direct collaboration between business and technical personnel normal, rather than a matter of training and coaching that may or may not “stick.” Structure begets function.

Expected challenges

The suggested mechanism to connect tactical and strategic IT assets is a SOA environment. Many companies have attempted this, with mixed results. Most of the attempts I am aware of have not turned out well. None has resulted in the sort of reality people were hoping for when they began, even after many years and many millions of dollars spent. Point to point interfaces remain in place, and many legacy systems have not been integrated nicely with the SOA solution. These problems have to be solved before the proposed organizational structure can work.

Recruitment, hiring, incentives, and retention of technical staff becomes at least twice as challenging as before. The organization must develop awareness of the deep differences between individuals who will feel at home in the central IT area and those who will feel at home working on market-differentiating applications in a business unit. In many respects, these are two very different worlds, and people cannot be shuffled between them at will without risking turnover.

Managers in business units may not readily accept the idea that they need to have technical staff in their own chain of command. To some, it may be obvious that the ubiquity of software in business naturally calls for business people and technical people to work directly together, but this will be a very significant change for many who have traditional assumptions about roles and functions within a company.

Further information

Utility vs. Strategic Dichotomy (Martin Fowler)
http://martinfowler.com/bliki/UtilityVsStrategicDichotomy.html

IT Doesn’t Matter (Nicholas Carr)

IT Doesn’t Matter

Purpose Alignment Model (Niel Nickolaisen)
http://www.informit.com/articles/article.aspx?p=1384195&seqNum=2