Years ago, as far back as the mid-1980s if memory serves, IT managers were worried about a phenomenon they called “shadow IT.” The term refers to cases when business managers implement their own departmental IT solutions in response to poor support from their internal IT departments. The phenomenon raised alarms in the minds of IT management. They were losing “control.” Others were treading on “their” territory. Lions and tigers and bears! Oh, my!
I was reminded of this when I saw an announcement of an upcoming webcast entitled, “What’s a CIO to do About the Rise of Shadow IT?” This suggests that many IT managers still don’t get it. So-called “shadow” IT is not a “problem.” It is not an intrusion on anyone’s sacred territory. It is an indicator that something about the status quo is causing people’s needs to go unmet.
The question should not be, “How dare they try to take over our territory?” The question should be, “What are people trying to tell us about their needs?”
The story goes (and I can’t find the original source just now) that one fine day a university constructed a new campus, and they were wondering how best to design the walkways to connect the various buildings. A number of design proposals were floated, and they all looked beautiful on paper and in the form of models. Fortunately, it occurred to someone with decision-making authority that it would be sensible to let students use the campus for a year and then see where they had tread on the grass. That would indicate the most logical paths for paved walkways. Starting in Year Two, students walked on paved pathways and didn’t tread on the grass, since the pathways were located just where they were needed.
“Shadow IT” is nothing more sinister than students walking across the grass on their way to and from class, choosing the most effective pathways to support their day-to-day activities. The trails they leave show us where we ought to build paved walkways. We could go ahead and build the walkways wherever we please, insisting that only we, the Central Walkway Authority, could possibly know what is best when it comes to walkways, and people ought to walk wherever we tell them to walk. Of course, the students will still walk the way they need to walk to get where they need to go. They don’t care where we think the walkways should be. They only care about going to and from class.
The trails in the grass left by “shadow IT” show us how IT resources can best be organized to support the needs of business units. They’re telling us that they need the software applications that support their operations to be part of their own organizations. The traditional relationship with the IT department as a service provider does not support the needs of business units adequately, even when IT management goes to some lengths to “align” those services with the administrative structures of the enterprise. We need integration, not alignment.
When a business unit needs a minor change to an application that only supports their operations, it does not serve them to have to spend $100,000+ and wait 4+ months just to get a “yes” or “no” from the IT department about whether they are willing to make the change, and then wait another year and spend another big chunk of money before they see the change in place. It’s better if the business unit has its own little team to support its own little set of applications running on its own little servers. Enterprise IT assets can be accessed through an ESB based on a SOA architecture using well-defined interfaces. Security, scalability, and governance can be supported behind the ESB. Multiple QOS levels can be supported behind the ESB, too. It isn’t necessary that anything and everything that involves computers be centralized. It’s only necessary to centralize those IT assets that legitimately ought to be centralized. Not to worry; there will still be plenty for the central IT department to do.
SOA and ESB
Well, I’m glad that’s been cleared up, and…what’s that? What are those acronyms, you ask? Sorry, I thought you knew. SOA stands for service-oriented architecture. It’s a conceptual architectural model for providing access to centralized, shared enterprise IT assets in a controlled manner. ESB stands for enterprise service bus. It’s a fancy term for an implemented interface to the aforementioned assets. QOS level means quality of service level. It refers to a set of promises about the capacity, responsiveness, availability, and recoverability of assets on which departmental applications depend, and to which different charge-back rates can be assigned.
I assumed you knew about those, as the approach has been strongly recommended by IT professionals for at least a decade now. If you don’t already have something along those lines in place, then you’re late to the game. Late to the 21st century, some might say. If you’re only now hearing about this, then I’m sorry to be the bearer of bad news. Please don’t shoot the messenger!
Here’s an important point: SOA is not the solution, it’s only an enabler. The solution is the restructuring of corporations such that central IT assets remain with the central IT department, and assets that support only local operations are integrated with the business units responsible for those operations. SOA enables this sort of structure by providing a practical means to achieve economies of scale, regulatory compliance, technology standardization and consistent roadmaps for upgrades, security, capacity management, and sharing of cross-departmental assets without hindering business units in the pursuit of their objectives.
Strategic vs. tactical assets
One of the key ideas in this is the distinction between strategic assets and tactical assets. A strategic asset is anything that supports the corporation’s long-term existence and success, or that is shared by two or more business units. A tactical asset is anything else. Thus, the enterprise Data Warehouse facility is a strategic asset; the Customer Relationship Management package is strategic; the ERP system is strategic; shared tools like legacy integration platforms, information portals, pooled hard drives, virtualized server farms, interfaces to external services such as credit card processors, and business rules engines are strategic.
But the Consumer Loan Application program is tactical. It’s only used by the consumer lending department. It does require access to the CRM system and other shared assets, but that in itself doesn’t mean it has to be owned and operated by the central IT department. It only means there has to be a well-known and consistent way for it to access the shared assets. It does so by making requests through the ESB.
Professionals in the IT department might argue that if the Consumer Loan Application program is badly written, it will be difficult to maintain and will have a relatively short production lifetime. Unless the coding is done by high-end professionals in the IT department, the business unit is at risk. They may be quite right about the technical quality of the code. What they overlook, perhaps, is that this really doesn’t matter. Because the loan application program is a tactical asset, it is, in a sense, disposable. As customers shift from using personal computers with web browsers toward using internet-connected television sets or smart phones, the loan application program is going to change anyway. A lot. You probably wouldn’t even use the same programming languages for those various client devices. And the business rules around loan approval and servicing will change, as well. Any given version of the program will be retired before technical debt becomes an issue.
The capability to initiate consumer loans is strategic, but any single implementation of software to support that capability is, in itself, tactical.
Similarly, if we have an automated packing room where a computer-controlled line folds cardboard boxes, fills them with our product, and transmits shipping manifests and routes to trucks waiting at the loading dock, there is no business value-add in having that software written and supported by the central IT department. It doesn’t even need access to the ESB. It runs in a self-contained universe.
The capability to pack and ship product is strategic, but any single implementation of the packing room control software is, in itself, tactical.
On reusability
One of the arguments against this sort of structure typically comes from the enterprise architecture group within the IT department. They claim that in order to control software development and maintenance costs, avoid duplication of effort, and eliminate redundant applications, any component of an application that might be reused in a second application ought to be designed and coded as a service, owned and supported by the central IT department, and live behind the ESB. There is merit to this argument. However, what happens in practice is that people start to think because any component of any application might someday be reused in a second application, every component of every application must be designed and built as a service. Otherwise, we might someday miss an opportunity for reuse.
In my view, this sort of argument comes from a failure to distinguish correctly between strategic and tactical assets. Because the local business applications developed in each business unit are tactical by nature, it really doesn’t make much difference if two or three of them happen to have components that are similar enough that they might conceivably have been written as shared services. Tactical assets have a limited lifespan anyway. In every organization I’ve seen personally where this has been done, the sharable components are hardly ever used by more than one application. A lot of effort, time, and money is sunk into making components into sharable services, but very little sharing actually occurs. It is a non-issue.
Okay, so what if we are talking about a strategic asset? Here’s a common situation: A company has a very old legacy system for maintaining customer information. Many companies call it a Customer Information System (CIS) or a Customer Information File (CIF). Maybe it was written in IBM mainframe assembly language in 1968. Maybe it was written in COBOL in 1980. Maybe the customer information lives in a database and maybe it lives in a set of master files. Doesn’t really matter. The thing is, even if it’s called a “file,” it’s really an application or a suite of applications. That means business rules are buried in its source code; source code your wet-behind-the-ears programming staff wouldn’t understand even if they had security clearance to look at it.
So, a few years ago your company invested in a Customer Relationship Management (CRM) system. There are a lot of them out there. Here’s a list: http://en.wikipedia.org/wiki/Comparison_of_CRM_systems. A CRM system is a commodity product. Yet, the specific features and the APIs for interacting with each CRM product are unique. That’s okay, because the CRM system is a strategic asset; you can create dependencies on strategic assets because they will be with you for a long time. But the different APIs mean that it’s a big hassle to convert all the client applications in your shop to work with the new CRM product.
In addition, because your legacy CIS or CIF has business rules embedded in it and isn’t a plain vanilla data store, you can’t just convert the data, load it into the new CRM product’s database and be done with it. Conversion will be a gradual process.
This is just the sort of situation where SOA comes in handy. Your technical staff can create service interfaces that provide client applications with the same logical view of customer information no matter where the data really lives in the back-end environment. Little by little, you can replace the functionality supported by the legacy CIS/CIF with features of the new CRM package, and client applications will be none the wiser. As the tactical and therefore temporary client applications are replaced, the old point-to-point interfaces to the legacy CIS/CIF (and the early point-to-point interfaces to CRM that were written when the package was still new and no services existed for it yet) will gradually go away by attrition.
This approach also gives you a layer of abstraction for accessing the CRM package, so if you decide to switch to a different CRM product it will actually be feasible to do so. Vendor independence has direct business value.
Getting back to shadow IT
We’ve drifted a bit from the opening topic of “shadow IT,” but I think the whole SOA thing is pretty relevant because it opens the door for us to restructure the company in a way that supports the needs of business units much more effectively than the old IT-as-service-provider model.
What would this sort of organizational structure look like? What would become of the IT department? Let’s walk an evolutionary path from the early days of corporate computing to a world in which companies are structured this way, and see how it can happen.
Before the era when bureaucracy grew up in the world of corporate IT, things looked more-or-less like this:
In this model, lines of business (LOB) interacted with whomever they could find in the IT area when they needed computer-related assistance. You might wonder why anyone would structure things this way on purpose. The reason is clear when you realize that the purpose wasn’t to provide excellent customer service, it was merely to separate those noisy machines and the strange people who loved them from the business folk who did the real work.
As you can see from the illustration, the machines wouldn’t have fit inside a normal office environment, and the people around them were, after all, pretty strange. At the very least, they are oddly dressed. What are those things around their necks? Nooses? Best to keep them in a windowless room in the back of the building.
Anyway, returning to organizational structure, all this ad hoc-ness wasn’t terribly helpful. So, IT management decided to get a little more control over how the work was assigned, scheduled, tracked, and delivered. Things started to look more like this:
This was a lot better. Yet, people in “the business” were still not getting the kind of service they needed. They never knew which technical staff members might be assigned to their projects. Would they be knowledgeable about the business domain or the particular departmental application they would be modifying? For that matter, would they be knowledgeable about computer programming? Business people wanted a bit more consistency. They wanted the technical staff to become familiar with the business domain so that every project didn’t devolve into a training exercise for a whole new team.
What we came up with, as an industry, was so-called alignment. With this model, working groups within the IT department were dedicated to the support of particular business units. The model looked something like this:
Looks pretty on paper, doesn’t it? Well, this isn’t exactly paper, but you know what I mean. I’ve seen diagrams like this that were absolutely stunning. They used curvy shapes and added gradients, 3D effects, shading and other artistic effects. Just lovely, really.
There’s just one tiny problem: Alignment still doesn’t get people what they need. If you study the diagram you’ll see why. Look carefully, as it’s a subtle point. Do you see it? Here’s the answer: There are two organizations. (Okay, you caught me, there. It isn’t subtle at all. It’s whomp-you-upside-the-head obvious.) The IT department is its own little world. It sees “the business side of the house” as an external customer, and the business people see the IT department as an external service provider.
One way you can tell when a company is structured this way is by checking so see whether the IT department does its own strategic planning separately from the rest of the enterprise. If the department has a function called Portfolio Management, then it’s very likely they are not well integrated with the business. From the moment you create two independent strategic plans, they start to diverge. Once that happens, you either have to admit you made a structural error or you have to create a whole new profession dedicated to managing the divergence, like Portfolio Management. Naturally, most people are quick to admit when they’ve made an error. (Okay, you caught me again. Sharp as an overcaffeinated soprano, you are.)
If you think about it, it’s a little bit crazy that people would do things this way. After all, the legal department doesn’t have its own strategic plan. The accounting department doesn’t have its own strategic plan. The marketing department doesn’t have its own strategic plan. Their work is driven from the overarching enterprise strategic plan. Why shouldn’t the work of the IT department be driven from the same plan?
Some argue that business planners simply aren’t aware of technical issues. It would never occur to a business person to ask whether the hardware platform that runs the company’s payroll application is about to go off support.
My reply is that this is the very reason the CIO is supposed to be sitting at the table with the other C-level executives. When the executive team determines that the company requires a certain business capability, and support for that capability will require work on the part of the IT department, the CIO is supposed to say so. The reality is that many CIOs are treated as nothing more than order-takers. They write down the work that is requested of them and then they go away and Manage their Portfolio.
This may also be one reason why your company doesn’t have a SOA environment in place yet, after all these years. The recognition that such an environment is necessary comes from the grassroots level; typically from the enterprise architecture group within the IT department. This group then has to up-sell the idea through the management hierarchy. There’s no natural connection with anything in the strategic plan.
What should happen is that some element of the strategic plan triggers the realization that a SOA environment will be necessary in order to provide the desired business capability. Same for any other “technical requirement.” Everything can flow down from the enterprise strategic plan, provided the right people have a chance to review the plan to look for such requirements.
To fix that, we have to unify strategic planning.
Okay, so we’ve unified strategic planning. Whew. Much better. But how long is this going to last? We still have two separate organizations, after all. And there really are things that ought to be centralized. We don’t want to go to the opposite extreme and set up a miniature copy of the IT department in each line of business. What’s a body to do? More to the point, what’s an organization to do? Well, how about this:
Now each line of business has all the people it needs to support its own operations, including application software programmers and system administrators for their departmental servers. Meanwhile, the central IT department takes care of enterprise-level assets, security, and governance. The connecting tissue between the two is our good friend, SOA.
If this seems both sensible and feasible, why don’t we see it more often in real companies? The reason was summarized by Eliyahu Goldratt in the oft-repeated quote, “Tell me how you will measure me and I will tell you how I will behave.”
Golf course performance assessment
How are IT managers measured? What must they be able to show prospective employers in order to qualify for the next job on their career ladders?
It isn’t a mystery if we bear in mind a couple of basic realities:
- IT departments are cost centers; and
- Cost center managers are respected in direct proportion to (a) the budget they manage, and (b) the number of people under them on the organization chart.
So, these four IT managers are playing golf. Jack says, “I’m responsible for an annual budget of $50 million, and I have 400 people under me.” He tees off and hits a 200 yard drive.
Bill says, “I’m responsible for an annual budget of $100 million, and I have 1,000 people under me.” He tees off and hits a 240 yard drive.
Mary says, “I’m responsible for an annual budget of $300 million, and I have 3,200 people under me.” She tees off and hits a 280 yard drive.
Frank says, “Stand back. This one’s going 320.”
Their old friend Pat doesn’t play with them any more. They’ve sent her over to the miniature golf place down the road. She made the career mistake of concentrating on how to deliver the most business value to her company, instead of concentrating on how to maximize her budget allocation and head count. When she saw “shadow IT” happening in her company, she interpreted it as a signal that business stakeholders were not getting what they needed from her department, instead of seeing it as a threat to her. Now she delivers the same value as Frank, but does it with a budget of $20 million and a staff of 125. She doesn’t look like a Big Shot to prospective new employers. She is very good at sinking putts, though. Isn’t that what the game is supposed to be about?
Excellent entertaining read. My thoughts exactly.