A CIO’s DevOps approach to resolving the agility-stability paradox

August 1, 2013

by



Tags: 

Share on FacebookShare on Google+Tweet about this on TwitterShare on LinkedIn

Real process change requires a break with the past.

For decades, CIOs have applied methods and adopted tools that lead to stability. With good reason: When an IT deployment goes badly, businesses lose customers and risk long-term reputational damage. CIOs can lose their jobs. So business units had to live with a rate of business process innovation that was tethered to the plodding pace of IT’s waterfall software development life cycle (SDLC). These IT systems methodologies are rigorous, but slow and linear.

“The requirements change so quickly that if IT’s pace is in weeks and months, you end up not delivering anything, because the problem changed by the time you deployed your solution,” says Ken Venner, CIO of aerospace company SpaceX.

Speed versus stability is a paradox for IT. Pushing for both at the same time seems contradictory, but the IT organization must strive to achieve this goal if the enterprise is ultimately to become more antifragile—that is, less vulnerable to catastrophe. (See the article, “The evolution from lean and agile to antifragile,” on page 06 for a further explanation of antifragility and its business relevance.)

Is it possible for the CIO to accommodate a fast-changing business environment while ensuring stability in IT deliverables? It is, but it requires breaking with the safe yet rigid workflow principles of the past. Some companies will be more comfortable with this break than others.

“We have to develop software in other than a Taylorian mindset, where we handed off detailed specifications to a software team to deploy,” says Jez Humble, a principal at ThoughtWorks Studios, a software development consulting firm.

IT can transform itself by adopting agility without giving up stability, but CIOs must change the IT mindset about how to achieve stability. As explored in this article, the key to this change is to adopt a DevOps philosophy that emphasizes speed and efficiency without sacrificing adequate stability. Specifically, CIOs can achieve IT agility and stability by making change a frequently recurring process, which they can accomplish through the adoption of appropriate organic, evolutionary practices used by antifragile, web-scale companies.


Speed versus stability is a paradox for IT. Pushing for both at the same time seems contradictory, but the IT organization must strive to achieve this goal if the enterprise is ultimately to become more antifragile—that is, less vulnerable to catastrophe.


The IT agility opportunity through DevOps

Historically, CIOs have trained their organizations and their users that volatility is a threat to stability. IT has responded by adopting Information Technology Infrastructure Library (ITIL) and IT service management (ITSM) standards and practices originally formulated by the British government for technology contractors, similar to the US Department of Defense’s creation of the waterfall methodology. These standard practices made sense at the time, when the world moved at a slower pace. Quality outcomes resulted only from establishing rigor and cross-validation beginning with specifications, followed by disciplined stages of design, development, testing, and release. The big downside: lead times were extended to accommodate this discipline, and that pace does not match today’s business dynamics.

CIOs could strive to balance the emphasis on stability inherent in ITSM and ITIL with a DevOps philosophy that emphasizes speed and efficiency. DevOps enables a closer working arrangement between developers and operations personnel; its best practitioners automate where possible and in the process they make developers responsible for ensuring their code functions well in an operational environment. (See Figure 1 for a summary of the main characteristics of the DevOps philosophy.)



Tools that further DevOps and continuous delivery goals help developers gain visibility into the operations environment, streamline their workflow, and automate a number of build, release, and deployment steps. (See the article, “Making DevOps and continuous delivery a reality,” on page 26 for more on related tools.) That visibility is consistent with better controls. (See Figure 2.)



There is no insurmountable inconsistency between ITIL/ITSM and agile methodologies/DevOps. (See the figure on page 13.) Agile methods add flexibility to accommodate variability, improve quality, and speed up the cycle. By using the tools, processes, and techniques described in the business and technical articles of this issue of the Technology Forecast, enterprise IT can change into an organization that offers strategic assets to support innovation, bring new IT products to market more quickly, and respond to dynamic business opportunities—all characteristics of antifragile enterprises.

“For enterprise IT, the whole environment has changed,” Humble says. “Not just in the tool chain, but also in customer expectations of a high-touch experience.”

Agile development, continuous delivery, and DevOps all require organizations to break down silos within the business, including the systems analysts, the software developers, the testers, and the release and operations teams. New features become available as soon as they are ready, allowing instant interaction and iteration, something waterfall processes allow only at the end of the release cycle.

ITIL and ITSM help solve the stability problem. Agility, continuous delivery, and DevOps help solve the IT speed problem. Achieving stable IT operations is a long-standing goal for both, with some major differences:

  • A fundamentally different theory drives stability: As originally envisioned, ITIL and ITSM controlled the process of change and ensured stability by reducing the frequency of change and triggering an IT process if change was introduced. DevOps sees change and evolution in a different light, as a way to boost the enterprise immune system—a key to antifragility. A DevOps and continuous delivery philosophy also views gestation periods leading to big bang deployments as extremely high risk. The longer the gestation, the more likely the business model won’t work, even if the code does. Business requirements change too quickly for a multiyear development plan to succeed. DevOps and continuous delivery advocate smaller, viable deliverables that attempt to address smaller parts of the requirement piece by piece. The operating principle behind DevOps is to make the system amenable to many changes and less likely to have a massive failure.
  • The rhythm shifts from an IT view of how IT changes can be introduced to a business view of changes necessary to compete: This view has been a fundamental gap in alignment between IT and business. Business units become frustrated that their internal IT team is slow to respond, expensive, and not error free. They don’t want to run their own IT, but they conclude they have no choice. IT is frustrated that business units go off on their own, possibly introducing a heterogeneous mix of components and apps that won’t work over time. Startup companies have been the most demonstrable examples of DevOps, but even older businesses can adopt these capabilities and may be in danger if they don’t. (See the sidebar “How two companies move toward the new model.”)
  • The rhythm of change is possible only with high levels of automation: IT must place a tremendous emphasis on design for automation—especially for testing, but also for configuration management, release management, infrastructure provisioning, and other delivery steps. Technical staff should devote quality time to devising and deploying the automation capabilities. Automation will make an important difference in continuous delivery and DevOps proficiency and productivity.

The waterfall approach is still de facto in several industries that have extreme business sensitivity to privacy, data security, independence, or process transparency reinforced by regulations. Financial services must follow strict regulatory compliance rules, reinforced in their public financial accounting (for example, FINRA). Healthcare, nuclear utilities, defense, and other industries also have such constraints, and not every CIO has the opportunity to help transform an entire industry from the outset. However, CIOs in any industry could apply continuous delivery at least somewhere in IT.

There might be an opportunity for agility to gain a foothold in a subsegment of the business, such as a new division, a spinoff, some internal IT (such as HR), or a new acquisition. CIOs can then bridge from that subsegment to more segments of the enterprise after proving the concept and knowing its limitations.

PwC advisory director Keith Katz describes an online equities broker-dealer that started its agility journey by establishing the pair-programming equivalent of co-creation of specifications and code. (See the sidebar “How two companies move toward the new model.”) It doesn’t require an entire industry or an entire company transformation to set the stage.

CIOs must put aside the notion that perfect software specifications are even possible with more laborious methods. Perfect specifications may be an illusion, because IT assumptions of what is needed may not match what the users want. With the agile development approach that DevOps and continuous delivery expand, IT does not design for one point in time for all time. Illusory precision from the waterfall method often didn’t work, either. There is no single deliverable, date, and budget. Continuous delivery is, by nature, iterative. Conversations with the customer will be ongoing, and deliverables will be multiple by design.


Deploying code daily is a jarring contradiction to everything seasoned IT professionals believe about stability. Fundamental changes are needed in people, processes, and technology.


The CIO action plan for agility

Deploying code daily is a jarring contradiction to everything seasoned IT professionals believe about stability. Fundamental changes are needed in people, processes, and technology. But those fundamental changes do realize benefits, as underscored in a recent Rebel Labs survey of 600 IT professionals, which contrasted the average workweek of traditional IT operations teams with the DevOps equivalent. (See Figure 3.)



The survey research confirms that more time spent on automation setup and general testing pays off in the reduced need for support, infrastructure management, and communications time. PwC recommends that CIOs implement the fundamental changes necessary for a DevOps environment.

Step 1: Address the IT skills gap

Competence with new tools and methods must be resolved early. “Technical competence is a major constraint,” says Charles Oppenheimer, founder of Prizzm. Agility is a new area for which most organizations have renegade talent but not organized competence. Some of the talent may be within enterprise IT or in shadow IT groups. Enthusiasts who already have the passion and some experience can form the basis of the team. Organizations also may need to contract some talent from outside. A banking industry CIO interviewed for this article is building on the substantial open source expertise within the bank, while bringing in outside mobility experience. “We may need someone to jump-start us,” he says.

Start by setting up a group in enterprise IT entirely focused on agility. Assign a leader who embraces agility and who grasps the concepts, opportunities, tools, and methods. Ancestry.com, for example, brought an engineering mindset about productivity to IT development, IT operations, and the business. “My group is called engineering productivity, and that name came about because the focus of my group is to improve the productivity and the ability for the business to deliver value to the customer more rapidly,” says John Esser, director of engineering productivity and agile development at Ancestry.com.

Some organizations, such as real estate company iProperty, have cooperated with local universities to recruit talent and train existing talent. Other organizations have found success using Facebook or other social media to recruit. SpaceX’s Venner has brought in recently college-trained IT professionals. “We find talent from a variety of different backgrounds and industries. Some of the talent I’m finding tends to be fresh out of college. They’ve generally been raised in this more agile development environment; they have a stronger orientation to open source tools and not as much focus on the waterfall-oriented development strategy.”

Enterprises may need to restructure their pay and corporate culture to attract and retain the necessary technical talent. Work with HR to assess and address this reality. In addition, resources should be assigned—not “planned” in the traditional sense of SDLC.

Step 2: Establish an agile and DevOps blueprint

Agility can’t be entirely without direction, so CIO leadership is needed to provide clarity on design goals, priorities, and resource assignments. The CIO needs a blueprint to ensure that the overall direction is consistent with the enterprise needs and resources.

Engage the business unit CXOs in a survey to identify the top five business opportunities for agility. The best candidates are scenarios where the dialogue with customers frequently changes and where the IT organization will need to iterate rapidly for new product ideas, markets, and customers.

Some initiatives already may be underway, set up independently from enterprise IT, which can be coalesced into the official road map. The CIO should not view this scenario as an opportunity to commandeer those initiatives, but instead should embrace them as pilot projects that are included in the opportunity road map and that have a goal of shared organizational learning.

Focus on the two-way conversations with the customer, or the potential opportunities for having these conversations, and on conversations that have a dynamic nature that warrants agility. Include IT suppliers, if they can add value, and by all means engage with businesses partners, especially suppliers, distributors, retailers, or staff in marketing, sales, and advertising.

Step 3: Invest in technical tools, software management approaches, and infrastructure

The CIO must determine what technology investments are required to get started. Early priorities include the following:

  • Choose a software management, version control, and change management and programming approach (such as pair programming).
  • Choose a social collaboration tool.
  • Automate everything.
  • Establish the rhythm of continuous delivery and DevOps.

An early step for the agile IT team is to determine how to manage each experiment in the road map. Change management helps ensure that the user community is informed of the change and the rationale. Release management ensures that all testing is completed satisfactorily and that new code will not conflict with something else in code, security systems, or infrastructure. Change management and release management will help ensure that design inconsistencies are caught before the code is promoted to production.

Despite the flexibility required for agility and for the mindset associated with experimentation, agility does not translate into chaos if CIOs provide proper oversight. Most importantly, agility does not mean the abolition of change management and release management disciplines. Agility-experienced IT groups advise that rigorous change management and release management disciplines should be in place. When press-worthy major outages occur in the DevOps world, they are traced to gaps in change or release management, and are not attributed to agile IT.

Nationwide Insurance began its agility journey by establishing change management for systems and for the organization, so roles and processes would be clear as solutions were deployed. Nationwide’s change management was a companion to the stepwise evolution of DevOps. Four years ago, Nationwide adopted a monthly release management schedule. “Before that, each of 26 business units had their own release schedule, which was chaotic for users,” says Vijay Gopal, vice president and enterprise chief architect at Nationwide.

Releases weren’t routine; each was unique and involved its own dependencies. IT was confusing users, and the organization needed to invest significantly in training each time. Moving to a monthly release schedule was a first step in Nationwide’s evolution toward agility, but an important and symbolic one. It demonstrated the consensus of the 26 business units that they had to trade some autonomy for a better, long-term result.

The infrastructure must be agile ready at the outset. Esser of Ancestry.com says CIOs should start on the technical dimensions by “asking themselves if the IT infrastructure can be leveraged, if it is an asset, or if the business believes it’s a liability.” Given the readily available cloud infrastructure for hire, answering those questions may be one of the easier tasks to accomplish quickly. “We’ve probably seen the largest gains on the infrastructure side,” Esser says. “Previously, even replicating a given server’s configuration was very challenging, if not impossible, and obviously that added a lot of time into the equation.”

Much of the automation required for agility assumes a private, public, or hybrid cloud architecture, and enterprise IT must provision it quickly. One way is for IT to act as a service broker. Phil Berman, a director in PwC’s infrastructure and operations practice, suggests that instead of setting up your own cloud infrastructure, it may be more practical and time efficient to go to a cloud provider (such as Amazon or Google) and acquire cloud services on behalf of the business, rather than the business negotiating for services on its own. “IT still keeps a level of control and governance around all IT, but basically resells Amazon or Google services back to the internal users.”

Berman describes a situation in which an enterprise IT organization entered into a “white label arrangement” with Amazon and Google, and through them provides cloud-computing services to the internal organization. Enterprise IT chooses the preferred providers, consolidates the purchase for lower costs, maintains the service catalog and governance over security, and charges the users for the cloud services and their value add.


Despite the flexibility required for agility and for the mindset associated with experimentation, agility does not translate into chaos if CIOs provide proper oversight.


Step 4: Automate

Currently, no tool automates everything. But automating various processes will improve agility. For example, reviewable scripts and logs can be created through automation. Then, if something goes wrong, the automation associated with that error can be fixed.

System setup and configuration can be automated (using Chef or Puppet, for example), but testing the setup is crucial. Automate code generation and integration, but test that, too. Add scripts that combine the code generation and integration. Test that, and so forth.

The fast-paced world of continuous delivery requires extensive pretesting, but techniques such as limited release also work well. In the limited release technique, code is rolled out to a sample of the production environment. If it does not meet expectations, the code is rolled back. If it meets expectations and proves stable, the code can be promoted to the enterprise universe.

Venner calls this method test-driven development. Some organizations might encourage different teams to experiment with different tools, especially at the outset. But it is also important to avoid the usual tendency to do extensive research before choosing just one tool and one approach.

Step 5: Remove barriers

The CIO’s job is to enable others to succeed. Unlike the CIO’s role in major SDLC projects, the CIO of a DevOps environment will not manage (some might say micromanage) the details of process work, design, development, or production. Instead, Venner says, “It’s finding, understanding, and removing blockers for the development team, which means having conversations around what obstacles they have—either in terms of the business and the business interaction, or the technology or some process that’s stopping them from getting the jobs done.”

Often, as many as 5, 10, 15, or 20 development teams will be working at the same time. The CIO can’t manage the details or the fast pace of everything, yet he or she will be responsible for the outcome.

The CIO alone can identify and resolve constraints such as infrastructure limits, preexisting contracts (outsourcing commitments or software licenses), business process traditions, organizational culture and behaviors, or lack of training. The transformation from a more traditional IT approach to a more agile-centric approach, the core being continuous delivery, will involve cultural and organizational changes.

The CIO’s horizon must encompass suppliers, channels, and customers. IT teams need the freedom to achieve agility and variability for the customer-facing systems (systems of engagement) while not jeopardizing the core legacy systems. Sometimes that balance isn’t clear, so the CIO must intervene with flexible judgment—agility in an executive sense. The technical tools and techniques may be new, but the CIO’s proven skills as a leader will help the organization succeed even in the face of many challenges.


The CIO alone can identify and resolve constraints. The transformation from a more traditional IT approach to a more agile-centric approach, the core being continuous delivery, will involve cultural and organizational changes.


Step 6: Integrate and modernize legacy systems

Most enterprise IT organizations have legacy software and data systems in place, such as enterprise resource planning (ERP), customer relationship management (CRM), and other systems of record. These systems run well after years of investment and tuning, and they are the core operations of the company. Eventually, CIOs will need to determine how to integrate the agile systems with the core and perhaps modernize the core itself.

Customer-facing systems, the systems of engagement, are an example. These systems may source data from the legacy core or create data that must synchronize with the core. In mobile banking systems, for example, taking a photo of a check with a smartphone to make a bank deposit requires an interface with the banking transaction and clearing systems.

Many legacy systems have significant residual sums on the organization’s balance sheets, and years of depreciation and life remain. These systems are a form of sunk cost, requiring a cost-benefit rationale for replacement. For some industries, the core systems may be subject to regulatory oversight, and for public companies, they are subject to extensive audit reviews.

Not all IT is amenable to agile approaches at the outset. A plan for modernizing the legacy systems will require a situation-specific analysis to disassemble the core and replace modules in a logical pattern. The tools and techniques for the legacy systems are the same as for new systems. To follow the principles of DevOps, a plan for legacy modernization must avoid long delivery cycles and big bang approaches.

Nationwide’s legacy modernization, for example, tackles the noncritical parts of the business first, proving concepts and adapting the user base. These systems are on a cloud, virtualized environment and have a fully secured interface back to the core. This approach allows “gearing” between the agile stack and the core through edge databases and edge applications. Nationwide’s design philosophy for replacement is to make the systems more intuitive, helping to ease the transition for the user base.

For those industries where the core is not readily amenable to agility, the CIO might consider spinning off the legacy into its own environment. Then wrap it with Java and application programming interfaces (APIs) so it can interface with the emerging new environment, and eventually phase it out.

The specifics of the blueprint—tools and methodologies—will vary depending on the new agile applications and the specifics of the core. The great news is that the building blocks are available. In fact, the IT organization may already have experience with many building blocks individually, but not in a holistic, agile way. The CIO has the opportunity to master these valuable capabilities to help transform the enterprise. The CIO should demonstrate agile IT leadership and then graduate to the next level.


Customer-facing IT services are the most strategic and are a logical starting point to help transform business processes. Customers expect a rapidly evolving relationship with your business that recognizes their needs and preferences.


Beyond IT: The CIO as agility leader for the business

To become more antifragile, a business will need agility, but it might lack experience in how to become agile. Sometimes the business is stuck because it applies its traditional approaches to change. Or, sometimes, business leaders think they are stuck because they operate in a nonagile IT environment. Either way, IT can prove itself capable in agile IT and serve as a role model for the enterprise.

Customer-facing IT services are the most strategic and are a logical starting point to help transform business processes. Customers expect a rapidly evolving relationship with your business that recognizes their needs and preferences.

IT has the potential to accomplish this transformation; now it needs to demonstrate its capability. “As we leverage more agile methods, tools, and processes in our [IT] group, we’re getting to the point where I feel the business has a good foundation so we can actually practice what I call business agility,” says Ancestry.com’s Esser, who learned that business process reengineering disciplines helped tremendously in applying agility concepts to the business.

The goal is to add strategic value to a dynamic marketplace. The method of choice builds on agile development and cloud computing with DevOps-centric approaches as well as ongoing evolution and delivery. Business agility will increase as IT agility does.

IT has the tools, methods, and infrastructure to adopt agility now. The CIO must be the agile leader for IT and for the business, because no one else has the toolkits or talents for this task. CIOs need to address the speed of business versus the speed of enterprise IT. CIOs also must lead the modernization of old thinking about stable IT systems. Certainly, no one else can harness the capabilities of the core IT legacy software and data—the systems of record.

Don’t force your internal business partners to go outside the organization or to watch as their competitive landscape is threatened. CIOs have the keys to agility and the unique leadership skills to demonstrate their value to the enterprise, both in IT and in the business overall.

 

Share on FacebookShare on Google+Tweet about this on TwitterShare on LinkedIn

Contacts

Chris Curran

Principal and Chief Technologist, PwC US Tel: +1 (214) 754 5055 Email

Vicki Huff Eckert

Global New Business & Innovation Leader Tel: +1 (650) 387 4956 Email

Pierre-Alain Sur

US Technology Industry Leader Tel: +1 (646) 471 6973 Email