It would seem that the devops discussion is mostly driven by development’s incentives, and appropriately so, given developers’ focus on building functionality for the business user. So it’s no surprise that development is the originator of the whole devops lifecycle, but is there dangers lurking in a one-sided focus on devops issues?
A significant portion of devops articles seem to come from writers of the development persuasion who are motivated by the legitimate frustrations of the application deployment process. The movement to agile development has been a key contributor in the increase of handicaps encountered as a result of more frequent transitions from development to operations IT groups. Online and verbal discussions identify the primary challenge as getting IT operations to be more creative and flexible in their approach to changes coming from the application development discipline1.
Given its critical involvement in deploying new and enhanced applications, why hasn’t IT operations been more active within the devops movement? Given the opportunity for IT operations to be more effectively heard by its primary IT companion organization, why hasn’t IT ops been more responsive to devops strategic initiatives? Given that changes to the operational model are almost guaranteed, why isn’t IT ops more proactive in anticipating such changes? Given the need for more effective alignment with the business user, whose automation needs are frequently accommodated by in-house development, why hasn’t IT ops seized the chance to leverage this obvious path to improving IT’s core value proposition to end-user communities? I’ve heard many theories, but few have resonated.
Devops is a maturing discipline and is obviously offering an increasingly visible value proposition for the enterprise. This growing devops “maturity” is driving more effective partnerships and better integration opportunities. That’s the good news. The bad news is that the devops partnerships and integrations are not coming fast enough to appease the significant IT market demand. One proof point is the rapid escalation in demand for SaaS application deployment, which is a symptom of the ongoing struggle between development and operations groups in satisfying end users’ business demands. Competitive options now exist (and are expanding) for businesses to choose alternatives to IT-delivered and -supported applications, which further fuels the customer’s questioning of IT’s contribution to achieving corporate business objectives.
“Services” Mindset of IT Operations Essential to Devops
Fundamental to understanding the mindset of IT operations is its emphasis on delivering and maintaining “services” that are of value to the business community.
- Solutions to business problems and support for business models, strategies, and operations are increasingly in the form of services. The popularity of shared services and outsourcing has contributed to the increase in the number of organizations who are service providers, including internal organizational units. This in turn has strengthened the practice of service management and at the same time imposed greater challenges upon it.2
An essential requirement for devops success is to think of, prepare, implement, and support the new application as a mandatory component of a service that is being offered to the business customer. That service is positioned as having an intrinsic value that is broader than the application by itself; as a result, the business executive can easily identify it as critical to achieving business goals. IT operations brings a rich experience of managing those services to devops initiatives. Ops needs to be more aggressive about sharing that unique perspective at the same time that IT development needs to be more attentive to those IT service management (ITSM) best practices that are documented, promoted, and used on behalf of devops by its core practitioners.
IT operations has struggled for decades to deliver more and more technology services with fewer and fewer resources — and is actually doing a fair job.3 Fundamental to that achievement has been a focus on ITSM best practices specifically designed for improving the operational alignment between IT and its business customers. These guidelines and processes have been captured and documented by leading ITSM professionals for the Information Technology Infrastructure Library (ITIL), funded by the UK’s Office of Government Commerce (OGC).
As a starting point for understanding the mindset, roles, and values of IT operations within the devops discussion, ITIL V3’s focus on delivering business value through more effective IT services is invaluable. ITIL positions these best practices as “based on expert advice and input from ITIL users [and] … both current and practical, combining the latest thinking with sound, common sense guidance.”4 The success of that “common sense” approach has contributed significantly to the rapid acceptance and global implementation of ITIL V2 and V3 by ITSM organizations over the last 10 years.
Bridging The Devops Gap
So what might the operations perspective on devops initiatives be? I would offer that ITIL’s V3 Application Management (AM) function5 is probably one of the more useful resources available for describing that operational viewpoint. It is written as a set of best practices or guidelines for supporting …
- the organization’s business processes by helping to identify functional and manageability requirements for application software, and then to assist in the design and deployment of those applications and the ongoing support and improvement of those applications.6
As such, ITIL AM is designed to help bridge the devops gap and articulate some of the common language or terminology now lacking between the development and operations organizations.
As a process improvement approach, the Capability Maturity Model Integration (CMMI) has become a highly successful, best practice model for software engineering. CMMI is a set of process guidelines driven primarily by development’s need to deliver applications of value to the business user and secondarily by how the end customer uses the application as a service to achieve business purposes. This is a subtle yet critical distinction. The CMMI model does not adequately address the service mentality and service priorities of the IT operations organization, which has fully embraced ITIL for that role instead.
In the August 2011 issue of Cutter IT Journal, Bill Phifer of HP Enterprise Services created a compelling case for embracing a host of ITSM processes, with the ITIL Service Lifecycle becoming the needed counterbalance to the development perspective that is incorporated and respected within CMMI.7 The strength of ITIL is its unrelenting emphasis on “serving” the needs of the business community and adapting any IT deliverable to the “whole” of the system — as perceived by the business customer, not IT operations. For devops, this requires a correlation of management processes and tools that can address the demands coming from end users, business functions, and technologies as well as applications. ITIL in its intended form8 does not create a focus on IT management processes for the sake of IT personnel. An ITIL-oriented project fails miserably when such an IT-only focus happens.
ITIL V3 Application Management Function
Coming at devops from a “Service Operation” perspective, ITIL V3 defines the ITIL AM function as a shared responsibility of IT operations and IT development in satisfying the service requirements of the end user. ITIL AM can therefore become a complementary yet essential best practice approach for devops due to its operational focus.
Figure 1 depicts the lifecycle of shared processes required of both Application Management and Application Development in order to effectively deliver the service that the business requires.9 The devops principle that becomes obvious in this diagram is that some phases in the ITIL AM lifecycle are clearly driven by the development group while others are best driven by the operations group. This raises the question, “Who is actually driving each of these phases?” Too often development is driving operations-oriented phases because operations is not yet on board. Successful devops initiatives find a way to ensure that each phase is led by whatever individual, team, or group possesses the necessary experience, skills, and vision to deliver that lifecycle stage as a business service component. This person or group may have a background in development or operations … or, optimally, experience in both.
As indicated previously, responsibility for the ITIL V3 AM function is shared between development and operations (devops). Given the secondary, yet complementary, role of IT operations in the development and deployment of applications, a successful devops team is going to see to it that IT ops makes its contribution in each of the AM lifecycle phases. This contribution is frequently the underappreciated half of the devops equation because, as we’ve seen, the drive for application deployment originates from development. The phases are as follows:
> Requirements phase. Clarity regarding manageability requirements is as critical to the service perspective of the end user as the functionality, architectural, and usability requirements, yet too often they are delayed until the second version and handled through an enhancement process. This has been such a consistent pattern in my ITSM career that I sound like a broken record in my emphasis on this up-front (not subsequent) inclusion. Delay in satisfying manageability requirements jeopardizes the acceptance and early use of the application by the business user, who can’t get the needed training, support, security, monitoring, and self-service.
Interface requirements that ensure the new application satisfies dependencies for existing IT management and security tools are paramount. Who can define those dependencies except those who manage and use these ITSM applications? How often do teams postpone the definition of service-level requirements until a later phase of the AM lifecycle, only to find out too late that they cannot adequately specify the application’s performance metrics (other than “It should initially work”), the quality of the application’s output (reporting, dashboards, business intelligence, etc.), or any other qualitative or quantitative aspect that the business customer or user needs to measure?
> Design phase. In addition to leveraging the requirements into the design of the application and environment, where is the design accommodated and reviewed for changes to the operational model that the application has to adhere to? For applications that are purchased rather than developed, the responsible business unit is typically able to offer feedback regarding functionality, but there is still a need for IT operations to provide feedback to the software vendor about the manageability and performance of the software. How early in the AM lifecycle is IT operations empowered to have this discussion with the software vendor or internal development group? Finally, will operations have an opportunity to provide input to the design of intended customization capabilities for manageability and reporting needs?
> Build phase. Because this is the phase where application components are coded, integrated, and tested, and the application and operational model are made ready for deployment, it is also the phase in which IT operations makes the least contribution. Yet it is during this phase that there should be a focus on whether the management specifications are satisfied. Testing is integral to this phase, so testing the management of the application within the operational model should be accommodated to the satisfaction and acceptance of IT operations.
> Deploy phase. With the operational model embedded in the existing IT infrastructure, any new application must be deployed within that IT environment and on top of that operational model. Because operations is responsible for this production environment, this is too often the first time non-devops “deployment” initiatives will reach out to IT ops in order to ensure the application actually works as promised after it has been downloaded and installed. IT operations and development are supposedly working together for a predefined period of time (Early Life Support) for testing, validation, and monitoring of the service that contains the new or revised application. However, if functions contained within phases 1-3 (Requirements, Design, and Build) of ITIL AM have been handled outside of IT ops, then frustration and accusations become the norm and devops simply cannot realize its promise.
> Operate phase. This phase is obviously the realm of operations; it’s where the name of this IT organization originates. According to ITIL V3, “The IT services organization ‘operates’ the application as part of delivering a service required by the business. The performance of the application in relation to the overall service is measured continually against the Service Levels and key business drivers.”10 Applications are only one of the many elements that are necessary to initiate, provide, and maintain a business service as defined and understood by the end users, and IT operations is responsible for operating all these service assets with their constant changes, unpredictable events, illogical problems, and difficult demands (requests) by the end users. Is it any wonder that ops staff overreact when confronted with an application for which they are unprepared?
> Optimize phase. Improving service levels and lowering the costs of an application are two IT operations strategies that should always be optimized. Neither will ever be “good enough.” As devops goes through this ITIL AM lifecycle for each new application, it must also go through the lifecycle for each application enhancement or modification. Operations must maintain previous versions of the application while simultaneously engaging in the devops lifecycle for the next version. Agile development ensures that an iterative devops approach is continually ongoing. In addition to optimizing the application, IT ops should also be constantly monitoring the ITIL AM function for continuous improvement possibilities.
ITSM-Oriented Activities for Devops
There are a host of activities that are part of IT operations’ unique contribution to devops — too many to attempt a complete list. Following is a small sampling of ops activities included within ITIL AM:
- Instrumentation of applications for generation of meaningful events
- Scenarios to predict impact of potential changes to utilization, functionality, manageability
- Application sizing, volume metrics, load testing for availability and capacity planning
- Recruiting and training programs to ensure skilled resources for AM
- Operational models to ensure optimal use of system resources
- Error analysis, bug tracking, patch management, antivirus for inhouse applications
- Vendor management to manage contracts and deliverables of purchased applications
- Project involvement for operating system upgrades, server consolidation, physical moves
- Definition, review, and maintenance of application documentation (user/admin manuals, standard operating procedures)
- Process metrics for response time, incident resolution, changes backed out, security issues
ITIL V3 Service Management Lifecycle
One of the important contributions of ITIL V3 is the way the AM function is accommodated within the larger Service Management Lifecycle (see Figure 2).11
Applications are services in the eyes of the business user, yet their design, development, deployment, and management are distinctly different from almost all other IT services. However, just because application-driven services are markedly different from other IT services doesn’t mean application developers can ignore the time-tested principles of Service Management. For example, ITIL V3 documents how Application Management is integrated with each of the five phases of the Service Management Lifecycle:12
1. Service Strategy — defines criteria for developing applications inhouse, outsourcing development, purchasing or customizing applications; defines how applications fit within the service portfolio of all IT services; includes ROI analysis for both applications and services they support
2. Service Design — establishes requirements for manageability in addition to functionality; works to ensure such manageability objectives are satisfied during the Requirements and Build phases of ITIL AM and therefore fit within the design of the targeted service offering
3. Service Transition — includes testing, validating, and deploying applications operationally within the targeted service offering as intended for the business end user; enables operational change processes to be examined, accommodated, and enforced
4. Service Operation — delivers support, maintenance, training, and metrics required by the business end user in the achievement of the service’s business goal; institutes service desk processes for event management, incident management, problem resolution, and request fulfillment for application support
5. Continual Service Improvement — measures the quality and relevance of the applications within the operation of the business service and provides recommendations on potential improvement; ensures feedback from the user to operations and development
Each of these ITSM activities should be addressed within the scope of devops. Documentation, training, and expertise on how to fulfill the AM function are available in the ITIL V3 resources.
IT’s Cultural Focus on Changes
Probably the number one guiding principle of IT operations is that unplanned changes are the root of most failures. That mindset drives every process, tool, and activity within the ITIL Service Management Lifecycle. Consequently, it is in the DNA of operations personnel to resist anything that might introduce unpredictable changes within the IT infrastructure. IT ops is rewarded (not penalized) for consistency and for preventing the unexpected or unauthorized from happening. Conversely, development staff are rewarded for creativity in solving business problems and for project completions in a timely manner. Metrics that foster these desirable behaviors are going to ensure the two groups are in natural conflict — hence the need for and promise of devops in resolving this disconnect.
Knowing that the IT operations culture has an aversion to change, the devops group would obviously address that mindset early and continuously in the ITIL AM lifecycle. ITIL change management identifies seven questions13 that IT ops staff must answer with regard to all changes proposed to the IT infrastructure and operational model:
1. Who raised the change?
2. What is the reason for the change?
3. What is the return required from the change?
4. What are the risks involved in the change?
5. What resources are required to deliver the change?
6. Who is responsible for the building, testing, and implementing the change?
7. What is the relationship between this change and other changes?
Without agreement on the answers to these seven questions, any assessment of impact, cost, or risk of the proposed changes cannot be completed. Furthermore, there will be no understanding of how responsibility for the changes to the desired business service is to be shared between development and operations. More important, the business benefit that is motivating development of the application will not be completely achieved; the application could even have a negative effect on the existing services provided by all of IT to the business end users. The core culture of IT operations demands that devops address the impact of changes to the existing and intended services offered to the business community.
Long-term success in devops is completely dependent upon the development of common metrics that are recognized and accommodated by both IT operations and development — not one side only. A change in perspective is needed where IT operations comes to the devops table and is motivated by something other than making development’s work easier. ITIL V3 Application Management assists devops efforts in detailing how the application itself fits into the larger delivery of service management for the business service purchased by the IT customer, providing guidelines and process definition for that operational integration — the thing that is most often missing from early devops initiatives.
Operations personnel should approach the devops opportunity with their own best practices defined and implemented for Application Management, recognizing that development’s objectives will be different from their own, but equally critical in fulfilling the business needs of the end user. Development should approach the devops opportunity with the intention of better understanding the unique needs and demands that are driving the operations function.
1 Read, Chris. “DevOps: State of the Nation.” Agile Web Development & Operations, 30 November 2010 (www.agileweboperations.com/devops-state-of-the-nation-by-chris-read).
2 Introduction to the ITIL Service Lifecycle. ITIL V3. Office of Government Commerce (OGC), 2010.
3 McAfee, Andrew, and Erik Brynjolfsson. “What Makes a Company Good at IT?” The Wall Street Journal, 25 April 2011.
5 ITIL Service Operation. ITIL V3. OGC, 2007. (ITIL published a “Summary of Updates” in August 2011; see www.itil-officialsite.com.)
6 ITIL Service Operation. See 5.
7 Phifer, Bill. “Next-Generation Process Integration: CMMI and ITIL Do Devops.” Cutter IT Journal, Vol. 24, No. 8, 2011.
8 Fry, Malcolm. ITIL Lite: A Road Map to Full or Partial ITIL Implementation. The Stationery Office, 2010.
9 ITIL Service Operation. See 5.
10 ITIL Service Operation. See 5.
11 Introduction to the ITIL Service Lifecycle. See 2.
12 ITIL Service Operation. See 5.
13 Introduction to the ITIL Service Lifecycle. See 2.