Introduction to DevOps

Agile Training Series

Spring 2016
Course Description

Course Description

The simultaneous needs for IT to 1) deploy new features and 2) keep systems up and running
creates a core conflict that challenges development, operations to the respond to business
needs customers in a timely manner. DevOps represents practices, tools, and a culture that
seeks to resolve this core conflict by enabling operations and development engineers to
participate together in the entire service life cycle, from design through the development
process to production support. This class will explore these practices, tools, and culture using
Gene Kim's "Three Ways of DevOps" as guideposts.

Course Audience

 Newcomers to Agile and DevOps will find this class a welcoming environment to learn the
basics on the DevOps mindset, The Three Ways, automation pipelines, common DevOps
systems and tools, and continuous integration / continuous delivery (CI/CD)

 Operations Engineers will appreciate the class focus on using DevOps to effectively
manage the large quantity and frequency of changes demanded in modern IT operations
while keeping systems stable

 Agile Teams already developing software using agile methods will find this class to be a
logical extension toward achieving synchronicity with operations and business using DevOps
2
Learning Goals

Today

Experience the DevOps way of thinking

Form beliefs about how DevOps can work for you

Tomorrow

Identify actions for your project

Weeks/Months

See improved results Introductions

Create DevOps experiences for others Who are you?

What are you working on?
Years How do you plan to apply
DevOps?

Build a widespread DevOps Culture in our organization 3

Let’s Review Our Progress with Agile So Far…

USCIS Agile
Projects/Portfolios
kanban

What results have we seen working this way? 4

Let’s Review Our Progress with Agile So Far…

We applied the agile empirical mindset and agile methods and observed
these results:
 Early and continuous delivery of  Emergent design
valuable software  Technical excellence
 Rapid feedback  Empowered self-organizing teams
 Empirical decision-making  Personal safety
 Satisfied customers  Sustainable pace
 Business people and technical  High trust environments
people working together
 Measurement-based forecasting  Lean processes
 Harnessing change for competitive  Continuous improvement
advantage

The Agile Manifesto and the agile methods that followed focused on software
development – DevOps is a logical evolution of a maturing agile process
5
DevOps: Key Concepts

6
 The Basics

Leave class able to confidently answer these questions:

Who is Dev? Who is Ops?

What is DevOps?

“The beginning of wisdom

is the definition of terms”
- Socrates

7
Traditional Development

The Inventors
 Create new features
and functionality in
“dev” environment

 Occasionally deliver
new product to
operators, along with
instructions

 May incorporate
feedback from
operators in future
deliveries

 Rewarded for
delivering new
features
8
The inventors are responsible for changing the system
Traditional Operations

The Mechanics
 Receive new product from
developers to be installed and
operated

 Expected to keep production

systems up and running

 Track problems, deployment

failures, and system outages

 May provide feedback to the

inventors for future consideration

 Penalized for downtime

9
The mechanics are responsible for keeping the system in operation
Differing Views on Change

Hero

Obstacle

Change Orientation Stability Orientation

Alienate customers b/c system Alienate customers b/c system

constantly changes doesn’t change

Logical extremes
10
We Have A LOT of Changes

USCIS needs to update IT capabilities to support field users

AND
USCIS needs to keep IT capabilities operational for use by field users
AND
USCIS needs to keep IT capabilities compliant with security, regulatory, and
compatibility requirements
Can we Can you Can you
deploy latest deploy this deploy this
version? one, small one, small
Can we
Change?
Can we apply deploy new CanChange?
you
Prod is this security patch for the stage this
running slow, patch? Can you release? new
Can you environment?
can you cycle deploy this deploy this
the server? Can you
one, small one, small
Production Change? Can you upgrade the
Change? deploy this database
server is
down, fix it one, small version?
Can you Change?
now!! upgrade the
operating
system?

USCIS applied over 4,000 changes in 2014

Separation of Dev and Ops: A History

As computers became more complex, dev and ops

became necessarily specialized:

 Accelerating pace of technology

 Increased demand for turning around new features
 Huge amounts of data and number of calculations
 More and more specialized tools
 Increasingly abstract architectures and design
patterns Augusta Ada King,
Countess of Lovelace
(1815-1852)
And these were the problems in 1945!

12
Nobody can be an expert in everything – your enterprise can’t rely on Brent!
The Reunification of Dev + Ops

13
DevOps in a Nutshell

DevOps is the practice of

operations and development
engineers participating together in
the entire service lifecycle, from Hmm… what would
design through the development happen if we extend
process to production support the core drivers of
successful agile
development to
People
Collaborating
operations?

Automated
Systems
What if we built a
bunch of great tools to
Monitoring,
Feedback, and
help us?
Action

14
Breaking the Silos:
Communication, Collaboration, Integration
Communication

Collaboration Integration

Development Operations

How can dev help system stability? How can ops help accelerate feature
delivery?

We can build cross-functional teams around “knowledge overlaps” – people

with experience on both sides and “Ops Devs” 15
 Breaking the Silos: Dev and Ops

Ops can
anticipate
how new Development
functionality
will effect
production Dev can respond
to bugs and
deployment
failures quickly
 Ops trusts dev will provide good
code
Dev and Ops can
work together to  Dev trusts ops will put code in
Operations permanently prod quickly
remove root causes
of bugs and failures  Visibility enables “trust but verify”
16
 Dev and Ops Working Together

 Create feedback loops between

inventors and mechanics

 Expose real-time metrics from

ops enabling dev to learn from
the system running under real
world conditions

 Expose real-time metrics from

dev enabling ops to anticipate
production needs and provide
early input

 Cross-functional teams
collaborate to deliver whole
working systems including all
infrastructure, software code, and
configurations

Feature delivery + stability become shared goals 17

Matching IT Capacity to Business Demand

18
Breaking the Silos:
Communication, Collaboration, Integration

Communication

Development Collaboration Integration

Operations Business

19
Breaking the Silos: Dev, Ops, and Business

Business
better
understands
capability for
Development changes to
features and
functionality

Dev can better

incorporate
needs of the
Operations business and Business
customers into
new
development

20
Breaking the Silos: Dev, Ops, and Business

Development

Business
better understands
Operations Business
operational
capabilities

Ops understands better how to

21
support business goals
 Business Demand:
Continuously Deliver Valuable Software

Modern business is dependent on IT

deploying new features

Need very fast time-to-value in the

face of change
 Immigration policy can change
rapidly – IT capacity must keep up
 Immigration Executive Action

Multiyear lead time no longer

acceptable

Expectations for delivery times

continue to decrease
 Business Demand:
Support Modern Norms of Customer Interaction

Software is increasingly customer-

facing, rather than internally-facing

Customers expect an interactive, self-

service interface

Customers expect deep, direct

engagement with their data, not a paper
system

Customers expect to be able to get

information immediately

Customers can now identify problems

in our systems directly – and they
expect us to fix them
Business Demand:
Rapidly Incorporate Latest Technology
Modern web interfaces

Mobile devices

Social media

Accessibility tools

Live customer interaction tools

Tools for online communities and user-generated content

Amazing new features

Business Demand: “Lean Bureaucracy”
Supporting Government Values
“Working in public”

Governance – many, many

stakeholders

Transparency in how we work

“Presentability” of what we produce

Mission alignment
DOES14 - Mark Schwartz

Risk aversion

Baked in support of values such as:

• Contracting preferences
• Hiring fairness
• Procurement fairness
25
Business Demand:
Respond to Feedback Very Quickly

System operations increasingly yields insights that must be acted

immediately to keep pace with demand

Availability of ubiquitous automated data collection yields expectations

that organizations will rapidly act on key data points to improve efficiency
in mission and services

With so many routes to innovation, organizations are expected to test and

identify the best options very quickly

Well run companies are expected to maintain very low MTTR (mean-time
to repair) times – delays in fixing problems can be catastrophic
Beware! Pain
Ahead!

If you turn back from

the journey now… … we won’t judge
 The Not-Recommended, All-Too-Familiar,
Pain-for-Everyone, To-Be-Avoided Approach
• Business makes even more audacious
Business starts commitments to catch up
missing commitments
• Developers see more and more urgent
to the outside world, projects coming in
and then…
• All effort is spent on features as opposed to
non-functional requirements

• More shortcuts, more technical debt, more

fragility
This approach preordains
us to failure • Deployments become more difficult – what
took a weekend now takes 3 days!
Creates a permanent
wedge between making • We try to fix this by doing less deployments,
urgent business changes increasing batch size
and maintaining stability

Working here is a major

• More moving parts, more failures… we are
drag consumed by unplanned work 28
Results: Puppet Labs State of DevOps 2014 Report

 Scientific study of relationship between organizational performance, IT

performance, and DevOps practices

 9200 respondents representing 110 countries

Findings

 DevOps adoption is accelerating

 High-performing IT organizations deploy code 30
times more frequently with 50% fewer failures
 Strong IT performance is a competitive advantage
 DevOps improves IT performance
 Organizational culture matters
 Job satisfaction is the No. 1 predictor of organizational
performance

High performing companies are good at getting better –

29
nobody starts out high performing
The Three Ways of DevOps by Gene Kim

The Three Ways describe the values that

frame the processes of DevOps and they
provide prescriptive steps
3rd Way
Culture of continual experimentation
Understanding that mastery requires
practice
2nd Way
Creating feedback
loops

1st Way
Emphasize entire system
performance versus a specific
silo of work
DevOps: The First Way

31
The First Way: Systems Thinking

Use systems thinking to ensure

work always flows forward
“Work moving backwards, or standing still, is almost always indicative of
problems that need to be solved, and will span people, process and
technology.” –Gene Kim 32
What is a silo, really?
Disconnection from other people
No shared context
Different management

Barriers build up
Different incentives
Different objectives

Bad handoffs
Lack of understanding
Lack of empathy

“The nature of a large, complex organization is to fall out of alignment

33
without deliberate effort – inertia pulls it apart” –Damon Edwards
 The First Way:
Understand the Flow of Work
 Work starts with a description of features
needed by the business

 Work ends with the stable, secure and

reliable delivery of services to the customer

 Additional sources of work:

 IT finds defects
 Help desk fields incident reports
 Security raises compliance
requirements
 Enterprise architecture initiatives e.g.
single sign-on

 Visualize work
 Measure the flow of work (cycle time, lead
time, wait times)
 Think about software production as a value stream similar to a 34
manufacturing value stream
Organizations are Complex Systems

Human complex system

Communication patterns
Locations
Work styles
Personalities
Roles and responsibilities
Skill sets

Technology complex system

Programming Languages
Tools
Networks
Configurations
Interconnections

One complex system working on another complex system 35

 The First Way:
Always Seek to Increase Flow

 Reduce work in progress (WIP)

 Reduce batch size of deliveries
 Reduce variation in size of work items

 Make policies explicit

 Eliminate inventory and
other waste
 Maintain a steady,
sustainable pace

Deliver often – and get really good at it 36

The First Way:
Optimize Flow Globally, Not Locally

 Focus on interactions between parts of the system

 Build controls into the system
 Local efficiencies are good, but should never jeopardize global goals
 Avoid tribal warfare!
 Know your bottlenecks … and elevate them
 The bottleneck is the lever of control for speed of flow through your process

Upstream Bottleneck Downstream

Queues to Flow is Starved of
be serviced restricted full flow

37
The First Way:
Never Consciously Pass Defects Downstream
 Create quality at the source
 Make rework visible
 Understand the origination point of defects in order to avoid recurrence
“This is legacy code, I’ll just make
the change for my story, I don’t
have time to fix the rest of this”
“That issue is a doozy… leave
it to fix in the hardening sprint”

“Just push this feature over to the

testers… it’s their job to find
defects, right?” “Call the story done. We know
there are still a few problems so
just open up some defects
against it” 38
The First Way:
The System of Profound Knowledge
Organizations are systems of interrelated processes and people
which form the system’s components

Components of the system must reinforce, not compete with

each other to accomplish the aim of the system

Workers’ success of depends on managing the balance between

each component to optimize the system

Understand business Understand cause and effect

goals – how value is
achieved Make informed decisions
based on rich, accurate,
Understand people,
and timely information
processes, and
technologies
Teach the organization
how to fix and regulate
Understand risks itself
and risk controls 39
The First Way:
Bringing It All Together

Dev

What is the minimum

viable product?
Is it profitable?
Do we have the

Business
capability to build it
and maintain it?

End Users

Ops 40
DevOps: The First Way – Practices

Communication

Collaboration Integration

41
 Communication

DevOps Practice:
Deploy Shippable Environments Collaboration Integration

SHIPPABLE CODE
AND SHIPPABLE
ENVIRONMENT
Traditionally, dev is responsible for applications while ops is responsible
for environments

In DevOps, we use a single repository for everything –functional code, test

code, environment configurations, and tool configurations
42
DevOps Practice:
Infrastructure as Code

“Programmable infrastructure”

“Fully automated configuration

management”

Code to automate provisioning

Code to manage configurations 43
 Communication

DevOps Practice:
One Step Environment Creation Collaboration Integration

Provision and configure environments at the

touch of a button

Make production-like environments available

early in the dev process

Build code & environment at the same time

Create a common dev, testing, and prod

environment creation process

44
Everyone uses a consistent environment
 Communication

DevOps Practice:
The Daily Build Collaboration Integration

 “Heartbeat of the project” and “clean room every day”

 Rebuild every line of code from scratch – be able to reconstitute the
system from “bare metal”

 Run all the tests!

 Check all dependencies
 Verify no defects introduced yesterday
 Build all versions
 Automate with Continuous Integration (CI) server

45
 Communication

DevOps Practice:
Deploy Early, Often, and Quickly Collaboration Integration

Small deployments mean Fast deployments mean more deployments mean easier
46
deployments mean lower cycle times means faster time to market
 Communication

DevOps Practice:
Classify Ops Work by Four Types Collaboration Integration

Systematically allocating time to the 4 types enables all the work to get
done and becomes routine

Business Projects Internal IT Projects

Types of
work

Changes Unplanned Work

Exercise: Classify real USCIS work according to the

four types 47
Doing DevOps at USCIS – First Way

 Understand and measure your flow of work using visualizations such as a

Kanban board and value stream map

 Use a single USCIS repository for source code, test code, and environment
configuration scripts

 Builds done by automated, script-driven retrieval of source code by a

Continuous Integration (CI) server

 Frequent deployments – no less than every two weeks

 Consistent record of successful deployments
 Baked in accessibility and security compliance – no compliance work flowing
backwards

What is the concept of a “Team-Managed Deployment” at USCIS? 48

DevOps: The Second Way

49
The Second Way: Amplify Feedback Loops

• Shorten and amplify “right to left” feedback loops

• Use feedback to create even higher quality at the

source

• Create and embed knowledge where it is needed to

provide immediate feedback

• Understand needs of all customers, internal and

external, and respond to their feedback

50
The goal of any process improvement is to shorten and amplify feedback loops
 The Second Way:
Shorten and Amplify Feedback Loops

Commit
Manual tester
Test Build
overloaded due
Develop to end of sprint
Operations
10 steps to get
feedback & VERY long
delay
Product Backlog

End Users

Product Owner/ Triage Issue Tracker Help Desk 51

Value Team
The Second Way:
Shorten and Amplify Feedback Loops (cont)

Test Build
Commit

5 steps to get
feedback
Develop

Manual Test

Issue Tracker
52
The Second Way:
Shorten and Amplify Feedback Loops (cont)

 Most users won’t call… some may just quit being customers
 Many defects remain latent for a long time
 By the time defects come back, dev forgets how the code works

Commit
4 Steps to get
feedback – automated Test Build

and quick!
Develop

Failed
Automated Test 53
The Second Way:
Use Feedback to Create Quality at the Source

 Development is the source of quality – or problems

 As applications evolve, changes must not negatively impact end user
experiences

 Developers need access to deep diagnostics so they can incorporate

latest operational concerns and understand impact of their changes

Browser performance metrics Traces from slow transactions that

suggest performance bottlenecks in
Application response times distributed applications

Server usage Service-oriented architecture issues

spanning multiple application tiers
Performance data by technology
component Correlation of application response
times on end-user satisfaction levels
Runtime code diagnostics including
database queries
 Communication

The Second Way:

Create and Embed Knowledge Collaboration Integration

• Ops and Security:

• Become part of the agile process – especially planning and prioritization
• Provide recommendations and requirements as new code developed
• Ensure relevant metrics are monitored early in the dev process

• Dev participates in incident handling to acquire knowledge to prevent future

problems:
• Incident escalation
• Root cause analyses
• Post-mortems
• Ops receives cross training
by dev and security

• Extend agile practices to all

teams
• Visible work
• Open meetings
• Working agreements
• Explicit policies
55
The Second Way:
Respond to Needs of All Customers

 Use a service model for both internal and external customers

 Agile encouraged dev and test to focus on customer collaboration with business
stakeholders and end users

 DevOps extends the service model to Dev and Ops treating each other as
customers

Customer

Service
Provider

Change Orientation Stability Orientation

56
DevOps: The Second Way - Practices

Communication

Collaboration Integration

57
 Communication

DevOps Practice:
Deployment Automation Collaboration Integration

 Problems in deployment procedure will be found quickly and can be

permanently eliminated

 Runs fast “smoke test” to ensure system is running as expected

 Built-in automatic rollback and/or redeploy
 Build confidence through frequent repetition – the prospect of
deployments and rollback no longer instill fear

 Create a virtuous cycle of successful deployments, smaller

deployments, and lower risk

58
DevOps Practice:
Communication

Operations Monitoring Collaboration Integration

Monitoring gives us continuous, live feedback about how the system is running
“Tell me what is happening before the phone rings”
User Feedback Approach Monitored Approach
Field user calls Automatic alert about a problem when
it happens
Multiple people call Monitoring tools show me how
widespread the problem is
Users can’t tell me the real source of I can see which component of the
the problem application is generating errors

59
Operations Monitoring – Needs and Challenges

Monitoring Challenges

 Operators typically responsible for numerous applications

 Environments can be complex with unique or complicated application
stacks
 Visibility into different components can be vague or non-existent
 Quantity of logged data can be overwhelming
 Combining monitoring tools into a single view that provides insight
Monitoring Needs

 Active production monitoring, not just reacting to downtime

 Easily monitor critical areas of application stability with minimal tooling
 Tune dashboard to display key database, network, server, and
application performance measurements in a holistic view
 Ability to quickly share potential performance issues with your team
60
 Communication

DevOps Practice:
Operations Monitoring Dashboard Collaboration Integration
Application
Performance
Index (User
Application
Satisfaction)
Response
Time

Application
Throughput

Transaction
Timings (drill down
capability to code
level, transaction Alerts
level)
 Communication

DevOps Practice:
Operations Monitoring Drives Dev & Ops Priorities
Collaboration Integration
 Communication

DevOps Practice:
Prioritize Fixing Production Defects Collaboration Integration

Prioritize fixing defects very fast

• Assume incidents will occur

• Ensure ops feedback will come

back rapidly

• Ensure developers will get the info

they need to fix the problem

• Add automated tests to ensure

problem cannot reoccur

• Get really good at fixing defects

very fast 63
 Communication

DevOps Practice:
Reusable Ops and Security User Stories Collaboration Integration

User Story
User Story
As security I want cross-site
As an ops engineer I
scripting attacks prevented so that
want to monitor how many people
access controls cannot be
are listening to audio feeds so I
bypassed
can tune playback quality during
Estimate Priority spikes in demand
5 points 1 (High) Estimate Priority
3 points 2 (Med)
Acceptance Criteria On back…
• Verify all input is filtered as Acceptance Criteria On back…
potentially malicious • Verify the count of active audio
• Verify all output of the page is sessions is displayed in the
encoded to the explicitly defined application’s admin dashboard
character set • Verify the count is accurate as
• Verify output is sanitized by escaping playback sessions are added or
dynamic content to properly enforce completed
separation of code and data

64
DevOps Practice:
Communication

Dev & Ops Common Communication System Collaboration Integration

Remove all barriers to internal communication, collaboration, and

integration

 Use common, intuitive dashboards combining information from all groups

 Key operational metrics
 Visible dev, ops, and security workflow (e.g. Kanban boards)
 List of recent and upcoming system changes
 Stability of the system
 Security status
 Schedules, planned release dates, and critical business dates

 Integrated alert policies

 Common internal note system – histories of defects and incidents can be
very useful

 Shared wikis, file repositories, chat spaces, specs, and documentation65

 Communication

DevOps Practice:
Track Dev & Ops Business Impact Collaboration Integration

 MTTR – Mean Time To Repair – How long is the system down?

 MTBF – Mean Time Between Failures – How often is the system down?
USCIS Example:
Key Performance Parameter (KPP) Service Agreement Objective Actual
Low Threshold

Reliability – uninterrupted correct 641 hours 712 hours 739 hours

function Exceeded Objective

Availability – 24/7 operations 97.63 % 98.88% 99.32%

Exceeded Objective

Maintainability – prompt restoration No more than 10 hours No more than 8 hours 5 hours
of service after outage Exceeded Objective
Doing DevOps at USCIS – Second Way

 Demonstrated information integration and collaboration between dev, ops,

security, and business

 Partial or completely automated deployment – rapid, reliable, testable,

repeatable

 Operational Monitoring Plan – preferably a dashboard

 Defined business impact measurements and thresholds

See Team-Managed Deployment Management Instruction for more information

67
Automating an
Integrated DevOps
System
Systems to Make Software

Communication
System U l Version
Control
(CM)


A good method of enabling DevOps
Monitoring
System is to simply begin connecting and
automation the systems you use to ^ Requirements

make software.

• Start where you are

• Identify possible
Deploy
System
R •
•
interconnections
Research tools to automate
Create future state roadmap
A Build
System

• Let pipeline emerge

• Continue to improve the
sequence of connections as
systems change
Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
69
Version Control

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Version Control
Deploy
System
R Ensures you’re working on the right
version of something
A Build
System

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
70
Requirements System

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Requirements System
Deploy
System
R Houses project requirements in a
prioritized list and allows for item
A Build
System

allocation to sprint/team member;

Allows for traceability of
dependencies between stories

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
71
Build System

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Build System
Deploy
System
R Software tools designed to
automate the process of program
A Build
System

compilation to create a
deployable package

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
72
Test System

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Test System

orchestrated set of tests, both

A
manual and automated that Build
Deploy
System
R ensure the functionality and
accuracy of code
System

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
73
Code Review System

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Code Review System

Ensures that code complies with

standards and identifies low level
Deploy
System
R
bugs and coding errors; Identifies
design and requirements
compliance
A Build
System

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
74
Issue Tracking System

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Issue Tracking System

Deploy
System
R
Collects issues throughout the
cycle of the project and track A Build
System

them through completion

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
75
Documentation System

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Documentation System

Deploy
System
R Repository of system information
throughout the lifecycle
A Build
System

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
76
Deploy System

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Deploy System

Deploy
System
R Installs and configures the
package created by the build
A Build
System

system into appropriate

environments

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
77
Monitoring

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Monitoring System

Deploy
System
R Collects current-state data to
determine health of all
A Build
System

environments

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
78
Communication System

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Communication System

Deploy
System
R Method for conveying information
between systems
A Build
System

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
79
Automate All the Connections!

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Deploy
System
R A Build
System

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
80
Orchestrating Integration with a Pipeline
Pipelines

A Pipeline is a chain of tasks that can be automated

Unit
User Commits Merge code Build Code Review Log Issues Deploy
test/coverage

 Integration tools use pipelines to perform tasks repetitively and

continuously

 The process is called Continuous Integration (CI)

 Pipelines keep work flowing forward in our DevOps system

82
We Need Something to Integrate the Systems

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Pipeline
Orchestration

Deploy
System
R A Build
System

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
83
Development Pipeline Example

Communication
System U l Version
Control
(CM)

Monitoring
System  ^ Requirements

Pipeline
Orchestration

Deploy
System
R A Build
System

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
84
Development Pipeline Example with Integration
System Commit code

Communication
System U l Version
Control
(CM)

Monitoring
System  Code is
committed and
Merged
^ Requirements

Pipeline
Orchestration

Deploy
System
R
Initiate Build

Initiate
A Build
System

Testing

Documentation
System  a Test System

Code
Issue
Tracking i  Review
System
85
Pipeline Stages
Continuous Integration

Code Done Unit Tests Integrate

Auto Auto
Continuous Delivery

Acceptance Deploy to
Code Done Unit Tests Integrate
Testing Production

Auto Auto Auto Manual

Continuous Deployment

Acceptance Deploy to
Code Done Unit Tests Integrate
Testing Production

Auto Auto Auto Auto 86

CI Pipeline Example

CI Pipeline

Commit
Build gate Staging gate Integration
gate
New Feature
CM Repository

Compile Code Compile Code

(NF)
Compile Code
Code Quality Functional

Integration
Gates Applied Tests Merge with

Staging
Commit

Master
Master
Smoke Tests If Successful,
Legacy Feature If Successful, Merge with Fortify Scans
(LF) Merge with Integration
Staging Branch Release is
Commit packaged

Bad Feature
(BF)

Fail Success Fail Success Fail Success

STOP STOP STOP

87
Using a CI/CD Pipeline for Team-Managed
Deployments at USCIS

Approval:

RRR OR eRRR OR TMD

CI/CD Pipeline:

Auto. Auto. Manual Deploy

Build Test Test

DevOps:
Development Operations

Team Managed Deployment (TMD) provides the approval step for a CI/CD
Pipeline. The CI/CD Pipeline provides the forward link from Development to
Operations.
Using a CI/CD Pipeline for Team-Managed
Deployments at USCIS (cont)

RRR Documents CI/CD Pipeline Artifacts

VDD Pipeline
Release Number Release Number

Deployment Scripts
Source Code File List

List of Changes
Version
Deployment Control
Source Code File List
Instructions
List of Changes

TAS
Test Results Test Tools
Test Results
Test History
Test History

Automation used in a CI/CD pipeline allows data to be collected as true

artifacts. In an RRR approach, the information is manually collected into
documents.
DevOps: The Third Way

90
 DevOps is Not…

A tool

A role

A team

Something that can be

purchased or simply switched
on

DevOps requires a culture of operations and development engineers participating

91 together in the entire service lifecycle
The Third Way: Culture of Improvement

• Continual experimentation, taking risks, and learning

from failure

• Understanding that repetition is the prerequisite to

master

92
The Third Way:
Experimentation, Risk-Taking, and Learning

Develop a culture that pushes into the

danger zone

Develop habits to survive danger

Build experimentation, risk-taking, and

learning into our way of doing business

Break things early and often

Intuit ran 165 experiments on their

TurboTax product in the 3 main months of
tax season – ideas made it to market a
year earlier and they increased customer
conversion rate by 50% 93
The Third Way:
Repetition for Mastery

• Do the hard parts often

• Work through pain points to make the

process easier

• Do painful things MORE frequently to make

it less painful

• Reduce anxiety about unexpected outcomes

• Automate!!!
94
DevOps: The Third Way - Practices

Communication

Collaboration Integration

95
 Communication

DevOps Practice:
Inject Failures Collaboration Integration

• Netflix services are hosted completely in

Amazon Web Services cloud

• Design each distributed system to expect

and tolerate failure

• Chaos Monkey randomly kills

services within architecture in order to
learn to tolerate and respond to failure

DevOps Approach Traditional Approach

 How does this system react if I do  Not in my system, you don’t
this?
 Can we continue operations  Not in my system, you don’t
without this server?
 Will the users prefer option A or  Not in my system, you don’t
option B? 96
 Communication

DevOps Practice:
Make Your Improvement Work Visible Collaboration Integration

Along with regular user stories, use

colored cards to indicate:
• Technical debt
• Unplanned work
• Experiments
• Learning backlog

Allocate time to improve daily work

Track the work needed to maintain overall

health of the system

97
 Communication

DevOps Practice:
Regularly Improve Technical Debt Collaboration Integration

Allocate 20% of cycles to Technical Debt Reduction

• Write tests to find misconfigurations – and fix them

• Constantly run automated static code analysis during

continuous integration and testing, and raise the bar in your
quality gates

• Enforce consistency in code, environments, and configurations

• Repeatedly tackle the hard stuff 98

 Communication

DevOps Practice:
Regularly Improve Tools Collaboration Integration

 Good tools are key to enabling

DevOps collaboration, automation,
and visibility

 Provide teams the best tools available

 Regularly invest time researching and
piloting new tools

 Provide expert support

99
 Communication

DevOps Practice:
Reward Contributions to a DevOps Culture Collaboration Integration

 Incentivize DevOps practices and behaviors

 Recognize experimentation and risk-taking that leads to valuable
learning

 Model honest self-assessment of organizational strengths and

weaknesses and use of improvement techniques such as Toyota
Kata

 Quantify and promote the link between DevOps practices and

organizational performance

100
 Communication

DevOps Practice:
Conduct Deliberate Culture Change Experiments Collaboration Integration

Decentralized,
emergent

Protected,
Boss’ orders dedicated
team
Org
Change
Patterns

Champions / Organizational
advocates baby steps

Discussion: What are our biggest cultural challenges? What experiments

101
should we run?
DevOps Team Profiles

DevOps Team Member

 End-to-end viewpoint DevOps Expert Support Team

 Contributes to and uses visibility  Helps introduce DevOps-

supportive processes and tools
 Automator
 Works with teams to automate
environment creation and
 Collaborative, cross-functional, deployment
friction reducer
 Helps teams use operational
 Participates in collective performance logs and
ownership of code and code dashboards
delivery
 Provides infrastructure support
 Personal success = team success
 Enjoys working this way 102
 Food for Thought – Maturing DevOps Practices

Level 1 Level 2 Level 3 Level 4 Level 5

 Frequent commits  One backlog team and a  Team collaboration  Dedicated tools team for  Cross functional teams
 Prioritized work master backlog  Remove boundary of automation  No rollbacks
  
Culture & Defined & documented Adopt basic agile dev & ops Deploy disconnected
process methods  Act on metrics from Release
Processes  Remove boundary of dev  Common processes for  Continuous improvement
& test all changes
 Define Context View,  Stage SDD wiki with  Review process in place  ?  ?
Logical Composition & Views in place
Physical Composition  Define related views
Architecture

 Versioned code base  Poling builds  Auto triggered builds  Zero downtime deploy  Zero touch continuous
 Scripted builds  Build are stored  Automated tag &  Multiple build machines deployments
 Basic scheduled builds  Manual Tag & Versioning versioning  Full automated DB
 Dedicated build server  First step towards  Automated bulk of DB deploys
 Documented manual standardized deploys changes

Build / Deploy deploy Basic pipeline with
deploy to prod
 Scripted configuration
changes
 Standard process for all
environments
 Automated Unit Tests  Automated Unit Tests  Automated Functional  Automated acceptance  All tests automated
(Coverage <50%) (Coverage >50% & < tests criteria (80%)  Coverage 100%
 Separate test 80%)  Automated acceptance  Automated performance
environment  Automated Integration criteria (<40%) tests

Test & Verification Tests (Coverage ??) Automated Security
Tests
 Automated Section 508
tests

 Baseline process metrics  Static code analysis  History of reports  Report trend analysis  Reports accessible via
Collaboration &  Manual reporting  Quality reports available  Graphing as a service common dashboard
 
Information Traceability built into Dynamic graphing
pipeline
Sharing 103
Wrap Up

How much more productive, effective, and enjoyable might our work be? How much
business value is left on the table due to unmatched demand and capacity?

Can we afford not to do DevOps?

Questions

104