145 BROUGHT TO YOU IN PARTNERSHIP WITH

CONTENTS

öö ABOUT CONTINUOUS DELIVERY

öö THE DEPLOYMENT PIPELINE

öö BENEFITS

Continuous Delivery öö

öö
CONFIGURATION MANAGEMENT

CONTINUOUS INTEGRATION (CI)

Patterns and Anti-Patterns öö

öö
TESTING

DEPLOYMENT PIPELINE

öö BUILD AND DEPLOYMENT

SCRIPTING

ORIGINAL BY PAUL DUVALL, CTO AND CO-FOUNDER, STELLIGENT öö AND MORE...

UPDATED BY MICHAEL OLSON, PRINCIPAL PRODUCT MARKETING MANAGER, PUPPET

ABOUT CONTINUOUS DELIVERY The purpose of the deployment pipeline is threefold:

With Continuous Delivery (CD), teams continuously deliver new
versions of software to production by decreasing the cycle time • Visibility: All aspects of the delivery process – building,

between an idea and usable software through the automation testing, deploying, and releasing – are visible to all team

of the entire software delivery process: code commit, build, test, members promoting collaboration.
DZO N E .CO M/ RE FCA RDZ

deployment, and release. CD is enabled through the Deployment

• Feedback: Team members learn of problems as soon as they
Pipeline, which encompasses a collection of patterns described in
occur so that issues are fixed as soon as possible.
this Refcard.
• Continually Deploy: Through a fully automated process,
CD is concerned with “…how all the moving parts fit together:
you can deploy and release any version of the software to
configuration management, automated testing, continuous
any environment faster and more frequently.
integration and deployment, data management, environment
management, and release management.” In the Deployment Pipeline diagram above, all of the patterns
are shown in context. There are some patterns that span multiple
THE DEPLOYMENT PIPELINE
stages of the pipeline, so I chose the stage where it’s most
predominantly used.

Critical Considerations
for Continuous Delivery
Standardize + automate
software delivery from
beginning to end.

Get the Ebook

1
 Get to continuous delivery faster with
Puppet Pipelines.
• Enable self-service builds and deployments for your developers
• Get deep visibility into your delivery pipeline & the status of every deployment
• Define continuous delivery pipelines for your cloud-native and traditional
apps in under 15 minutes

Meet Puppet Pipelines > Request a demo >

Learn more at puppet.com

 CD PATTERNS AND ANIT-PATTERNS

BENEFITS MAINLINE (3)

• Empowering Teams: Because the deployment pipeline is a
Minimize merging and keep the number of active code
pull system, testers, developers, operations, and others can Pattern
lines manageable by developing on a mainline.
self-service the application version into an environment
of their choice. Anti-
Multiple branches per project.
patterns
• Reducing Errors: Ensuring the correct version,
configuration, database schema, etc. are applied the same
MERGE DAILY (1)
way every time through automation.
Changes committed to the mainline are applied to
• Lowering Stress: Through push-button releases to Pattern
each branch on at least a daily basis.
production and rehearsing deployments, a release becomes
commonplace without the typical stress.
Anti- Merging every iteration once a week or less often than
• Deployment Flexibility: Instantiate a new environment or patterns once a day.
configuration by making a few changes to the automated
delivery system.
PROTECTED CONFIGURATION (5), (1)

• Practice Makes Perfect: Through the deployment Store configuration information in secure remotely
pipeline, the final deployment into production is being Pattern accessible locations such as a database, directory,
rehearsed every single time the software is deployed to any or registry.
target environments.
Anti-
Open text passwords and/or single machine or share.
patterns
DZO N E .CO M/ RE FCA RDZ

CONFIGURATION MANAGEMENT
Configuration Management is “the process by which all artifacts
relevant to your project, and the relationships between them, are REPOSITORY (3), (1)
stored, retrieved, uniquely identified, and modified.” (1)
All source files — executable code, configuration, host
Note: Each pattern is cited with a number in parentheses that Pattern environment, and data — are committed to a version
corresponds to the source in the References section. control repository.

CONFIGURABLE THIRD-PARTY SOFTWARE (1) Some files are checked in, others, such as environment
Anti- configuration or data changes, are not. Binaries –
Evaluate and use third-party software that can be patterns which can be recreated through the build and
Pattern
easily configured, deployed, and automated. deployment process – are checked in.

Procuring software that cannot be externally

Anti- SHORT-LIVED BRANCHES (1)
configured. Software without an API or command line
patterns
interface that forces teams to use the GUI only. Branches must be short lived – ideally less than a few
Pattern
days and never more than an iteration.

CONFIGURATION CATALOG (1)

Anti- Branches that last more than an iteration. Branches by
patterns product feature that live past a release.
Maintain a catalog of all options for each application,
how to change these options and storage locations for
Pattern
each application. Automatically create this catalog as SINGLE COMMAND ENVIRONMENT (1)

part of the build process.

Check out the project’s version-control repository
and run a single command to build and deploy the
Anti- Configuration options are not documented. The catalog Pattern
application to any accessible environment, including
patterns of applications and other assets is “tribal knowledge”. the local development.

3 BROUGHT TO YOU IN PARTNERSHIP WITH

 CD PATTERNS AND ANIT-PATTERNS

Forcing the developer to define and configure Scheduled builds, nightly builds, building periodically,
Anti-
Anti- environment variables. Making the developer install building exclusively on developer’s machines, or not
patterns
patterns numerous tools in order for the build/deployment building at all.
to work.

STOP THE LINE (5), (1), (4), (12)

SINGLE PATH TO PRODUCTION (1)
Fix software delivery errors as soon as they occur; stop
Configuration management of the entire system - Pattern the line. No one checks in on a broken build, as the fix
source, configuration, environment, and data. Any becomes the highest priority.
Pattern
change can be tied back to a single revision in the
version-control system. Builds stay broken for long periods of time, thus
Anti-
preventing developers from checking out
patterns
Anti- Parts of the system are not versioned. Inability to get functioning code.
patterns back to a previously configured software system.

INDEPENDENT BUILD (6)

CONTINUOUS INTEGRATION (CI) Write build scripts that are decoupled from IDEs.
BUILD THRESHOLD (5)
Pattern These build scripts are executed by a CI system so that

Fail a build when a project rule is violated – such software is built at every change.

Pattern as architectural breaches, slow tests, and coding

Anti- Automated build relies on IDE settings. Builds are
standard violations.
DZO N E .CO M/ RE FCA RDZ

patterns unable to be run from the command line.

Anti- Manual code reviews. Learning of code quality issues

patterns later in the development cycle. VISIBLE DASHBOARDS

Provide large visible displays that aggregate

COMMIT OFTEN (6)
information from your delivery system to provide
Pattern
Each team member checks in regularly to trunk — at high quality feedback to the Cross-Functional Team

Pattern least once a day but preferably after each task to in real time.

trigger the CI system.

Anti- Email-only alerts or not publicizing the feedback to
patterns the entire team.
Anti- Source files are committed less frequently than daily
patterns due to the number of changes from the developer.

TESTING
CONTINUOUS FEEDBACK (6) AUTOMATE TESTS

Send automated feedback from CI system to all cross- Automate the verification and validation of software
Pattern
functional team members. Pattern to include unit, component, capacity, functional, and
deployment tests
Notifications are not sent; notifications are ignored;
Anti-
CI system spams everyone with information they Anti- Manual testing of units, components, deployment,
patterns
cannot use. patterns and other types of tests.

CONTINUOUS INTEGRATION (6) Unit: Automating tests without any dependencies.

Building and testing software with every change Component: Automating tests with dependencies to other
Pattern components and heavyweight dependencies such as the database
committed to a project’s version control repository.
or file system.

4 BROUGHT TO YOU IN PARTNERSHIP WITH

 CD PATTERNS AND ANIT-PATTERNS

Deployment: Automating tests to verify the deployment and

configuration were successful. Sometimes referred to as a “smoke tests.” Anti- Separately defined processes and views of the check-
patterns in to release process.
Functional: Automating tests to verify the behavior of the software
from a user’s perspective.
BUILD AND DEPLOYMENT SCRIPTING
Capacity: Automating load and performance testing linear DEPENDENCY MANAGEMENT (5)
production conditions.
Centralize all dependent libraries to reduce bloat,
ISOLATE TEST DATA (1) class path problems, and repetition of the same
Pattern
dependent libraries and transitive dependencies from
Use transactions for database-dependent tests
project to project.
(e.g. component tests) and roll back the transaction
Pattern
when done. Use a small subset of data to effectively Multiple copies of the same binary dependencies
test behavior. Anti-
in each and every project. Redefining the same
patterns
information for each project. This is classpath hell!
Anti- Using a copy of production data for Commit Stage
patterns tests. Running tests against a shared database.
COMMON LANGUAGE (1)

PARALLEL TESTS (1) As a team, agree upon a common scripting language

Pattern — such as Perl, Ruby, or Python — so that any team
Run multiple tests in parallel across hardware
Pattern member can apply changes to the Single Delivery System.
instances to decrease the time in running tests.
Each team uses a different language making it difficult
Anti-
DZO N E .CO M/ RE FCA RDZ

Anti-patterns Running tests on one machine or for anyone to modify the delivery system reducing
Anti- patterns
instance. Running dependent tests that cannot be run cross-functional team effectiveness.
patterns
in parallel.

EXTERNALIZE CONFIGURATION (5)

STUB SYSTEMS (1)
Changes between environments are captured as
Use stubs to simulate external systems to reduce
Pattern configuration information. All variable values are
deployment complexity. Pattern
externalized from the application configuration into
build/deployment-time properties.
Anti- Manually installing and configuring interdependent
patterns systems for Commit Stage build and deployment.
Anti- Hardcoding values inside the source code or per
patterns target environment.

DEPLOYMENT PIPELINE
FAIL FAST (6)
A deployment pipeline is an automated
Pattern implementation of your application’s build, test, Fail the build as soon as possible. Design scripts
deploy, and release process. Pattern so that processes that usually fail run first. These
processes should be run as part of the commit stage.
Deployments require human intervention (other than
Anti-
approval or clicking a button). Deployments are not Anti- Common build mistakes are not uncovered until late
patterns
production ready. patterns in the deployment process.

VALUE-STREAM MAP (4) FAST BUILDS (6)

Create a map illustrating the process from check in to The commit build provides feedback on common build
Pattern the version control system to the software release to Pattern problems as quickly as possible — usually in under
identify process bottlenecks. 10 minutes.

5 BROUGHT TO YOU IN PARTNERSHIP WITH

 CD PATTERNS AND ANIT-PATTERNS

Throwing everything into the commit stage process, Anti- Production is taken down while the new release is
Anti- such as running every type of automated static patterns applied to production instance(s).
patterns analysis tool or running load tests such that feedback
is delayed.

SCRIPTED DEPLOYMENT (5)

All deployment processes can be written in a script,

Pattern checked in to the version-control system, and run as
part of the single delivery system.
DARK LAUNCHING (11)
Deployment documentation is used instead of
Anti-
automation. Manual deployments or partially Launch a new application or features when it affects
patterns Pattern
manual deployments. the least number of users.

Anti-
UNIFIED DEPLOYMENT (5) Software is deployed regardless of number of active users.
patterns

The same deployment script is used for each

Pattern deployment. The protected configuration – per ROLLBACK RELEASE (5)
environment — is variable but managed.
Provide an automated single command rollback of
Pattern
Different deployment script for each target environment changes after an unsuccessful deployment.
DZO N E .CO M/ RE FCA RDZ

Anti-
or even for a specific machine. Manual configuration
patterns
after deployment for each target environment. Manually undoing changes applied in a recent
Anti-
deployment. Shutting down production instances
patterns
while changes are undone.
DEPLOYING AND RELEASING APPLICATIONS
BINARY INTEGRITY (5)
SELF-SERVICE DEPLOYMENT (1)
Build your binaries once, while deploying the binaries
Pattern
to multiple target environments, as necessary. Any Cross-Functional Team member selects the
Pattern version and environment to deploy the latest
Anti- Software is built in every stage of the deployment working software.
patterns pipeline.

Deployments released to team are at specified intervals

Anti-
CANARY RELEASE by the “build team.” Testing can only be performed in a
patterns
shared state without isolation from others.
Release software to production for a small subset
Pattern of users (e.g. 10%) to get feedback prior to a
complete rollout.
INFRASTRUCTURE AND ENVIRONMENTS
Anti- AUTOMATE PROVISIONING (1)
Software is released to all users at once.
patterns
Automate the process of configuring your
Pattern environment to include networks, external services,
BLUE-GREEN DEPLOYMENTS (1)
and infrastructure.

Deploy software to a non-production environment (call

it blue) while production continues to run. Once it’s Configured instances are “works of art” requiring team
Pattern Anti-
deployed and “warmed up,” switch production (green) members to perform partially or fully manual steps to
patterns
to non-production and blue to green simultaneously. provision them.

6 BROUGHT TO YOU IN PARTNERSHIP WITH

 CD PATTERNS AND ANIT-PATTERNS

BEHAVIOR-DRIVEN MONITORING (1) DATA

DATABASE SANDBOX (7)
Automate tests to verify the behavior of the
Pattern infrastructure. Continually run these tests to provide Create a lightweight version of your database – using

near real-time alerting. the Isolate Test Data pattern. Each developer uses
Pattern
this lightweight DML to populate his local database

Anti- No real-time alerting or monitoring. System sandboxes to expedite test execution.

patterns configuration is written without tests.

Shared database. Developers and testers are unable
Anti-
to make data changes without it potentially adversely
IMMUNE SYSTEM (9) patterns
affecting other team members immediately.

Deploy software one instance at a time while

conducting behavior-driven monitoring. If an error DECOUPLE DATABASE (1)
Pattern
is detected during the incremental deployment, a
rollback release is initiated to revert changes. Ensure your application is backward and forward
Pattern compatible with your database so you can deploy
Anti- each independently.
Non-incremental deployments without monitoring.
patterns

Anti- Application code data are not capable of being

LOCKDOWN ENVIRONMENTS (1) patterns deployed separately.

Lock down shared environments from unauthorized

external and internal usage, including operations DATABASE UPGRADE (7)
DZO N E .CO M/ RE FCA RDZ

Pattern
staff. All changes are versioned and applied
Use scripts to apply incremental changes in each
through automation. Pattern
target environment to a database schema and data.

The “Wild West:” any authorized user can access

Anti- Manually applying database and data changes in each
Anti- shared environments and apply manual configuration
patterns target environment.
patterns changes, putting the environment in an unknown
state and leading to deployment errors.
SCRIPTED DATABASE (7)

PRODUCTION-LIKE ENVIRONMENTS (1)

Pattern Script all database actions as part of the build process.
Target environments are as similar to production
Pattern
as possible. Using data export/import to apply data changes.
Anti-
Manually applying schema and data changes to
Environments are “production-like” only weeks or patterns
Anti- the database.
days before a release. Environments are manually
patterns
configured and controlled.

INCREMENTAL DEVELOPMENT
BRANCH BY ABSTRACTION (2)
TRANSIENT ENVIRONMENTS

Instead of using version-control branches, create an

Utilizing the Automate Provisioning, Scripted
Pattern abstraction layer that handles both an old and new
Deployment, and Scripted Database patterns. Any
Pattern implementation. Remove the old implementation.
environment should be capable of terminating and
launching at will.
Branching using the version-control system leading
Anti-
Anti- Environments are fixed to “DEV," "QA,” or other to branch proliferation and difficult merging.
patterns
patterns predetermined environments. Feature branching.

7 BROUGHT TO YOU IN PARTNERSHIP WITH

 CD PATTERNS AND ANIT-PATTERNS

TOGGLE FEATURES (10) CATEGORY EXAMPLE TOOLS

Deploy new features or services to production but Atlassian JIRA, Jama, CA Rally, Aha!,
Pattern Product Planning
limit access dynamically for testing purposes. CollabNet VersionOne, Pivotal

Waiting until a feature is fully complete before GitHub, GitLab, Atlassian Bitbucket,
Anti- Source Code
patterns committing the source code. Microsoft Team Foundation Server,
Management
Perforce, Subversion

COLLABORATION Jenkins, CircleCI, CloudBees, GitLab,

DELIVERY RETROSPECTIVE (1) Atlassian Bamboo, Travis CI, JetBrains
Continuous Integration
TeamCity, Microsoft Azure Pipelines,
For each iteration, hold a retrospective meeting
Puppet Pipelines
where everybody on the Cross-Functional Team
Pattern
discusses how to improve the delivery process for the
Ant, Gant, Gradle, make, Maven, Rake,
next iteration. Build
Fabric, Func

Waiting until an error occurs during a deployment

Anti- Twist , AntUnit, Cucumber, DbUnit,
for Dev and Ops to collaborate. Having Dev and Ops
patterns webrat, easyb, Fitnesse, JMeter, JUnit,
work separately. Testing
NBehave, SoapUI, Selenium, RSpec,
SauceLabs, Perfecto
CROSS-FUNCTIONAL TEAMS (1)
DZO N E .CO M/ RE FCA RDZ

JFrog Artifactory, Ivy, Archiva,

Everybody is responsible for the delivery process. Any Artifact Repository
Sonatype Nexus, Bundler
Pattern person on the Cross-Functional Team can modify any
part of the delivery system.
Puppet Pipelines, AWS CodePipeline,
Continuous Delivery & CA Automic, Electric Cloud, IBM
Siloed teams: Development, Testing, and Operations
Anti- Release Automation UrbanCode, Octopus Deploy,
have their own scripts and processes and are not part
patterns Spinnaker, XebiaLabs
of the same team.

Infrastructure
Puppet Enterprise, Chef, Ansible
Amazon.com has an interesting take on this approach. They call it Automation
“You build it, you run it”. Developers take the software they’ve written
all the way to production. Cloud Provisioning &
HashiCorp Terraform, Puppet, Ansible
Orchestration
ROOT-CAUSE ANALYSIS (1)

Container Management Kubernetes, Mesos, HashiCorp

Learn the root cause of a delivery problem by asking
System & Application Nomad, Docker Swarm,
Pattern “why” of each answer and symptom until discovering
Platform-as-a-Service CloudFoundry
the root cause.

Anti- Puppet Container Registry, Docker

Accepting the symptom as the root cause of the problem.
patterns Container Registry Hub, AWS, Microsoft Azure, Google
Cloud, JFrog, Red Hat Quay, Harbor

CONTINUOUS DELIVERY TOOLS

This is meant to be an illustrative list, not an exhaustive list, to give Application New Relic, AppDynamics, Datadog,
you an idea of the types of tools and some of the vendors that help Performance Monitoring Splunk, Dynatrace
to enable effective Continuous Delivery.

8 BROUGHT TO YOU IN PARTNERSHIP WITH

 CD PATTERNS AND ANIT-PATTERNS

6. Paul M. Duvall, “Continuous integration. Improving Software

Software Delivery
Puppet Insights, CloudBees Quality and Reducing Risk”, Addison Wesley, 2007
Performance
DevOptics, XebiaLabs 7. Scott W. Ambler and Pramodkumar J. Saladage,
Management
“Refactoring Databases. Evolutionary Database Design”,

Collaboration Slack, JIRA, Trello Addison Wesley, 2006.

8. Paul M. Duvall, IBM developerWorks series “Automation for

the people” ibm.co/iwwvPX
REFERENCES
9. IMVU: bit.ly/jhqP5f
1. Jez Humble and David Farley, “Continuous Delivery: Reliable
Software Releases through Build, Test, and Deployment 10. Martin Fowler and Facebook: on.fb.me/miBrOM
Automation”, Addison Wesley Professional, 2010
11. Facebook Engineering: on.fb.me/miBrOM
2. Paul Hammant and continuousdelivery.com
12. Paul Julius, Enterprise Continuous Integration Maturity Model,
3. Stephen P. Berczuk and Brad Appleton, “Software bit.ly/m7h5vC
Configuration Management Patterns.”, Addison Wesley
Professional, 2003

4. Mary and Tom Poppendieck, “Leading Lean Software

Development”, Addison Wesley, 2009

5. Paul M. Duvall, “Continuous integration. Patterns and

Antipatterns”, DZone refcard #84, 2010 bit.ly/l8rfVS
DZO N E .CO M/ RE FCA RDZ

Written by Michael Olson , Principal Product Marketing Manager, Puppet

Michael Olson is a Principal Product Marketing Manager at Puppet, where he’s responsible for product launches and
go-to-market strategy for Puppet's products. When he’s not working closely with the product teams at Puppet, you
can find Michael traveling around the world to advise organizations about DevOps and Continuous Delivery practices
and running around on the soccer field.

DZone, Inc.
DZone communities deliver over 6 million pages each 150 Preston Executive Dr. Cary, NC 27513
month to more than 3.3 million software developers, 888.678.0399 919.678.0300
architects and decision makers. DZone offers something for
Copyright © 2018 DZone, Inc. All rights reserved. No part of this publication
everyone, including news, tutorials, cheat sheets, research
may be reproduced, stored in a retrieval system, or transmitted, in any form
guides, feature articles, source code and more. "DZone is a or by means electronic, mechanical, photocopying, or otherwise, without
developer’s dream," says PC Magazine. prior written permission of the publisher.

9 BROUGHT TO YOU IN PARTNERSHIP WITH