brought to you by...

#145
Get More Refcardz! Visit refcardz.com

CONTENTS INCLUDE:
n

n
The Deployment Pipeline
Benefits
Continuous Delivery
n
Configuration Management
Continuous Integration
n
Patterns and Antipatterns
n

n
Testing
Hot Tips and more...
in the Software Lifecycle
By Paul M. Duvall
•F
 eedback: Team members learn of problems as soon as they
ABOUT CONTINUOUS DELIVERY occur so that issues are fixed as soon as possible.
•C
 ontinually Deploy: Through a fully automated process, you
With Continuous Delivery (CD), teams continuously deliver new
can deploy and release any version of the software to any
versions of software to production by decreasing the cycle time
environment. (1)
between an idea and usable software through the automation of
the entire delivery system: build, deployment, test, and release. CD In the Deployment Pipeline diagram above, all of the patterns are
is enabled through the Deployment Pipeline, which encompasses a shown in context. There are some patterns that span multiple stages
collection of patterns described in this Refcard. of the pipeline, so I chose the stage where it’s most predominately
used.
CD is concerned with “…how all the moving parts fit together:
configuration management, automated testing, continuous
integration and deployment, data management, environment BENEFITS
management, and release management.” (1)
•E
 mpowering Teams: Because the deployment pipeline is a
THE DEPLOYMENT PIPELINE pull system, testers, developers, operations, and others can
self service the application version into an environment of their
choice.
•R
 educing Errors: Ensuring the correct version, configuration,
database schema, etc. are applied the same way every time
through automation.
•L
 owering Stress: Through push-button releases to production
and Rehearsing Deployments, a release becomes commonplace
without the typical stress.
•D
 eployment Flexibility: Instantiate a new environment or
configuration by making a few changes to the automated
delivery system.
• Practice makes Perfect: Through the deployment pipeline, the
final deployment into production is being rehearsed every single
time the software is deployed to any target environments. (1)
Continuous Delivery

The purpose of the deployment pipeline is threefold:

•V
 isibility: All aspects of the delivery system - building,
deploying, testing, and releasing – are visible to all team
members promoting collaboration.

DZone, Inc. | www.dzone.com

 2 Continuous Delivery: Patterns and Antipatterns in the Software Life Cycle

Short-Lived Branches (1)

CONFIGURATION MANAGEMENT Pattern Branches must be short lived – ideally less than a few
days and never more than an iteration.
Configuration Management is “the process by which all artifacts Anti-patterns Branches that last more than an iteration. Branches by
relevant to your project, and the relationships between them, are product feature that live past a release.
stored, retrieved, uniquely identified, and modified”. (1)

Note: Each pattern is cited with a number in parentheses that Single Command Environment (1)
corresponds to the source in the References section. Pattern Check out the project’s version-control repository
and run a single command to build and deploy the
application to any accessible environment, including
Configurable Third-Party Software (1) the local development.
Pattern Evaluate and use third-party software that can be Anti-patterns Forcing the developer to define and configure
easily configured, deployed, and automated. environment variables. Making the developer install
numerous tools in order for the build/deployment to
Anti-patterns Procuring software that cannot be externally
work.
configured. Software without an API or command-
line interface that forces teams to use the GUI only.
Single Path to Production (1)
Configuration Catalog (1) Pattern Configuration management of the entire system -
Pattern Maintain a catalog of all options for each application, source, configuration, environment and data. Any
how to change these options and storage locations for change can be tied back to a single revision in the
each application. Automatically create this catalog as version-control system.
part of the build process. Anti-patterns Parts of system are not versioned. Inability to get back
Anti-patterns Configuration options are not documented. The to a previously configured software system.
catalog of applications and other assets is “tribal
knowledge”.

CONTINUOUS INTEGRATION (CI)

Mainline (3)
Pattern Minimize merging and keep the number of active code Build Threshold (5)
lines manageable by developing on a mainline. Pattern Fail a build when a project rule is violated – such
as architectural breaches, slow tests, and coding
Anti-patterns Multiple branches per project.
standard violations.
Anti-patterns Manual code reviews. Learning of code quality
Merge Daily (1) issues later in the development cycle.
Pattern Changes committed to the mainline are applied to
each branch on at least a daily basis.
Commit Often (6)
Anti-patterns Merging every iteration once a week or less often
Pattern Each team member checks in regularly to trunk - at
than once a day.
least once a day but preferably after each task to
trigger the CI system.
Protected Configuration (5) ,(1) Anti-patterns Source files are committed less frequently than daily
Pattern Store configuration information in secure remotely due to the number of changes from the developer.
accessible locations such as a database, directory, or
registry. Continuous Feedback (6)
Anti-patterns Open text passwords and/or single machine or share. Pattern Send automated feedback from CI system to all
Cross-Functional Team members.
Anti-patterns Notifications are not sent; notifications are ignored; CI
system spams everyone with information they cannot
use.
Repository (3) , (1)
Pattern All source files - executable code, configuration, host
environment, and data - are committed to a version- Continuous Integration (6)
control repository. Pattern Building and testing software with every change
Anti-patterns Some files are checked in, others, such as committed to a project’s version control repository.
environment configuration or data changes, are not. Anti-patterns Scheduled builds, nightly builds, building
Binaries – that can be recreated through the build periodically, building exclusively on developer’s
and deployment process – are checked in. machines, not building at all.

DZone, Inc. | www.dzone.com

 3 Continuous Delivery: Patterns and Antipatterns in the Software Life Cycle

Stop the Line (5) , (1) , (4), (12) Parallel Tests (1)
Pattern Fix software delivery errors as soon as they occur; Pattern Run multiple tests in parallel across hardware instances
stop the line. No one checks in on a broken build as to decrease the time in running tests.
the fix becomes the highest priority. Anti-patterns Running tests on one machine or instance. Running
Anti-patterns Builds stay broken for long periods of time, dependent tests that cannot be run in parallel.
thus preventing developers from checking out
functioning code. Stub Systems (1)
Pattern Use stubs to simulate external systems to reduce
Independent Build (6) deployment complexity.
Pattern Write build scripts that are decoupled from IDEs. Anti-patterns Manually installing and configuring interdependent
These build scripts are executed by a CI system so that systems for Commit Stage build and deployment.
software is built at every change.
Anti-patterns Automated build relies on IDE settings. Builds are
unable to be run from the command line. DEPLOYMENT PIPELINE

Deployment Pipeline (1)

Visible Dashboards
Pattern A deployment pipeline is an automated
Pattern Provide large visible displays that aggregate
implementation of your application’s build, deploy,
information from your delivery system to provide high-
test, and release process.
quality feedback to the Cross-Functional Team in real
time. Anti-patterns Deployments require human intervention (other
than approval or clicking a button). Deployments are
Anti-patterns Email-only alerts or not publicizing the feedback to
not production ready.
the entire team.

Value-Stream Map (4)

Pattern Create a map illustrating the process from check in to
TESTING
the version-control system to the software release to
identify process bottlenecks.
Automate Tests Anti-patterns Separately defined processes and views of the
Pattern Automate the verification and validation of software checkin to release process.
to include unit, component, capacity, functional, and
deployment tests
Anti-patterns Manual testing of units, components, deployment, and
other types of tests. BUILD AND DEPLOYMENT SCRIPTING

Dependency Management (5)

Unit- Automating tests without any dependencies. Pattern Centralize all dependent libraries to reduce bloat,
Component- Automating tests with dependencies to other classpath problems, and repetition of the same
components and heavyweight dependencies such as the database dependent libraries and transitive dependencies
or file system. from project to project.

Deployment- Automating tests to verify the deployment and Anti-patterns Multiple copies of the same binary dependencies
configuration were successful. Sometimes referred to as a “smoke in each and every project. Redefining the same
tests”. information for each project. Classpath hell!

Functional- Automating tests to verify the behavior of the software

Common Language (1)
from a user’s perspective.
Pattern As a team, agree upon a common scripting
Capacity- Automating load and performance testing in near- language - such as Perl, Ruby, or Python - so that
production conditions. any team member can apply changes to the Single
Delivery System.
Isolate Test Data (1) Anti-patterns Each team uses a different language making it
Pattern Use transactions for database-dependent tests (e.g., difficult for anyone to modify the delivery system
component tests) and roll back the transaction when reducing cross-functional team effectiveness.
done. Use a small subset of data to effectively test
behavior. Externalize Configuration (5)
Anti-patterns Using a copy of production data for Commit Stage Pattern Changes between environments are captured as
tests. Running tests against a shared database. configuration information. All variable values are
externalized from the application configuration into
build/deployment-time properties.
Anti-patterns Hardcoding values inside the source code or per
target environment.

DZone, Inc. | www.dzone.com

 4 Continuous Delivery: Patterns and Antipatterns in the Software Life Cycle

Blue-Green Deployments (1)

Fail Fast (6) Pattern Deploy software to a non-production environment
Pattern Fail the build as soon as possible. Design scripts so (call it blue) while production continues to run. Once
that processes that commonly fail run first. These it’s deployed and “warmed up”, switch production
processes should be run as part of the Commit (green) to non-production and blue to green
Stage. simultaneously.
Anti-patterns Common build mistakes are not uncovered until late in Anti-patterns Production is taken down while the new release is
the deployment process. applied to production instance(s).

Fast Builds (6)

Pattern The Commit Build provides feedback on common
build problems as quickly as possible - usually in under
10 minutes.
Anti-patterns Throwing everything into the commit stage
process, such as running every type of automated
static analysis tool or running load tests such that
feedback is delayed.
Dark Launching (11)
Pattern Launch a new application or features when it affects
Scripted Deployment (5) the least amount of users.
Pattern All deployment processes are written in a script, Anti-patterns Software is deployed regardless of number of active
checked in to the version-control system, and run as users.
part of the Single Delivery System.
Anti-patterns Deployment documentation is used instead of Rollback Release (5)
automation. Manual deployments or partially
Pattern Provide an automated single command rollback of
manual deployments. Using GUI to perform a
changes after an unsuccessful deployment.
deployment.
Anti-patterns Manually undoing changes applied in a recent
deployment. Shutting down production instances
Unified Deployment (5) while changes are undone.
Pattern The same deployment script is used for each
deployment. The Protected Configuration – per
environment - is variable but managed. Self-Service Deployment (1)
Pattern Any Cross-Functional Team member selects the
Anti-patterns Different deployment script for each target version and environment to deploy the latest
environment or even for a specific machine. Manual working software.
configuration after deployment for each target Anti-patterns Deployments released to team are at specified intervals
environment. by the “Build Team”. Testing can only be performed in
a shared state without isolation from others.

DEPLOYING AND RELEASING APPLICATIONS

INFRASTRUCTURE AND ENVIRONMENTS

Binary Integrity (5)

Pattern Build your binaries once, while deploying the Automate Provisioning (1)
binaries to multiple target environments, as Pattern Automate the process of configuring your
necessary. environment to include networks, external services,
and infrastructure.
Anti-patterns Software is built in every stage of the deployment
pipeline.
Anti-patterns Configured instances are “works of art” requiring team
members to perform partially or fully manual steps to
Canary Release provision them.
Pattern Release software to production for a small subset
of users (e.g. , 10%) to get feedback prior to a
complete rollout. Behavior-Driven Monitoring (1)
Pattern Automate tests to verify the behavior of the
Anti-patterns Software is released to all users at once.
infrastructure. Continually run these tests to provide
near real-time alerting.
Anti-patterns No real-time alerting or monitoring. System
configuration is written without tests.

DZone, Inc. | www.dzone.com

 5 Continuous Delivery: Patterns and Antipatterns in the Software Life Cycle

Immune System (9) Scripted Database (7)

Pattern Deploy software one instance at a time while Pattern Script all database actions as part of the build
conducting Behavior-Driven Monitoring. If an error process.
is detected during the incremental deployment, a
Rollback Release is initiated to revert changes.
Anti-patterns Using data export/import to apply data changes.
Manually applying schema and data changes to the
Anti-patterns Non-incremental deployments without monitoring. database.

Lockdown Environments (1) INCREMENTAL DEVELOPMENT

Pattern Lock down shared environments from unauthorized
external and internal usage, including operations Branch by Abstraction (2)
staff. All changes are versioned and applied through Pattern Instead of using version-control branches, create an
automation. abstraction layer that handles both an old and new
Anti-patterns The “Wild West”: any authorized user can access implementation. Remove the old implementation.
shared environments and apply manual configuration Anti-patterns Branching using the version-control system leading
changes, putting the environment in an unknown state to branch proliferation and difficult merging. Feature
leading to deployment errors. branching.

Production-Like Environments (1) Toggle Features (10)

Pattern Target environments are as similar to production as
Pattern Deploy new features or services to production but limit
possible. access dynamically for testing purposes.
Anti-patterns Environments are “production like” only weeks or days
before a release. Environments are manually configured
Anti-patterns Waiting until a feature is fully complete before
and controlled. committing the source code.

Transient Environments COLLABORATION

Pattern Utilizing the Automate Provisioning, Scripted
Deployment and Scripted Database patterns, any Delivery Retrospective (1)
environment should be capable of terminating and
Pattern For each iteration, hold a retrospective meeting
launching at will.
where everybody on the Cross-Functional Team
Anti-patterns Environments are fixed to “DEV, QA” or other pre- discusses how to improve the delivery process for
determined environments. the next iteration.
Anti-patterns Waiting until an error occurs during a deployment for
DATA Dev and Ops to collaborate. Having Dev and Ops work
separately.
Database Sandbox (7)
Pattern Create a lightweight version of your database – using Cross-Functional Teams (1)
the Isolate Test Data pattern. Each developer uses Pattern Everybody is responsible for the delivery process.
this lightweight DML to populate his local database Any person on the Cross-Functional Team can
sandboxes to expedite test execution. modify any part of the delivery system.
Anti-patterns Shared database. Developers and testers are Anti-patterns Siloed teams: Development, Testing, and
unable to make data changes without it potentially Operations have their own scripts and processes
adversely affecting other team members and are not part of the same team.
immediately.
Amazon.com has an interesting take on this approach. They call
Decouple Database (1) it “You build it, you run it”. Developers take the software they’ve
Pattern Ensure your application is backward and forward written all the way to production.
compatible with your database so you can deploy each
independently Root-Cause Analysis (1)
Anti-patterns Application code data are not capable of being Pattern Learn the root cause of a delivery problem by
deployed separately. asking “why” of each answer and symptom until
discovering the root cause.
Anti-patterns Accepting the symptom as the root cause of the
problem.
Database Upgrade (7)
Pattern Use scripts to apply incremental changes in each TOOLS
target environment to a database schema and data.
Anti-patterns Manually applying database and data changes in This is meant to be an illustrative list, not an exhaustive list, to give
each target environment. you an idea of the types of tools and some of the vendors that help
to enable effective Continuous Delivery. The Java, .NET and Ruby
platforms are represented. The tools that span categories have been

DZone, Inc. | www.dzone.com

 6 Continuous Delivery: Patterns and Antipatterns in the Software Life Cycle

assigned to the most appropriate category or duplicated when Components and Ivy, Archiva, Nexus, Artifactory, Bundler
necessary. Dependencies
Category Example Software Tools Collaboration Mingle, Greenhopper, JIRA
Configuration Subversion (SVN), git, Perforce, PassPack,
Management PasswordSafe, ESCAPE, ConfigGen

Continuous Integration Bamboo, Jenkins, AntHill Pro, Go, TeamCity, REFERENCES

TFS 2010, Electric Commander. Supporting
tools: , Doxygen, Grand, GraphViz, JavaDoc, 1. Jez Humble and David Farley, “Continuous Delivery: Reliable Software Releases through Build,
NDoc, SchemaSpy, UmlGraph, CheckStyle, Test, and Deployment Automation”, Addison Wesley Professional, 2010
Clover, Cobertura, FindBugs, FxCop, JavaNCSS, 2. Paul Hammant and www.continuousdelivery.com
JDepend, PMD, Sonar, Simian
3. Stephen P. Berczuk and Brad Appleton, “Software Configuration Management Patterns.”,
Testing Twist , AntUnit, Cucumber, DbUnit, webrat, Addison Wesley Professional, 2003
easyb, Fitnesse, JMeter, JUnit, NBehave, SoapUI,
Selenium, RSpec, SauceLabs 4. Mary and Tom Poppendieck, “Leading Lean Software Development”, Addison Wesley, 2009

Deployment Pipeline Go, AntHill Pro 5. Paul M. Duvall, “Continuous integration. Patterns and Antipatterns”, DZone refcard #84, 2010
http://bit.ly/l8rfVS
Build and Deployment Ant, AntContrib, NAnt, MSBuild, Buildr, Gant,
6. Paul M. Duvall, “Continuous integration. Improving Software Quality and Reducing Risk”,
Scripting Gradle, make, Maven, Rake, Java Secure Channel,
Addison Wesley, 2007
ControlTier, Altiris, Capistrano, Fabric, Func
7. Scott W. Ambler and Pramodkumar J. Saladage, “Refactoring Databases. Evolutionary
Infrastructure and AWS EC2, AWS S3, Windows Azure, Google Database Design”, Addison Wesley, 2006.
Environments App Engine, AWS Elastic Beanstalk, Heroku,
Capistrano, Cobbler, BMC Bladelogic, CFEngine, 8. Paul M. Duvall, IBM developerWorks series “Automation for the people” http://ibm.co/iwwvPX
IBM Tivoli Provisioning Manager, Puppet, Chef,
9. IMVU: http://bit.ly/jhqP5f
Bcfg2, AWS Cloud Formation, Windows Azure
AppFabric, rPath, JEOS, BoxGrinder, CLIP, 10. Martin Fowler and Facebook: http://on.fb.me/miBrOM
Eucalyptus, AppLogic, CloudKick, CloudWatch,
11. Facebook Engineering: http://on.fb.me/miBrOM
Nagios, Zabbix, Zenoss
12. Paul Julius, Enterprise Continuous Integration Maturity Model, http://bit.ly/m7h5vC
Data Hibernate, MySQL, Liquibase, Oracle, PostgreSQL,
SQL Server, SimpleDB, SQL Azure, Ant, MongoDB,
dbdeploy

ABOUT THE AUTHOR RECOMMENDED BOOK

Paul M. Duvall is the CTO of Stelligent, (http://stelligent. Continuous Delivery: Reliable Software Releases
com/) a company that helps organizations with Continuous through Build, Test, and Deployment Automation
Integration and Delivery, Cloud Migrations and Elastic
Operations. Paul is the author of many books including the The book has two themes: automation and
Jolt-award winning book Continuous Integration: Improving collaboration. Delivering software of any complexity
Software Quality and Reducing Risk (http://amzn.to/cibook) and the recently involves people with a bunch of skills: testers,
published Startup@Cloud: 33 tips for running your company using cloud-based developers, and sysadmin / operations personnel.
software (http://amzn.to/startupatcloud). He is passionate about the cloud and
The last group of people often gets left out of the
process of delivering software until the end, and even testers are
automation and blogs at http://devopscloud.com/.
often not as heavily involved in the development process as they
should be.

DZone, Inc.
140 Preston Executive Dr.
Suite 100
Cary, NC 27513
DZone communities deliver over 6 million pages each month to 888.678.0399
more than 3.3 million software developers, architects and decision 919.678.0300
makers. DZone offers something for everyone, including news,
Refcardz Feedback Welcome
$7.95

tutorials, cheat sheets, blogs, feature articles, source code and more.
refcardz@dzone.com
“DZone is a developer’s dream,” says PC Magazine.
Sponsorship Opportunities
Copyright © 2011 DZone, Inc. All rights reserved. No part of this publication may be reproduced, stored in a
retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, sales@dzone.com Version 1.0
without prior written permission of the publisher.