84 BROUGHT TO YOU IN PARTNERSHIP WITH

CONTENTS

öö WHAT IS CONTINUOUS
INTEGRATION?

CI Patterns and
öö PATTERNS AND ANTI-PATTERNS

öö BUILD MANAGEMENT

öö BUILD PRACTICES

öö BUILD CONFIGURATION

Anti-Patterns öö

öö
DATABASES

TESTING AND CODE QUALITY

WRITTEN BY PAUL DUVALL, CTO AND CO-FOUNDER, STELLIGENT

UPDATED BY DAVID POSIN, SENIOR SOFTWARE ENGINEER AT SILICON PUBLISHING, INC

WHAT IS CONTINUOUS INTEGRATION? • Production Snapshot - Building from scratch means that the
Conventionally, the term "Continuous Integration" refers to the build on QA is going to be the same on UAT, which is going to
"build and test" cycle. Individual developers merge their code into be the same as production.
DZO N E .CO M/ RE FCA RDZ

one shared project master branch. CI servers rebuild the project

• Early Error Detection - Code that passes all the linting, unit
from scratch every time a code merge happens.
tests, and e-2-e tests can still fail to build for unexpected
Conversations about Continuous Integration tend to couple CI reasons. A build failure is identified immediately after the code
with Continuous Delivery (CD). CD differs from CI by referring to is merged in, making it easy to identify the broken commit(s).
the movement of code from one environment to another, such
• Project Confidence - All of this adds up to increased confidence
as development to QA to User Acceptance Testing. CI servers will
in the product from developers, managers, and customers.
usually do CD as well. The two concepts are inexorably linked since
they work hand in hand. There are other DZone Refcardz that you CI needs to be implemented properly to reap these benefits.
should consider downloading if you wish to understand Continuous Everything starts from the point of change. Every time a change is
Delivery better, such as, "Preparing for Continuous Delivery" or merged into the master branch, then a build job must be run.
"Continuous Delivery: Patterns and Anti-Patterns in the Software
Lifecycle". This Refcard is going to focus specifically on CI patterns,
although there will occasionally be some overlap since the two are
so closely related.

WHY DOES CONTINUOUS INTEGRATION MATTER?

Continuous Integration benefits any organization that implements it Code, collaborate,
correctly. Some of the key benefits are:
secure, and ship.
• Better Quality Code - Code that makes it into the project's Faster
master branch is of a consistent quality. The automated CI
system will perform code checking and code linting as part
45-Day Free Trial
of the merge and build process.

• Better Tested Code - Unit tests, end-to-end tests, and code

coverage reports can be run automatically to ensure all tests
pass and code coverage does not slip.

1
Build and ship software
faster with the largest open
source developer community.

GitHub Enterprise

Code, collaborate,
secure, and ship. Faster.

45-Day Free Trial

 CI PATTERNS AND ANTI-PATTERNS

PATTERNS AND ANTI-PATTERNS • An automated build script or CI server configuration to run

Patterns, and corresponding anti-patterns, are discussed in the when the repository has changed.
following sections. According to Google trends, the most searched • Some sort of feedback mechanism (such as e-mail or chat
for version control tool is Git and the most commonly searched CI software).
software is Jenkins.

Run a software build with every change

PATTERN
applied to the Repository.

Scheduled builds, nightly builds, building

ANTI-PATTERNS periodically, building exclusively on
developer's machines, not building at all.

VERSION CONTROL [1]

Versioning is one of the central pillars of CI. A good version control
system (VCS) will maintain a core functioning codebase. Developers
Google Trends
can then build off that main codebase by creating code branches
to add features, fixes, and patches without affecting anyone else.
Developer branches can then be merged into the main code branch,
called mainline or master, when complete.

Here are some important best practices to consider when working

with a version control system:
DZO N E .CO M/ RE FCA RDZ

• Private Workspace - Developers should be working on

their own machines (real or virtual) with local copies of
the repository. Developers should not be working over file
Google Trends systems that allow them to share the same code files, or on
the machine serving as a repository host (if the repository is
Where examples are provided, we will use the Git Command Line
not cloud-based).
Interface commands or Git Plugin hooks for Jenkins. This Refcard
will use those two technologies for examples based solely on • Repository -- In line with the point above, all code should be

search popularity. hosted in a repository. No successful Continuous Integration

plan will work with a file system hosted project.
BUILD SOFTWARE AT EVERY CHANGE
• Master - The main branch from which builds are run should
One of the most important patterns to remember when talking
be the master, or mainline, branch. This branch should be
about CI is to build the project every time a change is merged into
heavily protected. Developers should be able to merge code
master. Traditionally, the project is built on an arbitrary schedule,
into the master branch but should not be able to commit
such as nightly or every weekend. Scheduled builds are an anti-
code directly. The master branch will host your project's
pattern for CI.
main history and milestone builds. No code should ever be
One of the goals of building after every code merge is to added to it directly.
immediately identify when problems occur. If the newly merged
• Branching Policy - All teams working on a CI process should
code causes the build process to fail, then developers know which
have an agreed-upon branching policy. Developers should
section of code to examine. If builds are scheduled and an error
branch off of master using a naming convention agreed
occurs, the exact code change might not be obvious, and fixing it
upon by the team. Code should be merged into master
could require significant digging.
from individual developer branches through pull requests,
A Continuous Integration system should be set up with: or some other agreed-upon mechanism. Branches should
be pushed to the repository just like master so they can be
• A shared version control repository (i.e. Git). shared if needed.

3 BROUGHT TO YOU IN PARTNERSHIP WITH

 CI PATTERNS AND ANTI-PATTERNS

TASK-LEVEL COMMIT not be expected to kick off CI builds manually to avoid impacting
Most modern version control software integrates this best practice development progress.
into its core functionality. A developer moves code into the VCS
Setting up an automatic build between Git and Jenkins uses the
when they have made enough progress to consider saving it. Usually
following procedure. Inside the project's .git directory, update the
the process involves adding the changed code files to a commit, and
hooks/post-receive file to send the repository to your CI tool (in this
then committing them with a relevant message.
case Jenkins). Your CI tool should be installed with a Git plugin to
For example, in Git, a task-level commit uses the following commands: accept the curl.

git add -A curl http://yourserver/git/notifyCommit?url=<URL of the

git commit -m "message" (i.e., git commit -m "added Git repository>[&[,branch2]*][&sha1=<commit ID>]
file object class")
git push
• Automate all activities to build
software from a source without manual
Organize source code changes by task-
configuration.
PATTERN oriented units of work and submit changes as PATTERN
a Task-Level Commit. • Create build scripts that will be executed
by a CI system so that software is built at
Keeping changes local to development
every change.
for several days and stacking up changes
ANTI-PATTERNS until committing all changes. This often
Continually repeating the same processes
causes build failures or requires complex
troubleshooting. with manual builds or partially automated
ANTI-PATTERNS
builds requiring numerous manual
LABEL BUILD configuration activities.
DZO N E .CO M/ RE FCA RDZ

When master has reached an important development or release

milestone, give it a name to mark that code state. Typically, this will
BUILD PRACTICES
be, or will incorporate, the release version number using semantic
PRE-MERGE BUILD
versioning (major.minor.patch, i.e. 1.10.23). The automated build discussed above occurs on the CI server using
shared resources. VCS systems can be configured to perform a fast,
Git uses the term "tagging" instead of "labeling". To create a tag for
stripped-down build locally first to pre-check the code.
the current build, use the following commands:

git tag -a annotation -m "message" (i.e., git tag -a The procedure in Git works using the project's .git directory. Update
v1.2.12 -m "version 1.2.12") the hooks/pre-push file to run the CLI command for your project's
git push remote --tags build/compilation/bundling tool. For example, a JavaScript or Node
project might use a tool called Gulp:
Tag or Label the build with unique name so
PATTERN that you can refer to run the same build at gulp <my project>

another time.
Verify that your changes will not break the
Not labeling builds, using revisions or
ANTI-PATTERNS integration build by performing a pre-merge
branches as "labels." PATTERN
build---either locally or using Continuous
Integration.
BUILD MANAGEMENT
Code being merged into master should cause the CI server to Checking in changes to a version-control
automatically build the project, including the new changes. There ANTI-PATTERNS repository without running a build on a
should be no developer or DevOps engineer involvement required. developer's workstation.

AUTOMATED BUILD CONTINUOUS FEEDBACK

There should be hooks in the version control system or polling The results of a build are of special importance to the developer
in the CI system to force a new build when master changes. It is submitting the revised code. It is important that they are aware
important that builds happen after every merged code change so a of the results as soon as they are available. Positive and negative
breaking commit can be identified immediately. Developers should results are both important and developers should be trained

4 BROUGHT TO YOU IN PARTNERSHIP WITH

 CI PATTERNS AND ANTI-PATTERNS

to watch for all feedback before moving on to other tasks. The

PATTERN Fix build errors as soon as they occur.
method(s) of providing feedback will vary depending on your
organization's infrastructure. Methods to consider include: Allowing problems to stack up (build entropy)
ANTI-PATTERNS
• Email or waiting for them to be fixed in future builds.

• Hipchat
DEVELOPER DOCUMENTATION
• Slack
The build process is an excellent opportunity to generate
• SMS
documentation for your source code. Developers tend to dislike
• Web Push Notifications
writing documentation manually, and keeping documentation up
• Campfire
to date manually can be time-consuming. The preferred approach is
• Any infrastructure tool with extensions that your
to incorporate documentation into your code, and then having the
organization uses
build process generate documentation from the code. This keeps
documentation up-to-date and does not create more work for the
Send automated feedback from the CI server
development team.
PATTERN to development team members involved in
the build.
Generate developer documentation with
PATTERN
Sending minimal feedback that provides builds based on checked-in source code.
no insight into the build failure or is non-
actionable. Sending too much feedback, Manually generating developer
ANTI-PATTERNS
including to team members uninvolved with documentation. This is both a burdensome

the build. This is eventually treated like spam, ANTI-PATTERNS process and one in which the information

which causes people to ignore messages. becomes useless quickly because it does
DZO N E .CO M/ RE FCA RDZ

not reflect the checked-in source code.

EXPEDITIOUS FIXES
Mistakes will happen and occasionally code will make it into master BUILD CONFIGURATION
that breaks the build. The responsible pattern is to be ready to INDEPENDENT BUILD

handle and resolve the problem immediately. The worst possible Builds should happen the same way on all machines that run them.

way to handle a failure in a build is to ignore it, or expect it to A build on a developer machine should run the same procedure as

be resolved in a future build. You should consider taking all the the CI server. Therefore, train developers to not use the IDE build

following steps: process. Instead, the IDE can be configured to run the required build
scripts so that building can still happen from the IDE. Every project
• Fix Broken Builds Immediately - Although it is the team's should include its own build scripts so it can be built from anywhere
responsibility, the developer who recently committed code it is being worked on.
must be involved in fixing the failed build. It is possible the
problem was a result of a lack of knowledge, so it is a good idea Create build scripts that are decoupled from
to have a seasoned developer available to assist if needed. IDEs, but can be invoked by an IDE. These
PATTERN
• Always Pull Master and Build - Developers should pull the build scripts will be executed by a CI system as

latest code into their branch from master before pushing well so that software is built at every change.

committed code. After pulling master into their own branch,

Relying on IDE settings for Automated Build.
they should run unit tests and build locally to ensure nothing ANTI-PATTERNS
Build cannot run from the command line.
pulled from master breaks their code. This also allows the
developer a chance to fix conflicts that result from the merge
SINGLE COMMAND
before the merge gets to master.
Running a project build should be as simple as possible. It is best
• Don't Pull Broken Code - If master is broken, notify the to have a simple CLI command that can run everything required for
team. Developers should avoid pulling master into their a build in the correct order. This ensures that both developers and
own branch while it is broken. Development time could be servers use the exact same code in the exact same order. A single
wasted by other developers struggling with bad code that command-invoked build script can also be kept up to date with the
will be changed shortly. current state of the project.

5 BROUGHT TO YOU IN PARTNERSHIP WITH

 CI PATTERNS AND ANTI-PATTERNS

Build, compile, and testing phases can be time consuming for a

• Externalize all variable values from the
developer. In order to support the development team, provide flags
application configuration into build-time
on the CLI command to limit the build process to fit their needs. For
PATTERN properties.
example, a developer might be updating a class and only needs
• Use tokens so the build process knows
to compile the code. They are not at a point where they need to
where to add variable values.
test and build the whole project. The script could take a flag, such
as --compileonly, to only perform the compilation process, but this Hardcoding values in configuration files or
ANTI-PATTERNS
way there are not individual commands for developers to know. using GUI tools to do the same.
Everything goes through the same single CLI command.

DATABASE
Ensure all build processes can be run Databases are the cornerstones of all modern software projects. No
PATTERN
through a single command. project of any scale beyond a prototype can function without some
form of database. For this reason, databases should be included in
Requiring people or servers to enter
the Continuous Integration process. They should be treated with the
multiple commands and procedures in the
same extensibility and care as project software code.
deployment process, such as copying files,
ANTI-PATTERNS
modifying configuration files, restarting SCRIPTING DATABASE CHANGES
a server, setting passwords, and other All changes made to a database during development should be
repetitive, error-prone actions. recorded via database scripts. The CI process can then run scripts
as the project is built. It is an anti-pattern to expect any manual
DEDICATED RESOURCES manipulation of a database during the build process. A database for
CI builds of master should be performed on servers (real or virtual) the project should be able to be migrated to new changes regardless
DZO N E .CO M/ RE FCA RDZ

that are only tasked with building the project. These dedicated of timing or platform.
machines should have sufficient resources to build the project
smoothly and swiftly to limit developer downtime. Performing
All changes made to a database during
builds on a dedicated machine ensures a clean environment that
development should be recorded into
doesn't introduce unexpected variables. Clean builds give a certain PATTERN database scripts that can be run on every
degree of reassurance that the project will build successfully when
database on every platform hosting the project
being deployed to other environments.
(including developer machines, see below).

Run master builds on a separate dedicated Expecting database administrators to

PATTERN
machine or cloud service. manually compare databases between
ANTI-PATTERNS platforms for changes, or to create on-off
Relying on existing environmental and
ANTI-PAT- scripts that are only good for updating a
configuration assumptions (can lead to the "but it
TERNS
single platform.
works on my machine problem").

DATABASE SANDBOX
EXTERNALIZE AND TOKENIZE CONFIGURATION Every instance of the project should have its own version of the
Configuration information that is specific to a machine or deployment
database with a relevant set of data. This should include development
environment should be a variable in any build and configuration
machines, build machines, testing machines, testing servers, and QA
scripts. These values should come from the build process so they
machines. No individual or server working with the project should
can be build or environment specific. Files that use this information
have to worry about the integrity of their data, or the integrity of some
should use tokens so that the build process can replace them with
other entity's data, while coding, building, and testing.
actual values. For example, a build might be supplied a hostname,
then any configuration file that needs hostname should use This is true of schema, but not necessarily data. The data for each
$hostname for that value. The build process will go through all config environment should be a subset of the whole, and scrubbed of
files and replace $hostname with the correct value every time the sensitive information. For example, a developer should have
server is built. This lets the build process create the project on as access to a lightweight version of the database with a very small
many different machines as possible. subset of records. However, the data they do have shouldn't have

6 BROUGHT TO YOU IN PARTNERSHIP WITH

 CI PATTERNS AND ANTI-PATTERNS

real social security numbers, addresses, etc. Use discretion and the project. Tests should be as comprehensive as possible and can
ahere to any relevant regulations when deciding what data to use include unit tests, end-to-end tests, smoke tests, or UI tests.
for database population.
Testing should be done on all new code. Tests for new code should
The CI process should include a way to import the data correctly into be a requirement of a successful build. All code being merged
the database. Any data import or manipulation should be scripted into the project should be required to meet an appropriate code
so it can be performed via command line. Developers, testers, and coverage level. All modern test running suites will have some form of
build machines shouldn't have to know the intricacies of the data. coverage reporter that can be tied into the build process. Any code
The build process in particular will need a command line command submitted without sufficient test coverage should fail.
to call during the build.
Write automated tests for each code path,
PATTERN
both success testing and failure testing.
• Create a lightweight version of your
• Not running tests
database (only enough records to test
ANTI-PATTERNS • No regression testing
functionality)
• Manual testing
• Use a command line interface to populate
PATTERN
the local database sandboxes for each
BUILD QUALITY THRESHOLD
developer, tester, and build server The build is also an appropriate time to check for code quality and
• Use this data in development coverage percentages. The project team should have a minimum
environments to expedite test execution coverage percentage that the project is not allowed to dip below. If
new code is merged in without sufficient tests, that percentage will
ANTI-PATTERNS Sharing development database.
be lower than expected, and that should trigger a failure.
DZO N E .CO M/ RE FCA RDZ

UPDATE SCRIPTS STORED IN THE VCS Code quality is important to the long-term maintainability of a
All scripts to perform database and data operations should be stored project, and the build step is a great place to verify code quality. The
in the version control system being used by the codebase. Scripts project team should have standards and best practices for the code
should be named and/or annotated to refer to their appropriate so most of it looks and works the same way. The build step should
version number to simplify automation. Keeping old versions is verify that new code meets those standards.
useful for doing multiple scripts if necessary, and to maintain a
history of changes. • Notify team members of code aberrations
such as low code coverage or the use of
Store the scripts for updating the database coding anti-patterns.
PATTERN
PATTERN and its data in the version control system with • Fail a build when a project rule is violated.
the code and annotate appropriately • Use continuous feedback mechanisms to
notify team members.
Storing update scripts in an alternative
ANTI-PATTERNS
location, i.e. a shared file server • Deep dive reviews of every code change.
ANTI-PATTERNS • Manually calculating or guesstimating

TESTING AND CODE QUALITY code coverage.

The sometimes rapid pace of Continuous Integration can feel
daunting in the beginning, but it becomes comfortable very quickly. AUTOMATED SMOKE TEST
One of the reasons for that is testing. Testing and code quality Smoke tests are a subset of tests used to confirm the functionality
validation are a massively important part of CI. Tests running with of the most important elements of a project. They function as
every build ensures that nothing has been broken and that the code gatekeeper to confirm that building, full testing, or QA can continue.
is maintaining superior quality. CI cannot be successful without a A suite of well-designed smoke tests can save QA personnel time
robust testing method. Without testing, CI can become chaotic. and effort by checking the most likely candidates for failure first.

AUTOMATED TESTS This is also useful for Continuous Deployment. Smoke tests can be
Tests should run with every build. The build scripts described in designed to check functionality most sensitive to changes in the
the sections above should include running tests against all code in environment. It is an easy way to confirm that the deployment will work.

7 BROUGHT TO YOU IN PARTNERSHIP WITH

 CI PATTERNS AND ANTI-PATTERNS

Create smoke tests that can be used by CI

servers, developers, QA, and testing as a
PATTERN pre-check to confirm the most important
functionality as they work, or before
committing resources to a full build.

• Manually running functional tests.

• Forcing QA to run the full suite before
ANTI-PATTERNS every session
• Manually checking deployment sensitive
sections of the project

[1]
Addison-Wesley, Software Configuration Management Patterns,
2003, Berczuk and Appleton.

Written by David Posin, Senior Software Engineer at Silicon Publishing, Inc.

David Posin has been involved in the Information Technology Industry for two decades. Fifteen years of that time
was spent consulting with many companies in a wide range of industries to build solid technology stacks and robust
application architectures. David has watched the Cloud and the World Wide Web grow from their infancy, and now
spends every day fully entrenched in those worlds. Currently, David builds high-performance web applications and
offers professional technical writing services.
DZO N E .CO M/ RE FCA RDZ

DZone, Inc.
DZone communities deliver over 6 million pages each 150 Preston Executive Dr. Cary, NC 27513
month to more than 3.3 million software developers, 888.678.0399 919.678.0300
architects and decision makers. DZone offers something for
Copyright © 2018 DZone, Inc. All rights reserved. No part of this publication
everyone, including news, tutorials, cheat sheets, research
may be reproduced, stored in a retrieval system, or transmitted, in any form
guides, feature articles, source code and more. "DZone is a or by means electronic, mechanical, photocopying, or otherwise, without
developer’s dream," says PC Magazine. prior written permission of the publisher.

8 BROUGHT TO YOU IN PARTNERSHIP WITH