Getting Started with Terraform - Second Edition

Getting Started with Terraform

Second Edition

Manage production infrastructure as a code

Kirill Shirinkin

BIRMINGHAM - MUMBAI

< html PUBLIC -//W3C//DTD HTML 4.0 Transitional//EN http://www.w3.org/TR/REC-html40/loose.dtd>

Getting Started with Terraform

Second Edition

Copyright © 2017 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: January 2017

Second edition: July 2017

Production reference: 1280717

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-78862-353-7

www.packtpub.com

Credits

About the Author

Kirill Shirinkin is an IT consultant who focuses on Cloud technologies and DevOps practices. He has worked in companies of different sizes and areas, from an online language learning leader to a major IT provider for the global travel industry and one of the largest management consultancies. He is also a cofounder of online mentorship platform mkdev.me, where he leads a team and teaches his students all about DevOps.

About the Reviewer

Anton Babenko is currently working as a senior automation engineer at Stelligent Systems AB, where he specializes in infrastructure management and deployment using Amazon Web Services. He is an AWS certified professional with all five available certifications. Also, he has been working as a web developer, team lead, and chief technology officer for the last 10 years. He has been constantly involved in automation (from testing to marketing) and exploring ways to do it properly and as risk-free as possible. He has strong interest and experience in the DevOps toolset.

www.PacktPub.com

For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www.packtpub.com/mapt

Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Customer Feedback

Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at link.

If you'd like to join our team of regular reviewers, you can e-mail us at customerreviews@packtpub.com. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!

Table of Contents

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

Infrastructure Automation

What is Infrastructure as Code and why is it needed?

Declarative versus procedural tools for Infrastructure as Code

Infrastructure as Code in the Cloud

Requirements for infrastructure provisioner

Supports a wide variety of services

Idempotency

Dependency resolution

Robust integration with existing tools

Platform agnosticism

Smart update management

Ease of extension

Which tools exist for infrastructure provisioning?

Scripting

Configuration management

CloudFormation/Heat

Terraform

A short overview of Terraform

Journey ahead and how to read this book

Summary

Deploying First Server

History of Terraform

Preparing work environment

The many Terraform providers

Short introduction to AWS

Using Elastic Compute Cloud

Creating an instance through the Management Console

Creating an instance with AWS CLI

Configuring AWS provider

Static credentials

Environment variables

Credentials file

Creating an EC2 instance with Terraform

Working with state

Handling resource updates

Destroying everything we've built

Summary

Resource Dependencies and Modules

Creating an AWS Virtual Private Cloud

Understanding dependency graph

Playing with Terraform graphs

Controlling dependencies with depends_on and ignore_changes

Making sense of our template

Removing duplication with modules

Configuring modules

Retrieving module data with outputs

Using root module outputs

Summary

Storing and Supplying Configuration

Understanding variables

Using map variables

Using list variables

Supplying variables inline

Using Terraform environment variables

Using variable files

Configuring data sources

Providing configuration with template_file

Providing data from anywhere with external_data

Exploring Terraform configuration resources

Taking a quick look at Consul

Summary

Connecting with Other Tools

Returning data with outputs

Testing servers with Inspec

Provisioners

Provisioning with local-exec and Ansible

Provisioning with Chef

Provisioning with remote-exec and Puppet

Uploading files with a file provisioner

Reprovisioning machines with null_resource

Using third-party plugins

Summary

Scaling and Updating Infrastructure

Counting servers

Bringing in high availability

Load balancing and simulating conditionals

Immutable infrastructure

Baking images with Packer

Rolling out AMI upgrades with Terraform

Performing blue-green deployments

Refreshing infrastructure

Importing resources

Summary

Collaborative Infrastructure

Version control with Git 101

Moving templates to Git

Protecting secrets in a Git repository

Storing state files remotely

Connecting remote states together

Storing modules remotely

Locking state files with Terragrunt

Moving infrastructure updates to the CI pipeline

Integration testing of Terraform modules

Summary

Future of Terraform

Infrastructure as code and Terraform replacements

Learning AWS and compiling Terraform

Learning Consul

Provisioning and configuration management

Immutable infrastructure

Collaboration and CI/CD

The many tools around Terraform

The rapid development of Terraform

Closing thoughts on the future of Terraform

Summary

Preface

With ever-rising adoption of Cloud technologies and infrastructure SaaS products, as well as always the constantly sizes of infrastructures the need, to manage it all in the form of code becomes more and more apparent. Cloud providers such as Amazon Web Services have dozens of services and all of them require secure, re-usable and predictable configuration. Terraform, the primary tool for this job, appeared in 2014 and quickly gained popularity among system administrators and software developers. Since the first release, Terraform has achieved a lot of traction. It became the new de facto tool for managing the cloud environments. Terraform is also a tool that is quite new, that is changing with every release and that requires a new mindset and new practices from teams that adopt it.

In this book you will learn how Terraform works and how to use it, with many examples of real-life applications of it. You will explore modern approaches to managing the infrastructure, such as Infrastructure as Code and Immutable Infrastructure. You will also learn many new small utilities that either improve the experience of working with Terraform or cover the layers that Terraform is not supposed to manage. By the end of this book not only will you now how to use Terraform, but you will be in an expert in treating your whole Infrastructure as Code, with Terraform being the core of this procedure.

What this book covers

Chapter 1, Infrastructure Automation, covers infrastructure automation in general, why is it needed at all (with a list of the main reasons to do it) and which tools exist to solve this problem. By the end of this chapter you will know which problem Terraform solves and why it is the best tool for particular infrastructure automation tasks.

Chapter 2, Deploying First Server, walks through all the necessary steps to install Terraform, gives a short overview of AWS and EC2, and explain in detail how to create your very first EC2 instance with Terraform.

Chapter 3, Resource Dependencies and Modules, explains one of most important features of Terraform: dependency graph. You will figure out how dependencies work and see it in practice by extending the template from previous chapter. At the moment we find out our template is too big, we will use Terraform modules to DRY our code and also use more advanced dependency features.

Chapter 4, Storing and Supplying Configuration, teaches how to make Terraform templates more configurable. You will see all the possible ways to supply data to Terraform templates, to basic variables to using any external data source.

Chapter 5, Connecting with Other Tools, talks about how you can connect Terraform templates to external tools. It shows how to combine Terraform and Ansible, Puppet, or Chef, how to provision servers, and how to run Inspec tests against them.

Chapter 6, Scaling and Updating Infrastructure, dives deep into managing existing infrastructures with Terraform. It gives an overview of the various ways to perform updates with Terraform and explains what Immutable Infrastructure is and how to use it with Terraform. It gives a full example of performing both rolling updates and blue-green deployments, as well as tricks on running smaller updates.

Chapter 7, Collaborative Infrastructure, provides best practices of using Terraform in a team. It shows how to refactor and split Terraform templates into remote modules, how to organize your code to be re-usable, and how to handle sensitive data inside Terraform templates. It also teaches how to do full Continuous Integration of a Terraform-based infrastructure.

Chapter 8, Future of Terraform, speculates on the future of Terraform. It also recaps everything learned so far and gives some extra thoughts and hints on topics that were too small too deserve a separate chapter.

What you need for this book

This book assumes a basic level of understanding the Linux operating system. The book will go through configuring numerous AWS resources. Being familiar with AWS is a plus, but is not required, as all required services will be explained. Usage of some cloud services in this book will require you to spend a dollar or two on them. Although the book assumes Linux as the primary workstation operating system, all of the content applies to MacOS and most of it will work the same way on Windows as well.

Internet connectivity is required to install the necessary tools, including Terraform. It is also required to perform any Terraform operations.

Who this book is for

This book is essentially intended to both software developers and system administrators, as well as specialists who have knowledge of both areas: system reliability engineers, DevOps engineers, cloud architects and so on.

Conventions

In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: For some reason, instead of using DNS server, you want to hardcode the IP address of this box to the /etc/hosts file with a domain name repository.internal.

A block of code is set as follows:

host { 'repository.internal':

  ip => '192.168.0.5',

}

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

resource null_resource app_server_provisioner {

  triggers {

    server_id = ${join(,, aws_instance.app-server.*.id)}

  }

connection {     user = centos     host = ${element(aws_instance.app-server.*.public_ip, count.index)}   }

  provisioner file {

    source = ${path.module}/setup.pp

    destination = /tmp/setup.pp

  }

Any command-line input or output is written as follows:

$> curl -O https://releases.hashicorp.com/terraform/0.8.2/terraform_0.8.2_linux_amd64.zip $> sudo unzip terraform_0.8.2_linux_amd64.zip -d /usr/local/bin/

New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: Click on Launch Instance.

Warnings or important notes appear in a box like this.

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply e-mail feedback@packtpub.com, and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

You can download the code files by following these steps:

Log in or register to our website using your e-mail address and password.

Hover the mouse pointer on the SUPPORT tab at the top.

Click on Code Downloads & Errata.

Enter the name of the book in the Search box.

Select the book for which you're looking to download the code files.

Choose from the drop-down menu where you purchased this book from.

Click on Code Download.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR / 7-Zip for Windows

Zipeg / iZip / UnRarX for Mac

7-Zip / PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Getting-Started-with-Terraform-Second-Edition. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Downloading the color images of this book

We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from https://www.packtpub.com/sites/default/files/downloads/GettingStartedwithTerraformSecondEdition_ColorImages.pdf.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.

To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will