sbt in Action: The simple Scala build tool

Copyright

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact

     Special Sales Department

     Manning Publications Co.

     20 Baldwin Road

     PO Box 761

     Shelter Island, NY 11964

     Email: 

orders@manning.com

©2016 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

ISBN 9781617291272

Printed in the United States of America

1 2 3 4 5 6 7 8 9 10 – EBM – 20 19 18 17 16 15

Brief Table of Contents

Copyright

Brief Table of Contents

Table of Contents

Preface

Acknowledgments

About this Book

About the Cover Illustration

1. Why SBT?

Chapter 1. Why sbt?

Chapter 2. Getting started

2. Understanding sbt’s core concepts

Chapter 3. Core concepts

Chapter 4. The default build

3. Working with sbt

Chapter 5. Testing

Chapter 6. The IO and Process libraries

Chapter 7. Accepting user input

Chapter 8. Using plugins and external libraries

Chapter 9. Debugging your build

4. Extending sbt

Chapter 10. Automating workflows with commands

Chapter 11. Defining a plugin

5. Deploying your projects

Chapter 12. Distributing your projects

Appendix A. What you need to know about Ivy

Appendix B. Migrating to autoplugins

Appendix C. Advanced setting/task API

Index

List of Figures

List of Tables

List of Listings

Table of Contents

Copyright

Brief Table of Contents

Table of Contents

Preface

Acknowledgments

About this Book

About the Cover Illustration

1. Why SBT?

Chapter 1. Why sbt?

1.1. Why a build tool?

1.2. Apache Ant

1.3. Apache Maven

1.3.1. Build definition

1.3.2. Maven workflow—default lifecycle

1.3.3. Adding another task to a Maven build

1.3.4. Advantages of Maven

1.3.5. Limitations of Maven

1.4. Gradle

1.4.1. What is Gradle?

1.4.2. Gradle workflow

1.4.3. Adding another task to a Gradle build

1.4.4. Advantages of Gradle

1.4.5. Limitations of Gradle

1.5. Why sbt?

1.5.1. How sbt works

1.5.2. Plugin architecture vs. tasks

1.5.3. Phases vs. task dependencies

1.5.4. Passing information around your build

1.5.5. Working with Scala

1.5.6. Multiproject builds

1.5.7. Dependency resolution

1.6. Interactivity/reactivity of sbt

1.6.1. Quicker compilation and testing cycle

1.6.2. Quicker compilation and testing cycle

1.6.3. sbt and the Scala REPL—the console command

1.7. Summary

Chapter 2. Getting started

2.1. Setting up sbt

2.1.1. Setting the PATH in bash (for Linux/Mac)

2.1.2. Running sbt

2.2. Setting up a build

2.2.1. Tasks

2.2.2. Settings

2.3. Running code

src/main/scala/models.scala

2.4. Testing code

2.4.1. Running tasks when sources change

2.4.2. Selecting tests with interactive tasks

2.5. Summary

2. Understanding sbt’s core concepts

Chapter 3. Core concepts

3.1. Creating builds

3.2. Defining settings

3.2.1. Initializations

3.3. Creating and executing tasks

3.3.1. Task dependencies

3.4. Using configurations

3.5. Defining with subprojects

3.6. Putting it all together

3.7. Summary

Chapter 4. The default build

4.1. Compiling your code

4.2. Finding your sources

4.2.1. Standard organization of sources

4.2.2. Testing sources

4.2.3. Custom organization of sources

4.2.4. Filtering the source you want

4.3. Depending on libraries

4.3.1. Unmanaged dependencies

4.3.2. Managed dependencies

4.3.3. Managed dependencies and configurations

4.4. Packaging your project

4.4.1. Identifying your project

4.5. Summary

3. Working with sbt

Chapter 5. Testing

5.1. Configuring specs2 with sbt

5.1.1. Reports and forking tests

5.1.2. Digging deeper: other options for forking

5.2. JUnit and using custom code

5.2.1. Report generation with JUnit

5.3. ScalaCheck

5.4. Integration testing

5.4.1. ScalaTest and Selenium

5.4.2. Challenges of integration testing

5.4.3. Adding integration tests to sbt

5.5. Summary

Chapter 6. The IO and Process libraries

6.1. Packaging using processes

6.2. Packaging using the sbt.IO library

6.3. More mappings

6.4. Task dependencies

6.5. Logging using the sbt logger

6.6. Running your build using fork

6.7. Linking everything together: dependencies

6.8. Linking everything together: processes

6.9. Summary

Chapter 7. Accepting user input

7.1. Accepting user input with input tasks

project/databasehelper.scala

project/derby.scala

website/dbtest.sbt

7.2. Defining an interface with parsers

conf/evolutions/1.sql

7.3. Defining a database evolution parser

project/databaseevolutiontesting.scala

7.4. Helping the user with autocomplete

7.5. Summary

Chapter 8. Using plugins and external libraries

8.1. Using plugins to check your code

8.2. Using the Revolver plugin to restart the application

8.3. Creating your uber jar using the assembly plugin

8.4. Including a library in your plugin build

projects/scalastylereport.scala

8.5. Adding a plugin for use in all of your projects—signing your projects

8.6. Adding local credentials for deployment

8.7. Summary

Chapter 9. Debugging your build

9.1. Running sbt

9.2. Making your build compile

9.3. Ensuring tasks are run

9.4. Resolving dependency conflicts

9.4.1. Fixing resolution failures

9.4.2. Fixing resolution conflicts

9.5. Summary

4. Extending sbt

Chapter 10. Automating workflows with commands

10.1. Creating a simple command

10.2. Running tasks in parallel

10.3. Parsing input with commands

10.4. Creating useful help messages

10.5. Summary

Chapter 11. Defining a plugin

11.1. Reusing code using Scala files

analytics/build.sbt

11.2. Introducing the AutoPlugin interface

scalastyle-plugin/project/build.properties

scalastyle-plugin/build.sbt

scalastyle-plugin/src/main/scala/helloplugin.scala

scalastyle-plugin/src/main/scala/helloplugin.scala

scalastyle-test/build.sbt

scalastyle-test/project/build.properties

scalastyle-test/project/plugins.sbt

11.2.1. Taking a closer look—the plugin definition

11.2.2. Taking a closer look—task and setting dependencies

11.3. Testing a plugin with the scripted plugin

scalastyle-plugin/build.sbt

scalastyle-plugin/src/main/scala/scalastyleplugin.scala

scalastyle-plugin/src/main/scala/scalastyleplugin.scala

scalastyle-plugin/projects/plugins.sbt

scalastyle-plugin/build.sbt

scalastyle-plugin/src/sbt-test/config/does-not-exist/build.sbt

scalastyle-plugin/src/sbt-test/config/does-not-exist/test

11.4. Using configurations in your plugin

scalastyle-plugin/src/main/scala/scalastyleplugin.scala

scalastyle-plugin/src/main/scala/scalastyleplugin.scala

scalastyle-plugin/src/main/scala/scalastyleplugin.scala

11.5. Adding incremental tasks

scalastyle-plugin/src/main/scala/scalastyleplugin.scala

scalastyle-plugin/src/main/scala/scalastyleplugin.scala

scalastyle-plugin/src/main/scala/scalastyleplugin.scala

11.5.1. Adding incremental tasks with .previous—a sneak preview of 1.0

11.6. Making things easy for the user—more about the AutoPlugin interface

depend-plugin/build.sbt

src/main/scala/dependplugin.scala

depend-test/build.sbt

depend-test/project/plugins.sbt

depend-plugin/src/main/scala/dependplugin.scala

depend-test/build.sbt

depend-plugin/src/main/scala/dependplugin.scala

depend-plugin/build.sbt

11.7. Summary

5. Deploying your projects

Chapter 12. Distributing your projects

12.1. Publishing a library

12.2. Publishing locally

12.3. Publishing a simple server

12.4. Generating a distribution

$ ./analytics/target/universal/stage/bin/analytics

analytics/src/main/scala/main.scala

analytics/src/universal/conf/analytics.properties

12.5. Creating a Linux distribution

12.6. Summary

Appendix A. What you need to know about Ivy

A.1. Modules

A.2. Module identification

A.3. Artifacts

A.4. Differences in Ivy and Maven

A.5. Configuration

A.6. Resolvers

A.7. Resolving dependency conflicts

Appendix B. Migrating to autoplugins

B.1. Older sbt plugins

B.2. Migrating to autoplugins

Appendix C. Advanced setting/task API

C.1. Optional settings

C.2. Failing tasks

C.3. Dynamic tasks

C.4. Composing InputTasks

Index

List of Figures

List of Tables

List of Listings

Preface

It’s 2012-ish and I’m in the middle of bootstrapping Typesafe, the company behind Scala. I get a call from Manning about more book ideas for Scala. At the time, I was working on one of the least understood and most important tools in the Scala ecosystem: sbt. sbt was a build tool that had some rough edges, but was amazingly powerful and elegant in its core idioms. After a quick chat with Michael Stephens at Manning, we decided to put a proposal together and see if we could find a second author. While I really wanted to see an sbt book, as the community desperately needed one, at the time my commitments at home and work were too much for a new book. This lead to us eventually finding Matthew as a second author, and I’m very glad we did. Matthew has been great to work with and brings his own unique humor that I very much appreciated throughout the book writing process.

For some reason, I tend to wind up as the build guy for my day job. This inevitably happens due to two flaws in my personality:

A desire to understand everything I use

A desire to improve my daily workflow

At Typesafe, this meant I was thrown into using sbt. At the time I was a Maven guy, having fallen in love with the notional simplicity of lifecycles, plugins, and declarative builds. In fact, I had forks of most major Maven plugins at my company to tweak/work around issues we discovered. When I first picked up sbt, I didn’t like it.

The documentation at the time was sparse, and the idioms for the build were by far a lot different from what I was used to. However, as I started diving in, asking lots of questions of Mark Harrah, my coworker and the inventor of sbt, I found something beautiful underneath the rough exterior. sbt had (whether on purpose or by accident) turned into an automation delivery vehicle for your development experience.

By this, I mean that sbt allowed me to control and tune my build for my most common tasks, with an emphasis on interaction. In other words, sbt was not a one shot build; it was something you ran like an IDE, and it made coding Scala so much better than any other IDE at the time.

I’m so glad we were able to write this book to show you the vision of what I saw in this little tool. It still has some rough edges here and there, but I hope you’ll see the beauty and what I consider the future of build tools.

JOSH SUERETH

When Manning approached me about helping out with a book about sbt, I jumped at the chance. I’d contacted Manning to help with another book, and they suggested that I help with this one. I knew Josh through the Scala community, and I knew his reputation, so working with him would be great.

I’ve always been interested in code quality and software that helps me develop more easily. I like Larry Wall’s idea of being productively lazy. I’ve contributed to a number of open source projects to help with quality, such as Scala-IDE, JUnit, and most notably Scalastyle, the style checker for Scala that I wrote because I hate having to remember to put a copyright header on each source file.

The first real build tool I used was Make. Before that it was simple shell scripts. Make is great as long as you don’t have to build on multiple platforms. For me, the problems with Make are summed up in one project: Perl. There isn’t one makefile for Perl; it is generated by a 25,000-line shell script according to your system’s capabilities. Now, admittedly, Perl is built for a huge number of systems, but 25,000 lines to create a makefile? Really?

When I saw Apache Ant for the first time, it was a revelation. By abstracting out the differences between operating systems to the JVM, you didn’t need to worry if your system was being built on Microsoft Windows or Linux. And then came Maven. Maven took the abstraction even further—you said This is a Java project; build it rather than I want to compile my Java. But Maven was still a bit clunky for all of the custom tasks we had as part of our build.

Then I started using Scala. Some of the Scala developers used sbt, and I wanted to know more. I immediately saw the usefulness of some of the things that sbt was doing. First, the ~ operator has more rapid feedback. I loved saving a file and having my sbt tell me my unit tests had failed. What could be better than something automatically telling you that you had screwed up? Second, I loved the fact that the code was written in a statically typed language—I didn’t have to wait two hours for my build to fail with what would have been a compilation error.

We started writing the book in January 2013, and the result of that work is before you now. I hope you enjoy it.

I feel that the importance of build tools in a project is often underestimated. A good build tool will help you with your project. A bad build tool will get in your way.

MATTHEW FARWELL

Acknowledgments

When we signed up to write this book, we didn’t know how much effort it would require, or how long it would take. We started in 2013, and not only did we have to write the book, but sbt was changing underneath us as we wrote! It was Josh’s second book, so he had some idea, but Matthew had no idea. Without the assistance of many people, this book would definitely not be in your hands or on your screen right now. It’s much more likely that we would still be working on it.

From day one of the project, help has poured in, and all of it has been not only appreciated but essential. If we’ve forgotten anyone, please accept our sincere apologies. In no particular order, we’d like to thank the following people for their work and support.

Manning Publications

Many thanks to publisher Marjan Bace for accepting the initial proposal of two new and inexperienced authors. He helped the book take its first, and most important, step.

Thanks to Christina Rudolph and Michael Stephens, who were the first two people we talked with about the book.

Our development editor Jennifer Stout deserves a medal for the marathon work she did. Her patience and encouragement were invaluable. Special thanks to Candace Gillhoolley and Ana Romac for their marketing efforts.

Thanks also to many other members of the Manning team who helped us during production: Mary Piergies, Tiffany Taylor, Sharon Wilkey, Janet Vail, Melody Dolab, and Linda Recktenwald. And thanks to our technical editors Patrick Touhey and Matt Momont, as well as to our technical proofreader Thomas Lockney. Their work was invaluable in making our work better.

The sbt team

Without sbt itself, this book wouldn’t have been possible (or necessary). We’d like to thank the legion of volunteer developers responsible for the development of sbt. First, Mark Harrah, who started the sbt project and has carried it for most of its life. There are too many other contributors to mention, but we should at least mention Grzegorz Kossakowski, Eugene Yokota, Indrajit Raychaudhuri, Dale Wijnand, Nepomuk Seiler, Alexey Kardapoltsev, and of course Josh.

The reviewers

Thanks to our MEAP readers for their comments and corrections to the book and to the following reviewers who read the manuscript at various stages during its development and gave helpful feedback to our editors and to us: Alexander Myltsev, Bret Ikehara, Chad Michael Davis, Chris Nauroth, Dusan Kysel, Eric Weinberg, Iain Campbell, Joost den Boer, Logan Hauenstein, Michael Schleichardt, Mike Dalrymple, Nicolas Jorand, Ricky Yim, Tobias Kaatz, and William Wheeler; also to Mark Harrah, Sergii Vozniuk, Sergii Kuku, Philippe Genevès, Nicolas Jorand, and everyone who participated in the book’s Author Online forum.

About this Book

sbt takes the concept of a reliable build tool and extends it in two ways—using a DSL written in the Scala language and as an interactive development environment. sbt in Action is a step-by-step introduction to sbt. The text includes numerous real-world examples designed to demonstrate the power and flexibility of sbt, along with best practices for using it.

This book is intended for professional developers who want practical, battle-tested advice on how to get their own projects built using sbt. Developers building applications with sbt will typically use several related technologies as part of their projects. A book that provided complete information on each of these would fill many volumes. To avoid killing more trees than necessary, we assume that the reader is familiar with the JVM, and to a certain extent the Scala language syntax. However, we don’t feel that the reader needs too much knowledge about either of these. A wealth of information is available on these elsewhere.

Our focus is sbt and the problems it solves—the building and development of Scala and Java projects.

Roadmap

Chapter 1 talks about the need for a standard reliable build tool. We talk about some of the other options available for building Scala and Java projects, comparing and contrasting them with sbt. We end the chapter with our beliefs: why we use sbt

Chapter 2 introduces sbt—how to get started with it and basic usage. We introduce the basic commands and the basic concepts of tasks and settings. We show how to run your unit tests. This chapter will get you up and running with sbt.

Chapter 3 is a deeper dive into the core concepts of sbt: settings, tasks, and projects. We’ll see how to reuse values, introducing the notion of build dependencies. We’ll also use an Excel analogy to illustrate setting reuse. We’ll write our first custom task to create a Java properties file as part of your build. We’ll also introduce the notion of configurations, which are namespaces for settings and tasks.

Chapter 4 covers the default build—all of the default settings and tasks that mean that sbt gives you a standard build out of the box. We walk through compile, run, test, package, and publish to see how they work.

Chapter 5 covers how to test your code and how to use sbt to speed up your development. It also covers how to use different test frameworks within sbt. We cover Specs2, ScalaCheck, JUnit, and ScalaTest. We’ll perform some integration testing using the ScalaTest Selenium DSL. We’ll also cover how to get nicely formatted reports from your tests.

Chapter 6 goes for a deeper dive into the library made available by sbt for IO: sbt.IO. If you do any complex custom tasks in sbt, this library will do most of the heavy lifting for you.

Chapter 7 has more information about how to help developers develop using sbt. We create an sbt command which performs database migration through the sbt console as an example of how you can use sbt to help with non-build tasks which aid developers.

Chapter 8 covers plugins and external libraries—how to reuse components created by others in your build, and how to not reinvent the wheel.

Chapter 9 covers debugging your build—what to do when things go wrong.

Chapter 10 covers automating workflows. We create a simple release command which performs a number of checks on your project, then builds, creates, and pushes a version control tag. This is an example of how a workflow can be created to increase safety on your build.

Chapter 11 covers how to define plugins and when and how to reuse your build code. We cover how to create the plugin and how to test it, along with how to use configurations in your plugin and how to create incremental tasks. We cover the AutoPlugin interface, which allows dependencies between plugins.

Chapter 12 shows how to distribute your projects. We publish a library to a company repository and create a native package (a deb file, but this could also be an rpm or msi file).

Code conventions and downloads

When we present source code in the book, we use Courier typeface like this to separate it from normal text. Sometimes, when the context is not clear, the code is prefixed with the file name. These will appear in bold typeface.

Much of the source code shown in the earlier part of the book consists of fragments designed to illustrate the text. When a complete segment of code is given, it’s shown as a numbered listing; code annotations accompany some of the listings.

The source code for the example applications in this book is available to download via Manning’s website, www.manning.com/books/sbt-in-action. It is also available from the GitHub website at https://github.com/jsuereth/sbt-in-action-examples.

Author Online

Purchase of sbt in Action includes free access to a private web forum run by Manning Publications where you can make comments about the book, ask technical questions, and receive help from the authors and from other users. To access the forum and subscribe to it, point your web browser to www.manning.com/books/sbt-in-action. This page provides information on how to get on the forum once you are registered, what kind of help is available, and the rules of conduct on the forum.

Manning’s commitment to our readers is to provide a venue where a meaningful dialog between individual readers and between readers and the authors can take place. It is not a commitment to any specific amount of participation on the part of the author, whose contribution to the AO remains voluntary (and unpaid). We suggest you try asking the authors some challenging questions lest their interest stray!

The Author Online forum and the archives of previous discussions will be accessible from the publisher’s website as long as the book is in print.

About the authors

JOSH SUERETH is a senior software engineer at Typesafe and the author of Manning’s Scala in Depth. He is the author of several open source Scala projects, including the Scala automated resource management library, the PGP sbt plugin.

MATTHEW FARWELL is a senior software developer at Nexthink. He is the author of Scalastyle, the style checker for Scala, and is a contributor to JUnit and the Eclipse Scala IDE. Matthew lives in Lausanne, Switzerland, with his gorgeous wife and gorgeous daughter.

About the Cover Illustration

The figure on the cover of sbt in Action is captioned Habit of a Pilgrim of Mecca called Saquaz, who carries aromatic water to sell. The illustration is taken from Thomas Jefferys’ A Collection of the Dresses of Different Nations, Ancient and Modern (4 volumes), London, published between 1757 and 1772. The title page states that these are hand-colored copperplate engravings, heightened with gum arabic. Thomas Jefferys (1719–1771) was called Geographer to King George III. He was an English cartographer who was the leading map supplier of his day. He engraved and printed maps for government and other official bodies and produced a wide range of commercial maps and atlases, especially of North America. His work as a map maker sparked an interest in local dress customs of the lands he surveyed and mapped, which are brilliantly displayed in this four-volume collection.

Fascination with faraway lands and travel for pleasure were relatively new phenomena in the late 18th century and collections such as this one were popular, introducing both the tourist as well as the armchair traveler to the inhabitants of other countries. The diversity of the drawings in Jefferys’ volumes speaks vividly of the uniqueness and individuality of the world’s nations some 200 years ago. Dress codes have changed since then and the diversity by region and country, so rich at the time, has faded away. It is now often hard to tell the inhabitant of one continent from another. Perhaps, trying to view it optimistically, we have traded a cultural and visual diversity for a more varied personal life, or a more varied and interesting intellectual and technical life.

Manning celebrates the inventiveness and initiative of the computer business with book covers based on the rich diversity of regional life of two centuries ago, brought back to life by illustrations from old books and collections like this one.

Part 1. Why SBT?

In Part 1, we’ll go over a bit of history, including why you would want a tool like sbt rather than other available build tools. We’ll cover some of the advantages and disadvantages of each approach. Then we’ll get you up and running with sbt.

Chapter 1. Why sbt?

This chapter covers

Why use a build tool?

Apache Ant

Apache Maven

Gradle

Why sbt?

Interactivity/reactivity of sbt

In this chapter we’re going to explore why someone would use sbt (Simple Build Tool) to give a bit of a background to this book. If you’re convinced that you need to use sbt, you can probably skip this chapter. We’ll cover why you’d want an automated build and what other options you have for Scala and Java development. Finally, we’ll compare these other options to sbt and show why we think sbt is a good option for a large range of builds but especially for Scala builds.

For the impatient: sbt is a build tool that creates a stable build platform and increases developer productivity by taking the good ideas from other build tools such as

Minimal configuration—default project layouts

A number of built-in tasks (test, compile, publish)

and adding new features:

A reactive development environment. (You change your source code, and sbt reruns your tests.)

Allowing developers to use the Scala REPL on their projects.

Faster compilation through incremental compilation and caching.

Automatic recompilation with different versions of Scala.

Let’s start at the beginning.

1.1. Why a build tool?

If you spend any time working in a team of software developers, you’ll notice one thing: people make mistakes.