Mastering Embedded Linux Programming - Second Edition

< html PUBLIC -//W3C//DTD HTML 4.0 Transitional//EN http://www.w3.org/TR/REC-html40/loose.dtd>

Mastering Embedded Linux Programming

Second Edition

Unleash the full potential of Embedded Linux

Chris Simmonds

BIRMINGHAM - MUMBAI

< html PUBLIC -//W3C//DTD HTML 4.0 Transitional//EN http://www.w3.org/TR/REC-html40/loose.dtd>

Mastering Embedded Linux Programming

Second Edition

Copyright © 2017 Packt Publishing  

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: December 2015

Second edition: June 2017

Production reference: 1280617

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham 

B3 2PB, UK.

ISBN 978-1-78728-328-2

www.packtpub.com

Credits

About the Author

Chris Simmonds is a software consultant and trainer living in southern England. He has almost two decades of experience in designing and building open-source embedded systems. He is the founder and chief consultant at 2net Ltd, which provides professional training and mentoring services in embedded Linux, Linux device drivers, and Android platform development. He has trained engineers at many of the biggest companies in the embedded world, including ARM, Qualcomm, Intel, Ericsson, and General Dynamics. He is a frequent presenter at open source and embedded conferences, including the Embedded Linux Conference and Embedded World. You can see some of his work on the Inner Penguin blog at www.2net.co.uk.

I would like to thank Shirley Simmonds for being so supportive during the long hours that I was shut in my home office researching and writing this book. I would also like to thank all the people who have helped me with the research of the technical aspects of this book, whether they realized that is what they were doing or not. In particular, I would like to mention Klaas van Gend, Thomas Petazzoni, and Ralph Nguyen for their help and advice. Lastly, I would like to thank Sharon Raj, Vishal Mewada, and the team at Packt Publishing for keeping me on track and bringing the book to fruition.

About the Reviewers

Daiane Angolini has been working with embedded Linux since 2008. She has been working as an application engineer at NXP, acting on internal development, porting custom applications from Android, and on-customer support for i.MX architectures in areas such as Linux kernel, u-boot, Android, Yocto Project, and user-space applications. However, it was on the Yocto Project that she found her place. She has coauthored the books Embedded Linux Development with Yocto Project and Heading for the Yocto Project, and learned a lot in the process.

Otavio Salvador loves technology and started his free software activities in 1999. In 2002, he founded O.S. Systems, a company focused on embedded system development services and consultancy worldwide, creating and maintaining customized BSPs, and helping companies with their product's development challenges. This resulted in him joining the OpenEmbedded community in 2008, when he became an active contributor to the OpenEmbedded project. He has coauthored the books Embedded Linux Development with Yocto Project and Heading for the Yocto Project.

Alex Tereschenko is an embedded systems engineer by day, and an avid maker by night, who is convinced that computers can do a lot of good for people when they are interfaced with real-world objects, as opposed to just crunching data in a dusty corner. That's what's driving him in his projects, and this is why embedded systems and the Internet of Things are the topics he enjoys the most.

www.PacktPub.com

For support files and downloads related to your book, please visit www.PacktPub.com. Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.comand as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at service@packtpub.com for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www.packtpub.com/mapt

Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print, and bookmark content

On demand and accessible via a web browser

Customer Feedback

Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at https://www.amazon.com/dp/1787283283.

If you'd like to join our team of regular reviewers, you can e-mail us at customerreviews@packtpub.com. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!

Table of Contents

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

Starting Out

Selecting the right operating system

The players

Project life cycle

The four elements of embedded Linux

Open source

Licenses

Hardware for embedded Linux

Hardware used in this book

The BeagleBone Black

QEMU

Software used in this book

Summary

Learning About Toolchains

Introducing toolchains

Types of toolchains

CPU architectures

Choosing the C library

Finding a toolchain

Building a toolchain using crosstool-NG

Installing crosstool-NG

Building a toolchain for BeagleBone Black

Building a toolchain for QEMU

Anatomy of a toolchain

Finding out about your cross compiler

The sysroot, library, and header files

Other tools in the toolchain

Looking at the components of the C library

Linking with libraries – static and dynamic linking

Static libraries

Shared libraries

Understanding shared library version numbers

The art of cross compiling

Simple makefiles

Autotools

An example: SQLite

Package configuration

Problems with cross compiling

Summary

All About Bootloaders

What does a bootloader do?

The boot sequence

Phase 1 – ROM code

Phase 2 – secondary program loader

Phase 3 – TPL

Booting with UEFI firmware

Moving from bootloader to kernel

Introducing device trees

Device tree basics

The reg property

Labels and interrupts

Device tree include files

Compiling a device tree

Choosing a bootloader

U-Boot

Building U-Boot

Installing U-Boot

Using U-Boot

Environment variables

Boot image format

Loading images

Booting Linux

Automating the boot with U-Boot scripts

Porting U-Boot to a new board

Board-specific files

Configuring header files

Building and testing

Falcon mode

Barebox

Getting barebox

Building barebox

Using barebox

Summary

Configuring and Building the Kernel

What does the kernel do?

Choosing a kernel

Kernel development cycle

Stable and long term support releases

Vendor support

Licensing

Building the kernel

Getting the source

Understanding kernel configuration – Kconfig

Using LOCALVERSION to identify your kernel

Kernel modules

Compiling – Kbuild

Finding out which kernel target to build

Build artifacts

Compiling device trees

Compiling modules

Cleaning kernel sources

Building a kernel for the BeagleBone Black

Building a kernel for QEMU

Booting the kernel

Booting the BeagleBone Black

Booting QEMU

Kernel panic

Early user space

Kernel messages

Kernel command line

Porting Linux to a new board

A new device tree

Setting the board compatible property

Additional reading

Summary

Building a Root Filesystem

What should be in the root filesystem?

The directory layout

The staging directory

POSIX file access permissions

File ownership permissions in the staging directory

Programs for the root filesystem

The init program

Shell

Utilities

BusyBox to the rescue!

Building BusyBox

ToyBox – an alternative to BusyBox

Libraries for the root filesystem

Reducing the size by stripping

Device nodes

The proc and sysfs filesystems

Mounting filesystems

Kernel modules

Transferring the root filesystem to the target

Creating a boot initramfs

Standalone initramfs

Booting the initramfs

Booting with QEMU

Booting the BeagleBone Black

Mounting proc

Building an initramfs into the kernel image

Building an initramfs using a device table

The old initrd format

The init program

Starting a daemon process

Configuring user accounts

Adding user accounts to the root filesystem

A better way of managing device nodes

An example using devtmpfs

An example using mdev

Are static device nodes so bad after all?

Configuring the network

Network components for glibc

Creating filesystem images with device tables

Booting the BeagleBone Black

Mounting the root filesystem using NFS

Testing with QEMU

Testing with the BeagleBone Black

Problems with file permissions

Using TFTP to load the kernel

Additional reading

Summary

Selecting a Build System

Build systems

Package formats and package managers

Buildroot

Background

Stable releases and long-term support

Installing

Configuring

Running

Creating a custom BSP

U-Boot

Linux

Build

Adding your own code

Overlays

Adding a package

License compliance

The Yocto Project

Background

Stable releases and supports

Installing the Yocto Project

Configuring

Building

Running the QEMU target

Layers

BitBake and recipes

Customizing images via local.conf

Writing an image recipe

Creating an SDK

The license audit

Further reading

Summary

Creating a Storage Strategy

Storage options

NOR flash

NAND flash

Managed flash

MultiMediaCard and Secure Digital cards

eMMC

Other types of managed flash

Accessing flash memory from the bootloader

U-Boot and NOR flash

U-Boot and NAND flash

U-Boot and MMC, SD, and eMMC

Accessing flash memory from Linux

Memory technology devices

MTD partitions

MTD device drivers

The MTD character device, mtd

The MTD block device, mtdblock

Logging kernel oops to MTD

Simulating NAND memory

The MMC block driver

Filesystems for flash memory

Flash translation layers

Filesystems for NOR and NAND flash memory

JFFS2

Summary nodes

Clean markers

Creating a JFFS2 filesystem

YAFFS2

Creating a YAFFS2 filesystem

UBI and UBIFS

UBI

UBIFS

Filesystems for managed flash

Flashbench

Discard and TRIM

Ext4

F2FS

FAT16/32

Read-only compressed filesystems

squashfs

Temporary filesystems

Making the root filesystem read-only

Filesystem choices

Further reading

Summary

Updating Software in the Field

What to update?

Bootloader

Kernel

Root filesystem

System applications

Device-specific data

Components that need to be updated

The basics of software update

Making updates robust

Making updates fail-safe

Making updates secure

Types of update mechanism

Symmetric image update

Asymmetric image update

Atomic file updates

OTA updates

Using Mender for local updates

Building the Mender client

Installing an update

Using Mender for OTA updates

Summary

Interfacing with Device Drivers

The role of device drivers

Character devices

Block devices

Network devices

Finding out about drivers at runtime

Getting information from sysfs

The devices: /sys/devices

The drivers: /sys/class

The block drivers: /sys/block

Finding the right device driver

Device drivers in user space

GPIO

Handling interrupts from GPIO

LEDs

I2C

Serial Peripheral Interface (SPI)

Writing a kernel device driver

Designing a character driver interface

The anatomy of a device driver

Compiling kernel modules

Loading kernel modules

Discovering the hardware configuration

Device trees

The platform data

Linking hardware with device drivers

Additional reading

Summary

Starting Up – The init Program

After the kernel has booted

Introducing the init programs

BusyBox init

Buildroot init scripts

System V init

inittab

The init.d scripts

Adding a new daemon

Starting and stopping services

systemd

Building systemd with the Yocto Project and Buildroot

Introducing targets, services, and units

Units

Services

Targets

How systemd boots the system

Adding your own service

Adding a watchdog

Implications for embedded Linux

Further reading

Summary

Managing Power

Measuring power usage

Scaling the clock frequency

The CPUFreq driver

Using CPUFreq

Selecting the best idle state

The CPUIdle driver

Tickless operation

Powering down peripherals

Putting the system to sleep

Power states

Wakeup events

Timed wakeups from the real-time clock

Further reading

Summary

Learning About Processes and Threads

Process or thread?

Processes

Creating a new process

Terminating a process

Running a different program

Daemons

Inter-process communication

Message-based IPC

Unix (or local) sockets

FIFOs and named pipes

POSIX message queues

Summary of message-based IPC

Shared memory-based IPC

POSIX shared memory

Threads

Creating a new thread

Terminating a thread

Compiling a program with threads

Inter-thread communication

Mutual exclusion

Changing conditions

Partitioning the problem

Scheduling

Fairness versus determinism

Time-shared policies

Niceness

Real-time policies

Choosing a policy

Choosing a real-time priority

Further reading

Summary

Managing Memory

Virtual memory basics

Kernel space memory layout

How much memory does the kernel use?

User space memory layout

The process memory map

Swapping

Swapping to compressed memory (zram)

Mapping memory with mmap

Using mmap to allocate private memory

Using mmap to share memory

Using mmap to access device memory

How much memory does my application use?

Per-process memory usage

Using top and ps

Using smem

Other tools to consider

Identifying memory leaks

mtrace

Valgrind

Running out of memory

Further reading

Summary

Debugging with GDB

The GNU debugger

Preparing to debug

Debugging applications

Remote debugging using gdbserver

Setting up the Yocto Project for remote debugging

Setting up Buildroot for remote debugging

Starting to debug

Connecting GDB and gdbserver

Setting the sysroot

GDB command files

Overview of GDB commands

Breakpoints

Running and stepping

Getting information

Running to a breakpoint

Native debugging

The Yocto Project

Buildroot

Just-in-time debugging

Debugging forks and threads

Core files

Using GDB to look at core files

GDB user interfaces

Terminal user interface

Data display debugger

Eclipse

Debugging kernel code

Debugging kernel code with kgdb

A sample debug session

Debugging early code

Debugging modules

Debugging kernel code with kdb

Looking at an Oops

Preserving the Oops

Further reading

Summary

Profiling and Tracing

The observer effect

Symbol tables and compile flags

Beginning to profile

Profiling with top

Poor man's profiler

Introducing perf

Configuring the kernel for perf

Building perf with the Yocto Project

Building perf with Buildroot

Profiling with perf

Call graphs

perf annotate

Other profilers – OProfile and gprof

Tracing events

Introducing Ftrace

Preparing to use Ftrace

Using Ftrace

Dynamic Ftrace and trace filters

Trace events

Using LTTng

LTTng and the Yocto Project

LTTng and Buildroot

Using LTTng for kernel tracing

Using Valgrind

Callgrind

Helgrind

Using strace

Summary

Real-Time Programming

What is real time?

Identifying sources of non-determinism

Understanding scheduling latency

Kernel preemption

The real-time Linux kernel (PREEMPT_RT)

Threaded interrupt handlers

Preemptible kernel locks

Getting the PREEMPT_RT patches

The Yocto Project and PREEMPT_RT

High-resolution timers

Avoiding page faults

Interrupt shielding

Measuring scheduling latencies

cyclictest

Using Ftrace

Combining cyclictest and Ftrace

Further reading

Summary

Preface

Linux has been the mainstay of embedded computing for many years. And yet, there are remarkably few books that cover the topic as a whole: this book is intended to fill that gap. The term embedded Linux is not well-defined, and can be applied to the operating system inside a wide range of devices ranging from thermostats to Wi-Fi routers to industrial control units. However, they are all built on the same basic open source software. Those are the technologies that I describe in this book, based on my experience as an engineer and the materials I have developed for my training courses.

Technology does not stand still. The industry based around embedded computing is just as susceptible to Moore's law as mainstream computing. The exponential growth that this implies has meant that a surprisingly large number of things have changed since the first edition of this book was published. This second edition is fully revised to use the latest versions of the major open source components, which include Linux 4.9, Yocto Project 2.2 Morty, and Buildroot 2017.02. Since it is clear that embedded Linux will play an important part in the Internet of Things, there is a new chapter on the updating of devices in the field, including Over the Air updates. Another trend is the quest to reduce power consumption, both to extend the battery life of mobile devices and to reduce energy costs. The chapter on power management shows how this is done.

Mastering Embedded Linux Programming covers the topics in roughly the order that you will encounter them in a real-life project. The first 6 chapters are concerned with the early stages of the project, covering basics such as selecting the toolchain, the bootloader, and the kernel. At the conclusion of this this section, I introduce the idea of using an embedded build tool, using Buildroot and the Yocto Project as examples.

The middle part of the book, chapters 7 through to 13, will help you in the implementation phase of the project. It covers the topics of filesystems, the init program, multithreaded programming, software update, and power management. The third section, chapters 14 and 15, show you how to make effective use of the many debug and profiling tools that Linux has to offer in order to detect problems and identify bottlenecks. The final chapter brings together several threads to explain how Linux can be used in real-time applications.

Each chapter introduces a major area of embedded Linux. It describes the background so that you can learn the general principles, but it also includes detailed worked examples that illustrate each of these areas. You can treat this as a book of theory, or a book of examples. It works best if you do both: understand the theory and try it out in real life.

What this book covers

Chapter 1, Starting Out, sets the scene by describing the embedded Linux ecosystem and the choices available to you as you start your project. 

Chapter 2, Learning About Toolchains, describes the components of a toolchain and shows you how to create a toolchain for cross-compiling code for the target board. It describes where to get a toolchain and provides details on how to build one from the source code.

Chapter 3, All About Bootloaders, explains the role of the bootloader in loading the Linux kernel into memory, and uses U-Boot and Bareboot as examples. It also introduces device trees as the mechanism used to encode the details of hardware in almost all embedded Linux systems.

Chapter 4, Configuring and Building the Kernel, provides information on how to select a Linux kernel for an embedded system and configure it for the hardware within the device. It also covers how to port Linux to the new hardware.

Chapter 5, Building a Root Filesystem, introduces the ideas behind the user space part of an embedded Linux implementation by means of a step-by-step guide on how to configure a root filesystem.

Chapter 6, Selecting a Build System, covers two commonly used embedded Linux build systems, Buildroot and Yocto Project, which automate the steps described in the previous four chapters.

Chapter 7, Creating a Storage Strategy, discusses the challenges created by managing flash memory, including raw flash chips and embedded MMC (eMMC) packages. It describes the filesystems that are applicable to each type of technology.

Chapter 8, Updating Software in the Field, examines various ways of updating the software after the device has been deployed, and includes fully managed Over the Air (OTA) updates. The key topics under discussion are reliability and security.

Chapter 9, Interfacing with Device Drivers, describes how kernel device drivers interact with the hardware with worked examples of a simple driver. It also describes the various ways of calling device drivers from the user space.

Chapter 10, Starting Up – The Init Program, shows how the first user space program--init--starts the rest of the system. It describes the three versions of the init program, each suitable for a different group of embedded systems, ranging from the simplicity of the BusyBox init, through System V init, to the current state-of-the-art, systemd.

Chapter 11, Managing Power, considers the various ways that Linux can be tuned to reduce power consumption, including Dynamic Frequency and Voltage scaling, selecting deeper idle states, and system suspend. The aim is to make devices that run for longer on a battery charge and also run cooler.

Chapter 12, Learning About Processes and Threads, describes embedded systems from the point of view of the application programmer. This chapter looks at processes and threads, inter-process communications, and scheduling policies

Chapter 13, Managing Memory, introduces the ideas behind virtual memory and how the address space is divided into memory mappings. It also describes how to measure memory usage accurately and how to detect memory leaks.

Chapter 14, Debugging with GDB, shows you how to use the GNU debugger, GDB, together with the debug agent, gdbserver, to debug applications running remotely on the target device. It goes on to show how you can extend this model to debug kernel code, making use of the kernel debug stubs, KGDB.

Chapter 15, Profiling and Tracing, covers the techniques available to measure the system performance, starting from whole system profiles and then zeroing in on particular areas where bottlenecks are causing poor performance. It also describes how to use Valgrind to check the correctness of an application's use of thread synchronization and memory allocation.

Chapter 16, Real-Time Programming, provides a detailed guide to real-time programming on Linux, including the configuration of the kernel and the PREEMPT_RT real-time kernel patch. The kernel trace tool, Ftrace, is used to measure kernel latencies and show the effect of the various kernel configurations.

What you need for this book

The software used in this book is entirely open source. In almost all cases, I have used the latest stable versions available at the time of writing. While I have tried to describe the main features in a manner that is not version-specific, it is inevitable that some of the examples will need adaptation to work with later software.

Embedded development involves two systems: the host, which is used for developing the programs, and the target, which runs them. For the host system, I have used Ubuntu 16.04, but most Linux distributions will work with just a little modification. You may decide to run Linux as a guest in a virtual machine, but you should be aware that some tasks, such as building a distribution using the Yocto Project, are quite demanding and are better run on a native installation of Linux.

I chose two exemplar targets: the QEMU emulator and the BeagleBone Black. Using QEMU means that you can try out most of the examples without having to invest in any additional hardware. On the other hand, some things work better if you do have real hardware, for which, I have chosen the BeagleBone Black because it is not expensive, it is widely available, and it has very good community support. Of course, you are not limited to just these two targets. The idea behind the book is to provide you with general solutions to problems so that you can apply them to a wide range of target boards.

Who this book is for

This book is written for developers who have an interest in embedded computing and Linux, and want to extend their knowledge into the various branches of the subject. In writing the book, I assume a basic understanding of the Linux command line, and in the programming examples, a working knowledge of the C language. Several chapters focus on the hardware that goes into an embedded target board, and, so, a familiarity with hardware and hardware interfaces will be a definite advantage in these cases.

Conventions

In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning. Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: You configure tap0 in exactly the same way as any other interface.

A block of code is set as follows:

Any command-line input or output is written as follows:

New terms and important words are shown in bold.

Warnings or important notes appear in a box like this.

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply e-mail feedback@packtpub.com, and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for this book from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you. You can download the code files by following these steps:

Log in or register to our website using your e-mail address and password.

Hover the mouse pointer on the SUPPORT tab at the top.

Click on Code Downloads & Errata.

Enter the name of the book in the Search box.

Select the book for which you're looking to download the code files.

Choose from the drop-down menu where you purchased this book from.

Click on Code Download.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

WinRAR / 7-Zip for Windows

Zipeg / iZip / UnRarX for Mac

TAR for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Mastering-Embedded-Linux-Programming-Second-Edition. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Downloading the color images of this book

We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from https://www.packtpub.com/sites/default/files/downloads/MasteringEmbeddedLinuxProgrammingSecondEdition_ColorImages.pdf.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books-maybe a mistake in the text or the code-we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title. To view the previously submitted errata, go to https://www.packtpub.com/books/content/support  and enter the name of the book in the search field. The required information will appear under the Errata section.

Piracy

Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy. Please contact us at copyright@packtpub.com with a link to the suspected pirated material. We appreciate your help in protecting our authors and our ability to bring you valuable content.

Questions

If you have a problem with any aspect of this book, you can contact us at questions@packtpub.com, and we will do our best to address the problem.

Starting Out

You are about to begin working on your next project, and this time it is going to be running Linux. What should you think about before you put finger to keyboard? Let's begin with a high-level look at embedded Linux and see why it is popular, what are the implications of open source licenses, and what kind of hardware you will need to run Linux.

Linux first became a viable choice for embedded devices around 1999. That was when Axis (https://www.axis.com), released their first Linux-powered network camera and TiVo (https://business.tivo.com/) their first Digital Video Recorder (DVR). Since 1999, Linux has become ever more popular, to the point that today it is the operating system of choice for many classes of product. At the time of writing, in 2017, there are about two billion devices running Linux. That includes a large number of smartphones running Android, which uses a Linux kernel, and hundreds of millions of set-top-boxes, smart TVs, and Wi-Fi routers, not to mention a very diverse range of devices such as vehicle diagnostics, weighing scales, industrial devices, and medical monitoring units that ship in smaller volumes.

So, why does your TV run Linux? At first glance, the function of a TV is simple: it has to display a stream of video on a screen. Why is a complex Unix-like operating system like Linux necessary?

The simple answer is Moore's Law: Gordon Moore, co-founder of Intel, observed in 1965 that the density of components on a chip will double approximately every two years. That applies to the devices that we design and use in our everyday lives just as much as it does to desktops, laptops, and servers. At the heart of most embedded devices is a highly integrated chip that contains one or more processor cores and interfaces with main memory, mass storage, and peripherals of many types. This is referred to as a System on Chip, or SoC, and SoCs are increasing in complexity in accordance with Moore's Law. A typical SoC has a technical reference manual that stretches to thousands of pages. Your TV is not simply displaying a video stream as the old analog sets used to do.

The stream is digital, possibly encrypted, and it needs processing to create an image. Your TV is (or soon will be) connected to the Internet. It can receive content from smartphones, tablets, and home media servers. It can be (or soon will be) used to play games. And so on and so on. You need a full operating system to manage this degree of complexity.

Here are some points that drive the adoption of Linux:

Linux has the necessary functionality. It has a good scheduler, a good network stack, support for USB, Wi-Fi, Bluetooth, many kinds of storage media, good support for multimedia devices, and so on. It ticks all the boxes.

Linux has been ported to a wide range of processor architectures, including some that are very commonly found in SoC designs--ARM, MIPS, x86, and PowerPC.

Linux is open source, so you have the freedom to get the source code and modify it to meet your needs. You, or someone working on your behalf, can create a board support package for your particular SoC board or device. You can add protocols, features, and technologies that may be missing from the mainline source code. You can remove features that you don't need to reduce memory and storage requirements. Linux is flexible.

Linux has an active community; in the case of the Linux kernel, very active. There is a new release of the kernel every 8 to 10 weeks, and each release contains code from more than 1,000 developers. An active community means that Linux is up to date and supports current hardware, protocols, and standards.

Open source licenses guarantee that you have access to the source code. There is no vendor tie-in.

For these reasons, Linux is an ideal choice for complex devices. But there are a few caveats I should mention here. Complexity makes it harder to understand. Coupled with the fast moving development process and the decentralized structures of open source, you have to put some effort into learning how to use it and to keep on re-learning as it changes. I hope that this book will help in the process.

Selecting the right operating system

Is Linux suitable for your project? Linux works well where the problem being solved justifies the complexity. It is especially good where connectivity, robustness, and complex user interfaces are required. However, it cannot solve every problem, so here are some things to consider before you jump in:

Is your hardware up to the job? Compared to a traditional real-time operating system (RTOS) such as VxWorks, Linux requires a lot more resources. It needs at least a 32-bit processor and lots more memory. I will go into more detail in the section on typical hardware requirements.

Do you have the right skill set? The early parts of a project, board bring-up, require detailed knowledge of Linux and how it relates to your hardware. Likewise, when debugging and tuning your application, you will need to be able to interpret the results. If you don't have the skills in-house, you may want to outsource some of the work. Of course, reading this book helps!

Is your system real-time? Linux can handle many real-time activities so long as you pay attention to certain details, which I will cover in detail in Chapter 16, Real-Time Programming.

Consider these points carefully. Probably the best indicator of success is to look around for similar products that run Linux and see how they have done it; follow best practice.

The players

Where does open source software come from? Who writes it? In particular, how does this relate to the key components of embedded development—the toolchain, bootloader, kernel, and basic utilities found in the root filesystem?

The main players are:

The open source community: This, after all, is the engine that generates the software you are going to be using. The community is a loose alliance of developers, many of whom are funded in some way, perhaps by a not-for-profit organization, an academic institution, or a commercial company. They work together to further the aims of the various projects. There are many of them—some small, some large. Some that we will be making use of in the remainder of this book are Linux itself, U-Boot, BusyBox, Buildroot, the Yocto Project, and the many projects under the GNU umbrella.

CPU architects: These are the organizations that design the CPUs we use. The important ones here are ARM/Linaro (ARM-based SoCs), Intel (x86 and x86_64), Imagination Technologies (MIPS), and IBM (PowerPC). They implement or, at the very least, influence support for the basic CPU architecture.

SoC vendors (Atmel, Broadcom, Intel, Qualcomm, TI, and many others). They take the kernel and toolchain from the CPU architects and modify them to support their chips. They also create reference boards: designs that are used by the next level down to create development boards and working products.

Board vendors and OEMs: These people take the reference designs from SoC vendors and build them in to specific products, for instance, set-top-boxes or cameras, or create more general purpose development boards, such as those from Avantech and Kontron. An important category are the cheap development boards such as BeagleBoard/BeagleBone and Raspberry Pi that have created their own ecosystems of software and hardware add-ons.

These form a chain, with your project usually at the end, which means that you do not have a free choice of components. You cannot simply take the latest kernel from https://www.kernel.org/, except in a few rare cases, because it does not have support for the chip or board that you are using.

This is an ongoing problem with embedded development. Ideally, the developers at each link in the chain would push their changes upstream, but they don't. It is not uncommon to find a kernel which has many thousands of patches that are not merged. In addition, SoC vendors tend to actively develop open source components only for their latest chips, meaning that support for any chip more than a couple of years old will be frozen and not receive any updates.

The consequence is that most embedded designs are based on old versions of software. They do not receive security fixes, performance enhancements, or features that are in newer versions. Problems such as Heartbleed (a bug in the OpenSSL libraries) and ShellShock (a bug in the bash shell) go unfixed. I will talk more about this later in this chapter under the topic of security.

What can you do about it? First, ask questions of your vendors: what is their update policy, how often do they revise kernel versions, what is the current kernel version, what was the one before that, and what is their policy for merging changes up-stream? Some vendors are making great strides in this way. You should prefer their chips.

Secondly, you can take steps to make yourself more self-sufficient. The chapters in section 1 explain the dependencies in more detail and show you where you can help yourself. Don't just take the package offered to you by the SoC or board vendor and use it blindly without considering the alternatives.

Project life cycle

This book is divided into four sections that reflect the phases of a project. The phases are not necessarily sequential. Usually they overlap and you will need