0

Preface
This book is about porting software between UNIX platforms, the process of taking a software package in source form and installing it on your machine. This doesnt sound like a big deal at rst, but theres more to it than meets the eye: you need to know how to get the software, how to unpack what you get, how to modify the package so that it will compile on your system, how to compile and install the software on your system, and how to deal with problems if they crop up. Nevertheless, it doesnt involve anything that hasnt already been done to death in hundreds of well-written books: you can nd out about getting software from the Internet in The Whole Internet Users Guide and Catalog, by Ed Krol. Unpacking software is basically a matter of using standard tools described in dozens of good introductory textbooks. Compiling programs is so simple that most C textbooks deal with it in passing. Installation is just a matter of copying software to where you want it. Programming is the meat of lots of books on UNIX programming, for example Advanced Programming in the UNIX environment by Richard Stevens, So why yet another book? Most textbooks give you an idealized view of programming: This is the way to do it (and it works). They pay little attention to the ways things can go wrong. UNIX is famed for cryptic or misleading error messages, but not many books go into the details of why they appear or what they really mean. Even experienced programmers frequently give up when trying to port software. The probable advantage of completing the port just isnt worth effort that it takes. In this book, Id like to reduce that effort. If you take all the books I just mentioned, youll have to nd about 3 feet of shelf space to hold them. Theyre all good, but they contain stuff that you dont really want to know about right now (in fact, youre probably not sure if you ever want to know all of it). Maybe you have this pressing requirement to get this debugger package, or maybe you nally want to get the latest version of nethack up and running, complete with X11 support, and the last thing you want to do on the way is go through those three feet of paper. Thats where this book comes in. It covers all issues of porting, from nding the software through porting and testing up to the nal installation, in the sequence in which you perform them. It goes into a lot of detail comparing the features of many different UNIX systems, and offers suggestions about how to emulate features not available on the platform to which you
i

5 February 2005 02:09

ii

are porting. It views the problems from a practical rather than from a theoretical perspective. You probably wont know any more after reading it than you would after reading the in-depth books, but I hope that youll nd the approach more related to your immediate problems.

Audience
This book is intended for anybody who has to take other peoples software and compile it on a UNIX platform. It should be of particular interest to you if youre: A software developer porting software to a new platform. A system administrator collecting software products for your system. A computer hobbyist collecting software off the Internet.

Whatever your interest, I expect that youll know UNIX basics. If youre a real newcomer, you might like to refer to Learning the UNIX Operating System, by Grace Todino, John Strang and Jerry Peek. In addition, UNIX in a Nutshell, available in BSD and System V avours, includes a lot of reference material which I have not repeated in this book. The less you already know, the more use this book is going to be to you, of course. Nevertheless, even if youre an experienced programmer, you should nd a number of tricks to make life easier.

Organization
One of the big problems in porting software is that you need to know everything rst. While writing this book I had quite a problem deciding the order in which to present the material. In the end, I took a two-pronged approach, and divided this book into two major parts: 1. In the rst part, well look at the stages through which a typical port passes: getting the software, extracting the source archives, conguring the package, compiling the software, testing the results, and installing the completed package. In the second part, well take a look at the differences between different avours of UNIX, how they can make life hard for you, and how we can solve the problems.

2.

Operating System Versions

Nearly everything in this book is related to one version or another of UNIX,* and a lot of the text only makes sense in a UNIX context. Nevertheless, it should be of some use to users of other operating systems that use the C programming language and UNIX tools such as make. As in any book about UNIX, its difcult to give complete coverage to all avours. The examples in this book were made with six different hardware/software platforms:
* UNIX is, of course, a registered trademark of its current owner. In this context, I am referring to any operating system that presents a UNIX-like interface to the user and the programmer.

5 February 2005 02:09

Preface

iii

SCO XENIX/386 on an Intel 386 architecture (version 2.3.2). UNIX System V.3 on an Intel 386 architecture (Interactive UNIX/386 version 2.2). UNIX System V.4.2 on an Intel 386 architecture (Consensys V4.2). BSD on an Intel 386 architecture (BSD/386* 1.1 and FreeBSD). SunOS on a Sparc architecture (SunOS 4.1.3). IRIX 5.3 on an SGI Indy Workstation (mainly System V.4).

This looks like a strong bias towards Intel architectures. However, most problems are more related to the software platform than the hardware platform. The Intel platform is unique in offering almost every avour of UNIX that is currently available, and its easier to compare them if the hardware is invariant. I believe these examples to be representative of what you might nd on other hardware. The big difference in UNIX avours is certainly between UNIX System V.3 and BSD, while System V.4 represents the logical sum of both of them. At a more detailled level, every system has its own peculiarities: there is hardly a system available which doesnt have its own quirks. These quirks turn out to be the biggest problem that you will have to ght when porting software. Even software that ported just ne on the previous release of your operating system may suddenly turn into an error message generator.

Conventions used in this book

This book uses the following conventions: Bold is used for the names of keys on the keyboard. Well see more about this in the next section. Italic is used for the names of UNIX utilities, directories and lenames, and to emphasize new terms and concepts when they are rst introduced. Constant Width is used in examples to show the contents of les, the output from commands, program variables, actual values of keywords, for the names of Usenet newsgroups, and in the text to represent commands. Constant Italic is used in examples to show variables for which context-specic substitutions should be made. For example, the variable filename would be replaced by an actual lename. In addition it is used for comments in code examples. Constant Bold is used in examples to show commands or text that would be typed in literally by the user. Most examples assume the use of the Bourne shell or one of its descendents such as the Korn Shell, or the Free Software Foundations bash. Normally the prompt will be shown as the default $, unless it is an operation that requires the superuser, in which case it will be shown as #. When continuation lines are used, the prompt will be the standard >. In cases where the command wouldnt work with the C shell, I present an alternative. In the C shell examples, the prompt is the default %.
* Later versions of this operating system are called BSD/OS.

5 February 2005 02:09

iv

I have tried to make the examples in this book as close to practice as possible, and most are from real-life sources. A book is not a monitor, however, and displays that look acceptable (well, recognizable) on a monitor can sometimes look really bad in print. In particular, the utilities used in porting sometimes print out lines of several hundred characters. I have tried to modify such output in the examples so that it ts on the page. For similar reasons, I have modied the line breaks in some literally quoted texts, and have occasionally squeezed things like long directory listings.

Describing the keyboard

Its surprising how many confusing terms exist to describe individual keys on the keyboard. My favourite is the any key (Press any key to continue). We wont be using the any key in this book, but there are a number of other keys whose names need understanding: The Enter or Return key. Ill call this RETURN. Control characters (characters produced by holding down the CTRL key and pressing a normal keyboard key at the same time). These characters are frequently echoed on the screen as a caret () followed by the character entered. In keeping with other Nutshell books, Ill write control-D as CTRL-D. The ALT key, which emacs afcionados call a META key, works like a second CTRL key, but generates a different set of characters. These are sometimes abbreviated by prexing the character with a tilde () or the characters A-. Although these are useful abbreviations, they can be confusing, so Ill spell these out as CTRL-X and ALT-D, etc. NL is the new line character. In ASCII, it is CTRL-J, but UNIX systems generate it when you press the RETURN key. CR is the carriage return character, in ASCII CTRL-M. Most systems generate it with the RETURN key. HT is the ASCII horizontal tab character, CTRL-I. Most systems generate it when the TAB key is pressed.

Terminology
Any technical book uses jargon and technical terms that are not generally known. Ive tried to recognize the ones used in this book and describe them when they occur. Apart from this, I will be particularly pedantic about the way I use the following terms in this book: program Everybody knows what a program is: a series of instructions to the computer which, when executed, cause a specic action to take place. Source les dont t this category: a source program (a term you wont nd again in this book) is really a program source (a le that you can, under the correct circumstances, use to create a program). A program may, however, be interpreted, so a shell script may qualify as a program. So may something like an emacs macro, whether byte compiled or not (since emacs can interpret uncompiled macros directly).

5 February 2005 02:09

Preface

package

A package is a collection of software maintained in a source tree. At various stages in the build process, it will include source les: les that are part of the distribution. auxiliary les, like conguration information and object les that are not part of the source distribution and will not be installed.

installable les: les that will be used after the build process is complete. These will normally be copied outside the source tree so that the source tree can be removed, if necessary. Some software does not require any conversion: you can just install the sources straight out of the box. We wont argue whether this counts as a package. It certainly shouldnt give you any porting headaches. Well use two other terms as well: building and porting. Its difcult to come up with a hardand-fast distinction between the twowell discuss the terms in Chapter 1, Introduction.

Acknowledgements
Without software developers all over the world, there would be nothing to write about. In particular, the Free Software Foundation and the Computer Sciences Research Group in Berkeley (now defunct) have given rise to an incredible quantity of freely available software. Special thanks go to the reviewers Larry Campbell and Matt Welsh, and particularly to James Cox, Jerry Dunham, and Jrg Micheel for their encouragement and meticulous criticism of what initially was just trying to be a book. Thanks also to Clive King of the University of Aberystwyth for notes on data types and alignment, Steve Hiebert with valuable information about HP-UX, and Henry Spencer and Jeffrey Friedl for help with regular expressions. Finally, I cant nish this without mentioning Mike Loukides and Andy Oram at OReilly and Associates, who gently persuaded me to write a book about porting, rather than just presenting the reader with a brain dump.

5 February 2005 02:09

1
Introduction
One of the features that made UNIX successful was the ease with which it could be implemented on new architectures. This advantage has its down side, which is very evident when you compare UNIX with a single-platform operating system such as MS-DOS: since UNIX runs on so many different architectures, it is not possible to write a program, distribute the binaries, and expect them to run on any machine. Instead, programs need to be distributed in source form, and installation involves compiling the programs for the target hardware. In many cases, getting the software to run may be signicantly more than just typing make.

What is porting?
Its difcult to make a clear distinction between porting and building. In this book, well use three terms: building a package is the planned process of creating an installable software package. This is essentially the content of Chapter 5, Building the package. installation is the planned process of putting an installable software package where users can use it. This is what we talk about in Chapter 9, Installation. Some people use the term porting to describe a software installation requiring undocumented changes to adapt it to a new environment, not including the process of conguration if this is intended to be part of the build process. Although this is a useful denition, it contains an element of uncertainty: when you start, you dont know whether this is going to be a build or a port. Its easier to call the whole process porting, whether you just have to perform a simple build or complicated modications to the source. Thats the way well use the term in this book.

The effort required to port a package can vary considerably. If you are running a SparcStation and get some software developed specically for SparcStations, and the software does not offer much in the way of conguration options, you probably really can get it to run by reading the sources onto disk, and typing make and make install. This is the exception, however, not the rule. Even with a SparcStation, you might nd that the package is written for a different release of the operating system, and that this fact requires signicant modications. A more typical port might include getting the software, conguring the package, building the
1

5 February 2005 02:09

package, formatting and printing the documentation, testing the results and installing les in the destination directories.

How long does it take?

It is very difcult to gauge the length of time a port will take to complete. If a port takes a long time, its not usually because of the speed of the machine you use: few packages take more than a few hours to compile on a fast workstation. Even the complete X11R6 windowing system takes only about 4 hours on a 66 MHz Intel 486 PC. The real time-consumers are the bugs you might encounter on the way: if youre unlucky, you can run into big trouble, and you may nd yourself getting to know the package youre porting much more intimately than you wish, or even having to nd and x bugs. Probably the easiest kind of program to port is free software, that is to say, software that is freely redistributable. As a result of the ease of redistribution, it tends to be ported more frequently and to more platforms, so that conguration bugs get ironed out more evenly than in commercial software. Porting a product like bison* from the Free Software Foundation is usually just a matter of minutes:
$ configure checking how to run the C preprocessor ... messages from configure $ make ... messages from make $ make install

On an Intel 486/66, congure runs for 15 seconds, make runs for about 85 seconds, and make install runs for about 5 secondsall in all, less than two minutes. If everything were that simple, nobody would need this book. On the other hand, this simple view omits a point or two. bison comes with typeset documentation. Like most products of the Free Software Foundation, it is written in texinfo format, which relies on TEX for formatting. It doesnt get formatted automatically. In fact, if you look for the target in the Makele, youll nd that there isnt one: the Makele ignores printed documentation. I consider this a bug in the Makele. Never mind, its easy enough to do it manually:
$ tex bison.texinfo tex: not found

This is a fairly typical occurrence in porting: in order to port a package, you rst need to port three other, more complicated packages. In fact, most ports of bison are made in order to compile some other product, such as the GNU C compiler. In order to get our documentation printed, we rst need to port TEX, which is appropriately depicted in its own printed documentation as a shaggy lion. This is denitely a non-trivial port: TEX consists of dozens of different parts, the source tree varies greatly depending on where you get it from, the whole thing is written in Web, Donald Knuths own private dialect of Pascal, and once you get it to run you
* bison is a parser generator, compatible with yacc.

5 February 2005 02:09

Chapter 1: Introduction

discover that the output (deliberately) does not match any printer available, and that you need a so-called printer driver to output it to your favourite laser printeryet another port. Under these circumstances, it wouldnt be surprising if you give up and rely on the online documentation supplied with bison. bison has two different online reference documents: a man page and something called info, a cross-linked documentation reader from the Free Software Foundation. The man page is two pages long, the info runs to over 200K in ve les. There are no prizes for guessing where the real information is. But how do you run info? Simple: you port the GNU texinfo package. This time its not quite as bad as porting TEX, but its still more difcult than porting bison. This scenario is fairly typical: you set out to port something simple, and everything seems to be ne, and then you nd that a minor part of the port can really take up lots of time. Typically, this is the point where most people give up and make do with what they have achieved. This book is intended to help you go the whole distance.

Why we need to port

There are three main reasons why a port might be more than a simple recompilation: Different operating system. Depending on what features the operating system offers, the program may need to be modied. For example, when porting a program from UNIX to DOS, I will denitely have to do something about le naming conventions. If I port a System V.4 program to BSD I may nd I need to replace STREAMS calls with sockets calls. Different hardware. This is obvious enough with something like a display driver. If the driver you have is designed for a Sun workstation and youre porting it to a PC, you will be involved in some serious rewriting. Even in more mundane circumstances, things like the kind of CPU involved might inuence the program design. Local choices. These includes installation pathnames and cooperation with other installed software. For example, if I use the emacs editor, I may choose to use the etags program to cross-reference my source les; if I use vi, I would probably prefer to use ctags. Depending on the C compiler, I may need to use different compilation options. In many cases, this seems to be similar to the choice of operating system, but there is a signicant difference: in general, changing your kernel means changing your operating system. You can change the C compiler or even the system library without changing the basic system.

Unix flavours
UNIX spent the rst ten years of its existence as the object of computer science research. Developed in Bell Labs (part of AT&T), it was signicantly extended in the University of California at Berkeley (UCB), which started releasing signicant updates, the so-called Berkeley Software Distribution (BSD) in 1977. By the time AT&T decided to commercialize UNIX with System III in the early 80s, the fourth BSD was already available, and both System III and System V drew heavily from it. Nevertheless, the differences were signicant, and

5 February 2005 02:09

despite the advent of System V.4, which basically just added all features available in any UNIX dialect into one package, the differences remain. A good overview of the relationship between the Unixes can be found on page 5 of The Design and the Implementation of the 4.3BSD UNIX Operating System by Sam Lefer, Kirk McKusick, Mike Karels and John Quarterman. In this book I will concentrate on the differences that can be of importance when porting from one avour to another.

Research UNIX
Research UNIX is the original UNIX that has been developed inside Bell Labs since 1969. The last version that became widely available was the Seventh Edition, in 1978. This version can be considered the granddaddy of them all*, and is also frequently called Version 7. In this book, Ill make frequent references to this version. Work on Research UNIX continued until 1993, by which time it had reached the Tenth Edition. Its unlikely that youll have much to do with it directly, but occasionally ideas from Research UNIX trickle into other avours.

Berkeley UNIX (BSD)

The rst Berkeley Software Distribution was derived from the 6th edition in 1977 and ran on PDP-11s only. 2BSD was the last PDP-11 version: 2.11BSD is still available for PDP-11s, if you have a need (and a UNIX source licence). 3BSD was derived from 2BSD and the 7th edition via a short-lived version called 32Vin 1979. Since then, BSD has evolved relatively free of outside borrowings. With the closure of the Computer Science Research Group in Berkeley in autumn 1993 and the release of 4.4BSD in early 1994, the original BSD line has died out, but the public release of the complete sources will ensure the continued availability of Berkeley UNIX for a long time to come. Current BSD systems include BSD/OS (formerly called BSD/386), 386BSD, NetBSD and FreeBSD. These were all originally ports of the BSD Net-2 tape, which was released in 1991, to the Intel 386 architecture. These ports are interesting because they are almost pure BSD and contain no AT&T licensed code. BSD/OS is a commercial system that costs money and supplies support; the other three are available free of charge. It is not clear how long all three free versions will continue to exist side-by-side. 386BSD may already be dead, and the difference between NetBSD and FreeBSD is difcult to recognize. At the time of writing, current versions of BSD/OS and FreeBSD are based on 4.4BSD, and NetBSD is planning to follow suit.

XENIX
XENIX is a version of UNIX developed by Microsoft for Intel architectures in the early 80s. It was based mainly on the System III versions available at the time, though some ideas from other versions were included and a signicant amount of work was put into making it an easier system to live with. Not much effort was put into making it compatible with other versions of UNIX, however, and so you can run into a few surprises with XENIX. SCO still markets it,
* In fact, a number of UNIX avours, including System V and BSD, can trace their origins back to the Sixth Edition of 1976, but they all benetted from modications made in the Seventh Edition.

5 February 2005 02:09

Chapter 1: Introduction

but development appears to have stopped about 1989.

System V
System V was derived from the 6th and 7th editions via System III, with a certain amount borrowed from 4.0BSD. It has become the standard commercial UNIX, and is currently the only avour allowed to bear the UNIX trademark. It has evolved signicantly since its introduction in 1982, with borrowings from Research UNIX and BSD at several points along the way. Currently available versions are V.3 (SCO Open Desktop) and V.4 (almost everybody else). System V.3 lacked a number of features available in other Unixes, with the result that almost all V.3 ports have borrowed signicantly from other versions, mainly 4.2BSD. The result is that you cant really be sure what you have with System V.3 you need to consult the documentation for more information. In particular, vanilla System V.3 supports only the original UNIX le system, with le names length limited to 14 characters and with no symbolic links. It also does not have a standard data communications interface, though both BSD sockets and System V STREAMS have been ported to it. System V.3.2 is, as its name suggests, a version of System V.3. This version includes compatibility with XENIX system calls. As we saw above, XENIX went its own way for some time, resulting in incompatibilities with System V. These XENIX features should be supported by the kernel from System V.3.2 onwards. SCO UNIX is version V.3.2, and includes STREAMS support. System V.4 is the current version of System V. Previous versions of System V were often criticized for lacking features. This cannot be said of System V.4: it incorporates System V.3.2 (which already incorporates XENIX), 4.3BSD, and SunOS. The result is an enormous system which has three different ways to do many things. It also still has signicant bugs. Developing software under System V.4 is an interesting experience. Since the semantics of System V.3 and BSD differ in some areas, System V.4 supplies two separate sets of libraries, one with a System V personality and one with a BSD personality. There are no prizes for guessing which is more reliable: unless you really need to, you should use the System V libraries. When we discuss kernel and library differences in Part 2 of the book, the statement This feature is supported by System V.4 will mean that the System V library interface supports it. The statement This feature is supported by BSD also implies that it should be supported by the BSD library interface of System V.4.

OSF/1
OSF/1 is a comparatively recent development in the UNIX market. It was developed by the Open Systems Foundation, an industry consortium formed as a result of dissatisfaction with AT&Ts policy on UNIX. The kernel is based on CMUs Mach operating system, a so-called microkernel*. The original Mach operating system was styled on Berkeley UNIX. OSF/1 attempts to offer the same functionality as System V, though inevitably some incompatibilities
* A microkernel operating system is an operating system that leaves signicant operating system functionality to external components, usually processes. For example, device drivers and le systems are frequently implemented as separate processes. It does not imply that the complete system is any smaller or less functional than the monolithic UNIX kernel.

5 February 2005 02:09

exist.

POSIX.1
POSIX is a series of emerging IEEE standards applying to operating systems, utilities, and programming languages. The relevant standard for operating systems is IEEE 1003.1-1990, commonly called POSIX.1. It has also been adopted by the International Standards Organization (ISO) as standard ISO/IEC 9945.1:1990. POSIX.1 denes the interface between application programs and the operating system, and makes no demands on the operating system except that it should supply the POSIX.1 interface. POSIX.1 looks very much like a subset of UNIX. In fact, most users wouldnt notice the difference. This makes it easy for UNIX operating systems to supply a POSIX.1 interface. Other operating systems might need much more modication to become POSIX.1 compliant. From a UNIX viewpoint, POSIX.1 does not supply as rich a set of functions as any of the commercially available UNIX avours, so programming to POSIX specications can feel somewhat restrictive. This matter is discussed in the POSIX Programmers Guide by Donald Lewine. Despite these slight disadvantages, POSIX has a great inuence on operating system development: all modern avours of UNIX claim to be POSIX-compliant, although the degree of success varies somewhat, and other systems are also attempting to supply a POSIX.1 interface. The trend is clear: future UNIX-like operating systems will be POSIX-compliant, and if you stick to POSIX features, your porting problems will be over. And I have a supply of bridges for sale, rst come, rst served.

Other flavours
It doesnt take much effort to add a new feature to a kernel, and people do it all the time. The result is a proliferation of systems that mix various features of the leading products and additional features of their own. On top of that, the release of kernel sources to the net has caused a proliferation of free operating systems. Systems that you might well run into include: AIX, IBMs name for its UNIX versions. Current versions are based on System V.3, but IBM has stated an intent to migrate to OSF/1 (IBM is a leading member of the OSF). Compared to System V, it has a large number of extensions, some of which can cause signicant pain to the unwary. HP-UX, Hewlett Packards UNIX system. It is based on System V.3, but contains a large number of so-called BSD extensions. Within HP, it is considered to be about 80% BSDcompliant. Linux, a UNIX clone for the Intel 386 architecture written by Linus Torvalds, a student in Helsinki. It has absolutely no direct connection with traditional UNIX avours, which gives it the unique advantage amongst free UNIXes of not being a potential subject for litigation. Apart from that, it has a vaguely System V-like feeling about it. If you are porting to Linux, you should denitely subscribe to the very active network news groups (comp.os.linux.*).

5 February 2005 02:09

Chapter 1: Introduction

SunOS is the generic name of Sun Microsystems operating systems. The original SunOS was derived from 4.2BSD and 4.3BSD, and until release 4.1 it was predominantly BSD-based with a signicant System V inuence. Starting with version 5.0, it is a somewhat modied version of System V.4. These later versions are frequently referred to as Solaris, though this term properly applies to the complete system environment, including windowing system (OpenWindows), development tools and such, and does not apply only to the System V based versions. Solaris 1.x includes the BSD-based SunOS 4.1 as its kernel; Solaris 2.x includes the System V.4-based SunOS 5.x as its kernel. Ultrix is DECs port of 4.1BSD and 4.2BSD to the VAX and MIPS-based workstations. It is now obsolete and has been replaced by OSF/1.

I would have liked to go into more detail about these versions of UNIX, but doing so would have increased the size of the book signicantly, and even then it wouldnt be possible to guarantee the accuracy: most systems add functionality in the course of their evolution, and information that is valid for one release may not apply to an earlier or a later release. As a result, Ive made a compromise: nearly all UNIX features were introduced either in BSD or System V, so I will distinguish primarily between these two. Where signicant differences exist in other operating systemSunOS 4 is a good example I will discuss them separately. Where does this leave you with, say, NonStop UX version B30? NonStop UX version B is a version of UNIX System V.4 that runs on Tandems Integrity series of fault-tolerant MIPSbased UNIX systems. It includes some additional functionality to manipulate the hardware, and some of the header les differ from the standard System V.4. In addition, it includes a minimal carry-over of BSDisms from the System V.3 version. Obviously, you can start by treating it as an implementation of System V.4, but occasionally you will nd things that dont quite seem to t in. Since its a MIPS-based system, you might try to consider it to be like SGIs IRIX operating system version 5, which is System V.4 for SGIs MIPS-based hardware. Indeed, most IRIX 5.x binaries will also run unchanged on NonStop UX version B, but you will notice signicant differences when you try to port packages that already run on IRIX 5.x. These differences are typical of a port to just about every real-life system. There are very few pure System V.4 or pure BSD systems out thereeverybody has added something to their port. Ultimately, you will need to examine each individual problem as it occurs. Here is a strategy you can use to untangle most problems on UNIX systems: Interpret the error messages to gure out what feature or function call is causing the problem. Typically, the error message will come from the compiler and will point to a specic line in a specic le. Look up the feature or call in this book. Use the description to gure out what the original programmer intended it to do. Figure out how to achieve the same effect on your own system. Sometimes, I recommend a change which you can make and try the program again. If youre not sure how your system works, you can probably nd a manual page for the feature or call, and this book will help you interpret it.

5 February 2005 02:09

Recongure or change the code as necessary, then try building again.

Where you fit in

The effort involved in porting software depends a lot on the package and the way it is maintained. It doesnt make much difference whether the software is subject to a commercial license or is freely available on the net: the people who write and maintain it can never hope to port it to more than a fraction of the platforms available. The result is that there will always be problems that they wont know about. There is also a very good chance that the wellknown and well-used package you are about to port may never have been ported quite that way before. This can have some important consequences: You may run into bugs that nobody has ever seen before in a well-known and well-used package. The package that you ported in ten minutes last year and have been using ever since has been updated, and now you cant get the @&*(&@$( to compile or run.

This also means that if you do run into problems porting a package, your feedback is important, whether or not you can supply a x. If you do supply a x, it should t into the package structure so that it can be included in a subsequent release. To reiterate: it makes very little difference here whether we are talking about free or licensed software. The players involved are different, but the problems are not. In many ways, free software is easier, since there are fewer restrictions in talking about it (if you run into problems porting System V.4, you cant just send the code out on the net and ask for suggestions), and theres a chance that more people will have ported it to more platforms already. Apart from that, everything stays the same.

But can I do it?

Of course, maybe your concern is whether you can do it at all. If youve never ported a program before, you might think that this is altogether too difcult, that youll spend days and weeks of effort and confusion and in the end give it up because you dont understand what is going on, and every time you solve a problem, two new ones spring up in its place. Id like to say Dont worry, with this book nothing can go wrong, but unfortunately things arent always like that. On the other hand, its easy too overestimate the things that can go wrong, or how difcult a port might be. Lets look at the bad news rst: in most cases, you can assume that the worst thing that can happen when you try to port a package is that it wont work, but in some unfortunate cases you may cause your system to panic, especially if you are porting kernel software such as device drivers. In addition, if you are porting system utilities, and they dont work, you could nd that you can no longer perform such essential system functions as starting or shutting down the system. These problems dont occur very often, though, and they should not cause any lasting damage if you religiously back up your system (you do perform regular backups, dont you?).

5 February 2005 02:09

Chapter 1: Introduction

Apart from such possible dangers, there is very little that can go wrong. If you are building a package that has already had been ported to your platform, you should not run into any problems that this book cant help you solve, even if you have negligible background in programming and none in porting.

How to use this book

The way you approach porting depends on how difcult it is. If its a straightforward business, something that has been done dozens of times before, like our example of porting bison above, its just a matter of following the individual steps. This is our approach in the rst part of this book, where we look at the following topics: Getting the software. You might get the sources on tape, on CD-ROM, or by copying them from the Internet. Getting them from this format into a format you can use to compile them may not be as simple as you think. Well look at this subject in Chapter 2, Unpacking the goodies and Chapter 3, Care and feeding of source trees. Congure the package for building. Although UNIX is a relatively well dened operating system, some features are less well dened. For example, there are a number of different ways to perform interprocess communication. Many packages contain alternative code for a number of operating systems, but you still need to choose the correct alternative. People often underestimate this step: it seems simple enough, but in many cases it can be more work than all the rest put together. Conguration is a complicated subject, and various methods have evolved. In Chapter 4, Package conguration, well look at manual conguration, shell scripts, and imake, the X11 conguration solution. Build the package. This is what most people understand by porting. Well look at problems running make in Chapter 5, Building the package, and problems running the C compiler in Chapter 6, Running the compiler. Format and print the documentation, which well investigate in Chapter 7, Documentation. Test the results to make sure that they work. Well look at this in Chapter 8, Testing the package. Well discuss how to do installation correctly, accurately and completely in Chapter 9, Installation. Tidy up after the build. In Chapter 10, Where to go from here, well look at what this entails.

Fortunately, almost no package gives you trouble all the way, but its interesting to follow a port through from getting the software to the nished installation, so as far as is possible Ill draw my examples in these chapters from a few free software packages for electronic mail and Usenet news. Specically, well consider Taylor uucp, the electronic mail reader elm, and C news. In addition, well look at the GNU C compiler gcc, since it is one of the most

5 February 2005 02:09

10

frequently ported packages. Well port them to an Intel 486DX/2-66 machine running BSD/386 Version 1.1.*

Part 2
As long as things go smoothly, you can get through the kind of port described in the rst part of this book with little or no programming knowledge. Unfortunately, things dont always go smoothly. If they dont, you may need to make possibly far-reaching changes to the sources. Part 1 doesnt pay much attention to this kind of modicationthats the topic of part 2 of this book, which does expect a good understanding of programming: In Chapter 11, Hardware dependencies, well look at problems caused by differences in the underlying hardware platform. In the following ve chapters, well look at some of the differences in different UNIX avours. First well look at a number of smaller differences in Chapter 12, Kernel dependencies, then well look at some of the more common problem areas in Chapter 13, Signals, Chapter 14, File systems, Chapter 15, Terminal drivers, and Chapter 16, Timekeeping. Well look at the surprising number of headaches caused by header les in Chapter 17, Header les, and at system library functionality in Chapter 18, Function libraries. Well examine the differences between various avours of the more important tools in Chapter 19, Make, Chapter 20, Compilers, and Chapter 21, Object les and friends.

Finally, there are a number of appendixes: Appendix A, Comparative reference to UNIX data types, describes the plethora of data types that have developed since the advent of ANSI C. Appendix B, Compiler ags, gives you a comparative reference to the compiler ags of many common systems. Appendix C, Assembler directives and ags, gives you a comparative reference to assembler directives and ags. Appendix D, Linker ags, gives you a comparative reference to linker ags. Appendix E, Where to get sources, gives you information on where to nd useful source les, including a number of the packages we discuss in this book.

* With the exception of Taylor uucp, BSD/OS, which at the time was called BSD/386, is supplied with all these packages, so you would only be need to port them if you wanted to modify them or port a new version.

5 February 2005 02:09

Chapter 1: Introduction

11

Preparations
You dont need much to port most packages. Normally everything you needa C compiler, a C library, make and some standard toolsshould be available on your system. If you have a system that doesnt include some of these tools, such as a System V release where every individual program seems to cost extra, or if the tools are so out-of-date that they are almost useless, such as XENIX, you may have problems. If your tools are less than adequate, you should consider using the products of the Free Software Foundation. In particular, the GNU C compiler gcc is better than many proprietary compilers, and is the standard compiler of the Open Software Foundation. You can get many packages directly from the Internet or on CD-ROM. If you are going to be doing any serious porting, I recommend that you get at least the GNU software packages, 4.4BSD Lite, and T X, preferably on CD-ROM. In particular, the GNU software and 4.4BSD Lite contain the E sources to many library functions that may be missing from your system. In addition, many of the GNU packages are available in precompiled binary form from a number of sources. Ill refer to these packages frequently in the text.

5 February 2005 02:09

2
Unpacking the goodies
Before you can start porting, you need to put the sources on disk. We use the term source tree to refer to the directory or hierarchy of directories in which the package is stored. Unpacking the archives may not be as trivial as it seems: software packages are supplied in many different formats, and it is not always easy to recognize the format. In this chapter, well look at how to extract the sources to create the source tree. In Chapter 3, Care and feeding of source trees, well see how the source tree changes in the course of a port, and what you can do to keep it in good shape.

Getting the sources

The standard way to get software sources is on some form of storage medium, such as CDROM or tape. Many packages are also available online via the Internet. The choice is not as simple as it seems:

Software from the Internet

If you have an Internet connection, and if the software is available on the net, its tempting to just copy it across the net with ftp. This may not be the best choice, however. Some packages are very big. The compressed sources of the GNU C compiler, for example, occupy about 6 MB. You cant rely on a typical 56 kb/s line to transfer more than about 2 kilobytes per second.* At this speed, it will take nearly an hour to copy the archives. If youre connected via a SLIP line, it could take several hours. Gaining access to the archive sites is not always trivial: many sites have a maximum number of users. In particular, prep.ai.mit.edu, the prime archive site for gcc, is frequently overloaded, and you may need several attempts to get in. In addition, copying software over the net is not free. It may not cost you money, but somebody has to pay for it, and once you have the software, you need somewhere to store it, so you dont really save on archive media.
* Of course, it should approach 7 kilobytes per second, but network congestion can pull this gure down to a trickle. 13

5 February 2005 02:09

14

Choice of archive medium

If you do choose to get your software on some other medium, you have the choice between CD-ROM and tape. Many archive sites will send you tapes if you ask for them. This may seem like a slow and old-fashioned way to get the software, but the bandwidth is high:* DAT and Exabyte tapes can store 2 GB per tape, so a single tape could easily contain as much software as you can duplicate in a week. In addition, you dont need to make a backup before you start. Software on CD-ROM is not as up-to-date as a freshly copied tape, but its easy to store and reasonably cheap. Many companies make frequent CD editions of the more widely known archive sites for example, Walnut Creek CD-ROM has editions of most commonly known software, frequently pre-ported, and Prime Time Freeware issues a pair of CD-ROMs twice a year with 5 GB of compressed software including lesser-known packages. This can be worth it just to be able to nd packages that you would otherwise not even have known about. If you have already ported a previous version of the package, another alternative is to use diffs to bring the archive up to date. Well look at this on page 29.

Archives
You frequently get pure source trees on CD-ROM, but other media, and also many CD-ROMs, transform the source tree several times: A source tree is usually supplied in an archive, a le containing a number of other les. Like a paper bag around groceries, an archive puts a wrapper around the les so that you can handle them more easily. It does not save any space in fact, the wrapper makes it slightly larger than the sum of its les. Archives make it easier to handle les, but they dont do anything to save space. Much of the information in les is redundant: each byte can have 256 different values, but typically 99% of an archive of text or program sources will consist of the 96 printable ASCII characters, and a large proportion of these characters will be blanks. It makes sense to encode them in a more efcient manner to save space. This is the purpose of compression programs. Modern compression programs such as gzip reduce the size of an archive by up to 90%. If you want to transfer archives by electronic mail, you may also need to encode them to comply with the allowable email character set. Large archives can become very unwieldy. We have already seen that it can take several hours to transfer gcc. If the line drops in this time, you may nd that you have to start the le again. As a result, archives are frequently split into more manageable chunks.

The most common form of archive youll nd on the Internet or on CD-ROM is gzipped tar, a tar archive that has been compressed with gzip. A close second is compressed tar, a tar
* To quote a fortune from the fortune program: Never underestimate the bandwidth of a station wagon full of tapes..

5 February 2005 02:09

Chapter 2: Unpacking the goodies

15

archive that has been compressed with compress. From time to time, youll nd a number of others. In the following sections well take a brief look at the programs that perform these tasks and recover the data.

Archive programs
A number of archive programs are available: tar, the tape archive program, is the all-time favourite. The chances are about 95% that your archive will be in tar format, even if it has nothing to do with tape. cpio is a newer le format that once, years ago, was intended to replace tar. cpio archives suffer from compatibility problems, however, and you dont see them very often. ar is a disk archive program. It is occasionally used for source archives, though nowadays it is almost only used for object le archives. The ar archive format has never been completely standardized, so you get an ar archive from a different machine, you might have a lot of trouble extracting it. Well look at ar formats again in , on page 383. shar is the shell archive program. It is unique amongst archive programs in never using non-printing characters, so shar archives can be sent by mail. You can extract shar archives simply by feeding them to a (Bourne) shell, though it is safer to use a program like unshar.

Living with tar

tar is a relatively easy program to use, but the consequences of mistakes can be far-reaching. In the following sections, well look at how to use tar and how to avoid trouble.

Basic use
When it comes to unpacking software, one or two tar commands can meet all your needs. First, you often want to look at the contents before unpacking. Assuming that the archive is named et1.3.tar, the following command lists the les in the archive:
$ tar tf et1.3.tar et1.3/ et1.3/bell.c pet1.3/bltgraph.c et1.3/BLURB

The t option stands for table of contents, and the f option means use the next parameter in the command (et1.3.tar) as the name of the archive to list. To read in the les that were listed, use the command:
$ tar xfv et1.3.tar et1.3/ et1.3/bell.c pet1.3/bltgraph.c et1.3/BLURB

5 February 2005 02:09

16

The list looks the same, but this time the command actually creates the directory et1.3 if necessary, and then creates the contents. The x option stands for extract, and the f option has the same meaning as before. The v option means verbose and is responsible for generating the list, which gives you the assurance that the command is actually doing something. To bundle some les into an archive, use a command like:
$ tar cvf et1.3.tar et1.3

This command packs everything in the et1.3 directory into an archive named et1.3.tar (which is where we started). The c option stands for create and the v option for verbose. This time, the f means use the next parameter in the command (et1.3.tar) as the archive to create.

Absolute pathnames
Many versions of tar have difculties with absolute pathnames. If you back up a directory /usr/foo, they will only be able to restore it to exactly this directory. If the directory is /usr/bin, and youre trying to restore programs like sh, this could give you serious problems. Some versions of tar have an option to ignore the leading /, and others, such as GNU tar, ignore it unless you tell them otherwise.

Symbolic links
Many versions of tar will only back up a symbolic link, not the le or directory to which it points. This can be very embarrassing if you send somebody a tape with what should be a complete software package, and it arrives with only a single symbolic link.

Tape block size

Many DDS (DAT) drives work better with high blocking factors, such as 65536 bytes per block (128 tape blocks). You can do this with the option b (block size):
$ tar cvfb /dev/tape 128 foo-dir

Unfortunately, this can cause problems too. Some DDS drives cannot read tapes with block sizes of more than 32768 bytes, and some versions of tar, such as SGI IRIS 5.x, cannot handle tapes blocked larger than 20 tape blocks (10240 bytes). This is a show-stopper if you have a tape which is really blocked at more than this size: you just wont be able to read it directly. You can solve this problem by installing GNU tar or piping the archive through dd:
$ dd if=/dev/rmt/ctape0 ibs=128b obs=2b | tar xvf -

File names
Most versions of tar perform lename matching based on the exact text as it appears on the tape. If you want to extract specic les, you must use the names by which they are known in the archive. For example, some versions of tar may end up writing absolute names with two leading slashes (like //usr/bin/sh, for example). This doesnt worry the operating system, which treats multiple leading slashes the same as a single leading slash, but if you want to

5 February 2005 02:09

Chapter 2: Unpacking the goodies

17

extract this le, you need to write:

$ tar x //usr/bin/sh

File name sorting

A tar archive listing with tar tv deliberately looks very much like a listing done with ls -l. There is one big difference, however: ls -l sorts the le names by name before displaying them, whereas tar, being a serial archive program, displays the names in the order in which they occur in the archive. The list may look somewhat sorted, depending on how the archive was created, but you cant rely on it. This means that if you are looking for a le name in an archive, you should not be misled if its not where you expect to nd it: use tools like grep or sort to be sure.

tar: dir - cannot create

With System V systems, you may see things like:
$ tar xvf shellutils-1.9.4.tar tar: shellutils-1.9.4/ - cannot create x shellutils-1.9.4/COPYING, 17982 bytes, 36 tape blocks x shellutils-1.9.4/COPYING.LIB, 25263 bytes, 50 tape blocks tar: shellutils-1.9.4/lib/ - cannot create x shellutils-1.9.4/lib/Makefile.in, 2868 bytes, 6 tape blocks x shellutils-1.9.4/lib/getopt.h, 4412 bytes, 9 tape blocks

This bug has been around so long that you might suspect that it is an insider joke. In fact, it is a benign compatibility problem. The POSIX.2 standard tar format allows archives to contain both directory and le names, although the directory names are not really necessary: assuming it has permission, tar creates all directories necessary to extract a le. The only use of the directory names is to specify the modication time and permissions of the directory. Older versions of tar, including System V tar, do not include the directory names in the archive, and dont understand them when they nd them. In this example, we have extracted a POSIX.2 tar archive on a System V system, and it doesnt understand (or need) the directory information. The only effect is that the directories will not have the correct modication timestamps and possibly not the correct permissions.

Losing access to your files

Some versions of tar, notably System V versions, have another trick in store: they restore the original owner of the les, even if that owner does not exist. That way you can lose access to your les completely if they happen to have permissions like rw-------. You can avoid this by using the o ag (restore ownership to current user). It would be nice to be able to say make a rule of always using the o ag. Unfortunately, other versions of tar dene this ag differently check your man pages for details.

5 February 2005 02:09

18

Multivolume archives
tar can also handle multi-volume archives, in other words archives that go over more than one tape. The methods used are not completely portable: one version of tar may not be able to read multivolume archives written by a different version. Some versions of tar just stop writing data at the end of one tape and continue where they left off at the beginning of the next reel, whereas others write header information on the second tape to indicate that it is a continuation volume. If possible, you should avoid writing multivolume archives unless you are sure that the destination system can read them. If you run into problems with multivolume archives you cant read, you might save the day with something like:
$ (dd if=$TAPE ++ echo 1>&2 Change tapes and press RET ++ read confirmation the name of the variable isnt important ++ dd if=$TAPE ++ echo 1>&2 Change tapes and press RET ++ read confirmation ++ dd if=$TAPE) | tar xvf -

This uses dd to copy the rst tape to stdout, then prints a message and waits for you to press the enter key, copies a second tape, prompts and waits again, and then copies a third tape. Since all the commands are in parentheses, the standard output of all three dd commands is piped into the tar waiting outside. The echo commands need to go to stderr (thats the 1>&2) to get displayed on the terminalotherwise they would be piped into the tar, which would not appreciate it. This only works if the version of tar you use doesnt put any header information (like reel number and a repeat of the le header) at the beginning of the subsequent reels. If it does, and you cant nd a compatible tar to extract it again, the following method may help. Assuming a user of an SCO system has given you a large program foo spread over 3 diskettes, each of which contains header information that your tar doesnt understand, you might enter
$ $ $ $ $ $ $ $ tar x foo mv foo foo.0 tar x foo mv foo foo.1 tar x foo mv foo foo.2 cat foo.* >foo rm foo.* extract first part from first floppy save the first part extract second part from second floppy save the second part extract third part from third floppy save the third part concatenate them and remove the intermediate files

Extracting an archive with tar

Using tar to extract a le is normally pretty straightforward. You can cause a lot of confusion, however, if you extract into the wrong directory and it already contains other les you want to keep. Most archives contain the contents of a single directory as viewed from the parent directory in other words, the name of the directory is the rst part of all le names. All GNU software follows this rule:

5 February 2005 02:09

Chapter 2: Unpacking the goodies

$ tar tvf groff-1.09.tar drwxr-xr-x jjc/staff 0 Feb 19 drwxr-xr-x jjc/staff 0 Feb 19 -rw-r--r-- jjc/staff 607 Sep 21 -rw-r--r-- jjc/staff 1157 Oct 30 -rw-r--r-- jjc/staff 1377 Aug 3 -rw-r--r-- jjc/staff 1769 Aug 10

19

14:15 14:13 12:03 07:38 12:34 15:48

1994 1994 1992 1993 1992 1992

groff-1.09/ groff-1.09/include/ groff-1.09/include/Makefile.sub groff-1.09/include/assert.h groff-1.09/include/cmap.h groff-1.09/include/cset.h

Others, however, show the les from the viewpoint of the directory itselfthe directory name is missing in the archive:
$ tar tvf blaster.tar -rw-r--r-- 400/1 -rw-r--r-- 400/1 -r--r--r-- 400/1 -rw-r--r-- 400/1 -rw-r--r-- 400/1 -rw------- 400/1 -rw-r--r-- 400/1 5666 3638 2117 2420 3408 10247 1722 Feb Feb Feb Feb Feb Feb Feb 14 14 14 14 14 14 14 01:44 01:44 01:44 15:17 01:44 01:44 04:10 1993 1993 1993 1993 1993 1993 1993 README INSTALL LICENSE Makefile sb_asm.s stream.c apps/Makefile

If you have an archive like the rst example, you want to be in the parent directory when you extract the archive; in the second case you need to rst create the directory and then cd to it. If you extract the second archive while in the parent directory, you will face a lot of cleaning up. In addition, there is a good chance that les with names like README, INSTALL and LICENSE may already be present in that directory, and extracting this archive would overwrite them. There are a couple of ways to avoid these problems: Always look at the archive contents with tar t before extracting it. Once you have looked at the archive contents, you can change to the correct directory into which to extract it. In the case of groff above, you might choose a directory name like /mysources*. In the case of blaster, you could create a directory /mysources/blaster and extract into that directory. Alternatively, you can always create a subdirectory and extract there, and then rename the directory. In the rst example, you might create a directory /mysources/temp. After extraction, you might nd that the les were in a directory /mysources/temp/groff-1.09, so you could move them with
$ mv groff-1.09 ..

If they extract directly into temp, you can rename the directory:
$ cd .. $ mv temp groff-1.09

This method may seem easier, but in fact there are a couple of problems with it: You need to choose a directory name that doesnt clash with the real name. Thats why we used the name temp in this example: otherwise it wont be possible to rename the directory in the rst example, since you would be trying to overwrite the directory with one of its own subdirectories.

* A number of shells use the shorthand notation / to refer to your home directory.

5 February 2005 02:09

20

Not all avours of UNIX allow you to move directories.

The command to extract is almost identical to the command to list the archive a clear case for a shell with command line editing:
$ tar tvf groff-1.09.tar list the archive $ tar xvf groff-1.09.tar extract the archive

Frequently your tar archive will be compressed in some way. There are methods for extracting les directly from compressed archives. Well examine these when we look at compression programs on page .

Compression programs
If the archive is compressed, you will need to uncompress it before you can extract les from it. UNIX systems almost invariably use one of three compression formats: compressed les are created with the compress program and extracted with uncompress. They can be up to 70% smaller than the original le. The zcat program will uncompress a compressed le to the standard output. gzipped les are created by gzip and extracted by gunzip. They can be up to 90% smaller than the original le. gunzip will also uncompress compressed or packed les. packed les are obsolete, though you still occasionally see packed man pages. They are created by the pack program and uncompressed by the unpack program. The pcat program will uncompress a packed le to the standard output.

Each of these programs is installed with three different names. The name determines the behavior. For example, gzip is also known as gunzip and zcat:
$ ls -li /opt/bin/gzip /opt/bin/gunzip /opt/bin/zcat 13982 -rwxr-xr-x 3 grog wheel 77824 Nov 5 1993 /opt/bin/gunzip 13982 -rwxr-xr-x 3 grog wheel 77824 Nov 5 1993 /opt/bin/gzip 13982 -rwxr-xr-x 3 grog wheel 77824 Nov 5 1993 /opt/bin/zcat

The -i option to ls tells it to list the inode number, which uniquely identies the le. In this case, you will see that all three names are linked to the same le (and that the link count eld is 3 as a result). You will notice that gzip has also been installed under then name zcat, replacing the name used by compress. This is not a problem, since gzcat can do everything that zcat can do, but it can lead to confusion if you rely on it and one day try to extract a gzipped le with the real zcat.

Encoded files
Most archive programs and all compression programs produce output containing non-printable characters. This can be a problem if you want to transfer the archive via electronic mail, which cannot handle all binary combinations. To solve this problem, the les can be encoded: they are transformed into a representation that contains only printable characters. This has the disadvantage that it makes the le signicantly larger, so it is used only when absolutely

5 February 2005 02:09

Chapter 2: Unpacking the goodies

21

necessary. Two programs are in common use: uuencode is by far the most common format. The companion program uudecode will extract from standard input. btoa format is used to some extent in Europe. It does not expand the le as much as uuencode (25% compared to 33% with uuencode), and is more resistant to errors. You decode the le with the atob program.

Split archives
Many ftp sites split large archives into equal-sized chunks, typically between 256 kB and 1.44 MB (a oppy disk image). Its trivial to combine them back to the original archive: cat will do just that. For example, if you have a set of les base09.000 through base09.013 representing a gzipped tar archive, you can combine them with:
$ cat base09.* > base09.tar.gz

This will, of course, require twice the amount of storage, and it takes time. Its easier to extract them directly:
$ cat base09.* | gunzip | tar xvf drwxr-xr-x root/wheel 0 Aug 23 06:22 1993 ./sbin/ -r-xr-xr-x bin/bin 106496 Aug 23 06:21 1993 ./sbin/chown -r-xr-xr-x bin/bin 53248 Aug 23 06:21 1993 ./sbin/mount_mfs ... etc

cat pipes all archives in alphabetical le name order to gunzip. gunzip uncompresses it and pipes the uncompressed data to tar, which extracts the les.

Extracting a linked file

tar is clever enough to notice when it is backing up multiple copies of a le under different names, in other words so-called hard links. When backing up, the rst time it encounters a le, it copies it to the archive, but if it encounters it again under another name, it simply creates an entry pointing to the rst le. This saves space, but if you just try to extract the second le, tar will fail: in order to extract the second name, you also need to extract the le under the rst name that tar found. Most versions of tar will tell you what the name was, but if you are creating archives, it helps to back up the most-used name rst.

Whats that archive?

All the preceding discussion assumes that you know the format of the archive. The fun begins when you dont. How do you extract it? Your primary indication of the nature of the le is its lename. When archives are created, compressed and encoded, they usually receive a le name sufx to indicate the nature of the le. You may also have come across the term extension, which comes from the MS-DOS world. These sufxes accumulate as various steps proceed. A distribution of gcc might come in a le called gcc-2.5.8.tar.gz.uue. This name gives you the following information:

5 February 2005 02:09

22

The name of the package: gcc. The revision level: -2.5.8. You would expect the name of the root directory for this package to be gcc-2.5.8. The archive format: .tar. Since this is a GNU package, you can expect the name of the uncompressed archive to be gcc-2.5.8.tar. The compression format: .gz (gzip format). The name of the compressed archive would be gcc-2.5.8.tar.gz. The encoding format: .uue (encoded with uuencode).

Some operating systems, notably System V.3 and Linux, still provide le systems which restrict le names to 14 characters. This can lead to several problems.* Archives distributed for these systems frequently use variants on these names designed to make them shorter; gcc-2.5.8.tzue might be an alternate name for the same package. The following table gives you an overview of archive le sufxes you might encounter. Well look at source le sufxes in Chapter 20, Compilers, page
Table 21: Common le name sufxes

Name sufx # ,v .a .arc .arj .cpio .diff .gif .gz .hqx .jpg .lzh .orig .rej .shar .sit .tar .uu

Format Alternate patch reject le name. emacs backup les, also used by some versions of patch. RCS le. Created by ci, extracted by co. ar format. Created by and extracted with ar. Created by and extracted with arc. DOS arj format Created by and extracted with cpio. Difference le, created by diff, can be applied by patch. Graphics Interchange Format gzip format. Created by gzip, extracted with gunzip. HQX (Apple Macintosh) JPEG (graphics format) LHa, LHarc, Larc Original le after processing by patch. patch reject le. Shell archive: created by shar, extracted with any Bourne-compatible shell. Stuff-It (Apple Macintosh) tar format. Created by and extracted with tar. uuencoded le. Created by uuencode, decoded with uudecode.

* If you have one of these systems, and you have a choice of le systems, you can save yourself a lot of trouble by installing one that allows long le names.

5 February 2005 02:09

Chapter 2: Unpacking the goodies Table 21: Common le name sufxes (continued)

23

Name sufx .uue .Z .z .zip .zoo

Format Alternative for .uu Compressed with compress, uncompressed with uncompress, zcat or gunzip. Two different formats: either pack format, compressed by pack, extracted with pcat, or old gzip format, compressed by gzip, extracted with gunzip. Zip (either PKZip or Zip/Unzip) Zoo

Identifying archives
Occasionally youll get an archive whose name gives you no indication of the format. Under these circumstances, nding the kind of archive can be a matter of trial and error, particularly if it is compressed. Here are a couple of ideas that might help:

file
The UNIX le command recognizes a lot of standard le types and prints a brief description of the format. Unfortunately, the le really needs to be a le: le performs some le system checks, so it cant read from standard input. For example,
$ file * 0install.txt: English text base09.000: gzip compressed data - deflate method , original file name , last modified: Mon Aug 23 07:53:21 1993 , max compression os: Unix base09.001: data ...more of same base09.011: DOS executable (COM) man-1.0.cpio: cpio archive tcl7.3.tar.gz: empty tex: directory tk3.6.tar: POSIX tar archive

The information for base09.000 was one output line that wrapped around onto 3 output lines. Most les have certain special values, so-called magic numbers, in specic locations in their headers. le uses a le, usually /etc/magic, which describes these formats. Occasionally it makes a mistakewe can be reasonably sure that the le base09.011 is not a DOS executable, but it has the right number in the right place, and thus fools le. This version of le (from BSD/OS) recognizes base09.000and none of the following pieces of the archive as a gzip archive le, and even extracts a lot of information. Not all versions of le do this. Frequently, it just tells you that the archive is data in this case, the rst assumption should be that the archive is compressed in a format that your version of le doesnt recognize. If the le is packed, compressed or gzipped, gzip expands it, and otherwise it prints an error message, so the next step might look something like:

5 February 2005 02:09

24
$ gunzip < mystery > /tmp/junk $ aha! it didnt complain $ file /tmp/junk /tmp/junk: POSIX tar archive

In this case, we have established that the le mystery is, in fact, a compressed tar archive, though we dont know what kind of compression, since gzip doesnt tell. If le tells you that the le is ASCII or English text, then you can safely look at it with more or less:
$ more strange-file Newsgroups: comp.sources.unix From: clewis@ferret.ocunix.on.ca (Chris Lewis) Subject: v26i014: psroff 3.0, Patch09 Sender: unix-sources-moderator@pa.dec.com Approved: vixie@pa.dec.com Submitted-By: clewis@ferret.ocunix.on.ca (Chris Lewis) Posting-Number: Volume 26, Issue 14 Archive-Name: psroff3.0/patch9 This is official patch 09 for Psroff 3.0. ... intervening lines skipped clewis@ferret.ocunix.on.ca (Chris Lewis)

Patchwrapped: 920128230528 Index: ./lib/lj3.fonts *** /tmp/PATCHold/./lib/lj3.fonts Tue Jan 28 23:03:45 1992 --- ./lib/lj3.fonts Tue Jan 28 23:03:46 1992

This is a plain text patch le: you can pass it straight through the patch program, since patch doesnt worry about junk at the beginning or the end of the le. Well look at patch in depth in Chapter 3, Care and feeding of source trees, page 30.
Newsgroups: comp.sources.unix From: lm@Sunburn.Stanford.EDU (Larry McVoy) Subject: v26i020: perfmon - interface to rstatd(8) Sender: unix-sources-moderator@pa.dec.com Approved: vixie@pa.dec.com ... more stuff omitted #! /bin/sh # This is a shell archive. Remove anything before this line, # then unpack it by saving it into a file and typing "sh file".

As the text tells you, this is a shell archive. To extract it, you can remove all text up to the line starting with #!/bin/sh and extract it with the Bourne shell, or pass it through unshar as it is.
begin 666 magic.gz MXL("_!NRTV5A<W1E<@!-4KV.VS,WO,4W(N;:\9:B+3)T.*1HT*DH M<+3$V+I(HB*2?/V)14W=YMED-\OGW8HE0K0.#[![V/A!4B<(M4_>1C>ZTS MNW&$:<D5>!J9_(0\@:@C?SJ#SU@]IP7V&4L6V=TOAF?Y[N%C#U\@D0B. M!%/PGK+NV[)A\/!*KH)C3[:,!<>"R9T<<KGZC3Z4K9*VUE&B.O"C?H&Q4 MA+,8C"(I2&&/((7&H?![;JX4O0?X]$Y)!\HR3\%U.FT(TE#I>#0YE$*M

5 February 2005 02:09

Chapter 2: Unpacking the goodies

MU$C>%#UPT>&L?WY\ZQKNU_[_S</SN@1226061"15.!K);DF4#4RHFD7 M2;/R8BI/=)5:U*1TMG\W>C=O0PJF]N:(U[L45\B*NIIGPDN%..49+$T%8 MXA7>ZEWS"B;<\3+%O30(.%[%8)TK&<I/O6[6\!M>TPDM"U1+Y3%NXA#K! M28*%RR?MZKA6:NWI5L?&&UM7I1>8,(S05K<!(D+44<N&E$R;OKD%#7!-P M<?66PQR.R73X>E,D0U_"QFUP@YFCJ$&IVST=)2L0:-OH%(QNHF:MMI$>O8 I3#PH#VM<#H4>_]<O$)*>PYU)JPJE7>;*:>5!)4S]9O,/(PQ?IS4#!I end

25

This is a uuencoded le. The rst line contains the word begin, the default security (which you cant change) and the name of the archive (magic.gz). The following lines usually have the same length and begin with the same letter (usually M)this is the encoded length specication for the line. If they dont, something has probably gone wrong in the transmission. The last data line is usually shorter, and thus has a different rst character. Finally, the archive contains two end lines: the rst is usually the single character , and the second is the word end on a line by itself. To extract the le, rst pass it through uudecode, which will create the le magic.gz, then gunzip it to create the le magic. Then you might need to use le to nd out what it is.
$ uudecode < magic.uue $ gunzip magic.gz $ file magic magic: English text

Dont confuse uuencode format with this:

xbtoa5 78 puzzle.gz Begin +,C1(V%L;!!?e@F*(u6!)69ODSn.:h/s&KF-$KGlWA8mP,0BTe$Y<$qSODDdUZO:_0iqn&P/S%8H [AX_&!0:k0$N5WjWlkG?U*XLRJ6"1SE;mJ.kEa#$EL9q3*Bb.c9J@t/K/N>62BM=7Ujbp7$YHN ,m"%IZ93t15j%OV"_S#NMI4;GC_N=%+k5LX,A*uli>IBE@i0T4cP/A#coB""a]![8jgS1L=p6Kit X9EU5N%+(>-N=YU4(aeoGoFH9SqM6#c1(r;;K<aBE/aZRX/:.cbh&9[r.f3bpQJQ&fW:*S_7DW9 6No0QkC7@A0?=YtSYlAc@01eeX;bF/9%&4E627AA6GR!u]3?Zhke.l4*T=U@TF9@1Gs4\jQPjbBm\H K24N:$HKre7#7#jG"KFmedjs!<<*"N xbtoa End N 331 14b E 5c S 75b7 R b506b514

This is a btoa encoded le, probably also gzipped like the previous example. Extract it with btoa -a and then proceed as with uuencoded les.

Whats in that archive?

Now you have discovered the format of the archive and can extract the les from it. Theres a possibility, though, that you dont know what the archive is good for. This is frequently the case if you have a tape or a CD-ROM of an ftp server, and it contains some cryptic names that suggest the les might possibly be of interest. How do you nd out what the package does?

README
By convention, many authors include a le README in the main directory of the package. README should tell you at least:

5 February 2005 02:09

26

The name of the package, and what it is intended to do. The conditions under which you may use it.

For example, the README le for GNU termcap reads:

This is the GNU termcap library -- a library of C functions that enable programs to send control strings to terminals in a way independent of the terminal type. Most of this package is also distributed with GNU Emacs, but it is available in this separate distribution to make it easier to install as -ltermcap. The GNU termcap library does not place an arbitrary limit on the size of termcap entries, unlike most other termcap libraries. See the file INSTALL for compilation and installation instructions. Please report any bugs in this library to bug-gnu-emacs@prep.ai.mit.edu. You can check which version of the library you have by using the RCS ident command on libtermcap.a.

In some cases, however, there doesnt seem to be any le to tell you what the package does. Sometimes you may be lucky and nd a good man page or even documentation intended to be printed as hardcopysee Chapter 7, Documentation for more information. In many cases, though, you might be justied in deciding that the package is so badly documented that you give up. There may also be les with names like README.BSD, README.SYSV, README.X11 and such. If present, these will usually give specic advice to people using these platforms.

INSTALL file
There may be a separate INSTALL le, or the information it should contain might be included in the README le. It should tell you: A list of the platforms on which the package has been ported. This list may or may not include your system, but either way it should give you a rst inkling of the effort that lies in store. If youre running System V.4, for example, and it has already been ported to your hardware platform running System V.3, then it should be easy. If it has been ported to V.4, and youre running V.3, this can be a completely different matter. A description of how to congure the package (well look at this in Chapter 4, Package conguration). A description of how to build the package (see Chapter 4, Package conguration and Chapter 19, Make for more details on this subject).

It may, in addition, offer suggestions on how to port to other platforms or architectures.

5 February 2005 02:09

Chapter 2: Unpacking the goodies

27

Other files
The package may include other information les as well. By convention, the names are written in upper case or with an initial capital letter, so that they will be stand out in a directory listing. The GNU project software may include some or all of the following les: ABOUT is an alternative name used instead of README by some authors. COPYING and COPYING.LIB are legal texts describing the constraints under which you may use the software. ChangeLog is a list of changes to the software. This name is hard-coded into the emacs editor macros, so its a good chance that a le with this name will really be an emacsstyle change log. MANIFEST may give you a list of the les intended to be in the package. PROBLEMS may help you if you run into problems. SERVICE is supplied by the Free Software Foundation to point you to companies and individuals who can help you if you run into trouble.

A good example of these les is the root directory of Taylor uucp:

$ gunzip </cd0/gnu/uucp/uucp-1.05.tar.gz |tar tvf drwxrwxr-x 269/15 0 May 6 06:10 1994 uucp-1.05/ -r--r--r-- 269/15 17976 May 6 05:23 1994 uucp-1.05/COPYING -r--r--r-- 269/15 163997 May 6 05:24 1994 uucp-1.05/ChangeLog C$

This archive adheres to the GNU convention of including the name of the top-level directory in the archive. When we extract the archive, tar will create a new directory uucp-1.05 and put all the les in it. So we continue:
$ cd /porting/src the directory in which I do my porting $ gunzip </cd0/gnu/uucp/uucp-1.05.tar.gz |tar xf $

After extraction, the resultant directory contains most of the standard les that we discussed above:
$ cd uucp-1.05 $ ls -l total 1724 drwxrwxr-x 7 drwxrwxrwx 44 -r--r--r-1 -r--r--r-1 -r--r--r-1 -rw-r--r-1 -r--r--r-1 -r--r--r-1 -r--r--r-1 -r--r--r-1

grog grog grog grog grog grog grog grog grog grog

wheel wheel wheel wheel wheel wheel wheel wheel wheel wheel

1536 3584 17976 163997 499 14452 4283 7744 23563 32866

May Aug May May May May May May May May

6 19 6 6 6 6 6 6 6 6

06:10 14:34 05:23 05:24 05:24 06:09 05:24 05:24 05:24 05:24

. .. COPYING ChangeLog MANIFEST Makefile.in NEWS README TODO chat.c

5 February 2005 02:09

28
-r--r--r--rwxrwxr-x -r--r--r-...etc 1 grog 1 grog 1 grog wheel wheel wheel 19032 May 6 05:24 config.h.in 87203 May 6 05:27 configure 11359 May 6 05:24 configure.in

5 February 2005 02:09

3
Care and feeding of source trees
In Chapter 2, Unpacking the goodies, we saw how to create an initial source tree. It wont stay in this form for long. During a port, the source tree is constantly changing: Before you can even start, you may apply patches to the tree to bring it up to date. After unpacking and possibly patching, you may nd that you have to clean out junk left behind from a previous port. In order to get it to compile in your environment, you perform some form of conguration, which modies the tree to some extent. Well look at package conguration in Chapter 4, Package conguration. During compilation, you add many new les to the tree. You may also create new subdirectories. After installation, you remove the unneeded les, for example object les and possibly the nal installed les. After cleaning up, you may decide to archive the tree again to save space on disk.

Modifying the source tree brings uncertainty with it: what is original, what have I modied, how do I remove the changes I have made and get back to a clean, well-dened starting point? In this chapter well look at how to get to a clean starting point. Usually this will be the case after you have extracted the source archive, but frequently you need to add patches or remove junk. Well also look at how to build a tree with sources on CD-ROM, how to recognize the changes you have made and how to maintain multiple versions of your software.

Updating old archives

You dont always need to get a complete package: another possibility is that you might already have an older version of the package. If it is large again, for example, the GNU C compiler you might nd it better to get patches and update the source tree. Strictly speaking, a patch is any kind of modication to a source or object le. In UNIX parlance, its almost always a diff, a le that describes how to modify a source le to produce a newer version. Diffs are almost always produced by the diff program, which we describe in Chapter 10,
29

5 February 2005 02:09

30

Where to go from here, page 144. In our case study, we have gcc version 2.5.6 and want to update to 2.5.8. We discover the following les on the le server:
ftp> ls 200 PORT command successful. 150 Opening ASCII mode data connection for -rw-rw-r-- 1 117 1001 10753 Dec 12 19:15 -rw-rw-r-- 1 117 1001 14726 Jan 24 09:02 -rw-rw-r-- 1 117 1001 5955006 Dec 22 14:16 -rw-rw-r-- 1 117 1001 5997896 Jan 24 09:03 226 Transfer complete. ftp>

/bin/ls. gcc-2.5.6-2.5.7.diff.gz gcc-2.5.7-2.5.8.diff.gz gcc-2.5.7.tar.gz gcc-2.5.8.tar.gz

In other words, we have the choice of copying the two diff les gcc-2.5.6-2.5.7.diff.gz and gcc-2.5.7-2.5.8.diff.gz, a total of 25 kB, and applying them to your source tree, or copying the complete 6 MB archive gcc-2.5.8.tar.gz.

Patch
diff les are reasonably understandable, and you can apply the patches by hand if you want, but its obviously easier and safer to use a program to apply the changes. This is the purpose of patch. patch takes the output of the program diff and uses it to update one or more les. To apply the patch, it proceeds as follows: 1. First, it looks for a le header. If it nds any junk before the le header, it skips it and prints a message to say that it has done so. It uses the le header to recognize the kind of diff to apply. It renames the old le by appending a string to its name. By default, the string is .orig, so foo.c would become foo.c.orig. It then creates a new le with the name of the old le, and copies the old le to the new le, modifying it with the patches as it goes. Each set of changes is called a hunk.

2. 3.

The way patch applies the patch depends on the format. The most dangerous kind are ed style diffs, because there is no way to be sure that the text is being replaced correctly. With context diffs, it can check that the context is correct, and will look a couple of lines in each direction if it doesnt nd the old text where it expects it. You can set the number of lines it will look (the fuzz factor) with the -F ag. It defaults to 2. If the old version of the le does not correspond exactly to the old version used to make the diff, patch may not be able to nd the correct place to insert the patch. Except for ed format diffs, it will recognize when this happens, and will print an error message and move the corresponding hunk to a le with the sufx .rej (for reject). A typical example are the patches for X11R5. You might start with the sources supplied on the companion CD-ROM to X Window System Administrators Guide by Linda Mui and Eric Pearce. This CD-ROM includes the complete X11R5 sources to patch level 21. At the time of writing, ve further patches to X11R5 have been released. To bring the source tree up to patch level 26, you would proceed as follows:

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

31

First, read the header of the patch le. As we have seen, patch allows text before the rst le header, and the headers frequently contain useful information. Looking at patch 22, we see:
$ gunzip < /cd0/x11r5/fix22.gz | more X11 R5 Public Patch #22 MIT X Consortium To apply this patch: cd to the top of the source tree (to the directory containing the "mit" and "contrib" subdirectories) and do: patch -p -s < ThisFile Patch works silently unless an error occurs. You are likely to get the following warning messages, which you can ignore:

In this example we have used gunzip to look at the le directly; we could just as well have used GNU zcat. The patch header suggests the ags -s and -p. The -s ag to patch tells it to perform its work silentlyotherwise it prints out lots of information about what it is doing and why. The -p ag is one of the most complicated to use: it species the pathname strip count, how to treat the directory part of the le names in the header. Well look at it in more detail in the section Cant nd le to patch on page 36. This information is important: patch is rather like a chainsaw without a guard, and if you start it without knowing what you are doing, you can make a real mess of its environment. In this case, we should nd that the root of our source tree looks like:
$ cd /usr/x11r5 $ ls -FC mit Imakefile RELNOTES.ms extensions/ rgb/ LABEL bug-report fonts/ server/ Makefile clients/ hardcopy/ util/ Makefile.ini config/ include/ RELNOTES.PS demos/ lib/ RELNOTES.TXT doc/ man/ ... that looks OK, were in the right place $ gunzip < /cd0/x11r5/fix22.gz | patch -p -s

Weve taken another liberty in this example: since the patch le was on CD-ROM in compressed form, we would have needed to extract it to disk in order to patch the way the le header suggests. Instead, we just gunzip directly into the patch program. Its easy to make mistakes when patching. If you try to apply a patch twice, patch will notice, but you can persuade it to reapply the patch anyway. In this section, well look at the havoc that can occur as a result. In addition, well disregard some of the advice in the patch header. This is the way I prefer to do it:
$ gunzip < /cd0/x11r5/fix23.gz | patch -p &> patch.log

This invocation allows patch to say what it has to say (no -s ag), but copies both the standard output and the error output to the le patch.log, so nothing appears on the screen. You can, of course, pipe the output through the tee program, but in practice things happen so fast

5 February 2005 02:09

32

that any error message will usually run off the screen before you can read it. It certainly would have done so here: patch.log had a length of 930 lines. It starts with
Hmm... Looks like a new-style context diff to me... The text leading up to this was: -------------------------| Release 5 Public Patch #23 | MIT X Consortium ... followed by the complete header |Prereq: public-patch-22

This last line is one safeguard that patch offers to ensure that you are working with the correct source tree. If patch nds a Prereq: line in the le header, it checks that this text appears in the input le. For comparison, heres the header of mit/bug-report:
To: xbugs@expo.lcs.mit.edu Subject: [area]: [synopsis] [replace with actual area and short description]

VERSION: R5, public-patch-22 [MIT public patches will edit this line to indicate the patch level]

In this case, patch nds the text. When it does, it prints out the corresponding message:
| |*** /tmp/,RCSt1006225 Tue Mar 9 14:40:48 1993 |--- mit/bug-report Tue Mar 9 14:37:04 1993 -------------------------Good. This file appears to be the public-patch-22 version.

This message shows that it has found the text in mit/bug-report. The rst hunk in any X11 diff changes this text (in this case to public-patch-23), so that it will notice a repeated application of the patch. Continuing,
Patching file mit/bug-report using Plan A... Hunk #1 succeeded at 2. Hmm... The next patch looks like a new-style context diff to me... The text leading up to this was: -------------------------|*** /tmp/,RCSt1005203 Tue Mar 9 13:45:42 1993 |--- mit/lib/X/Imakefile Tue Mar 9 13:45:45 1993 -------------------------Patching file mit/lib/X/Imakefile using Plan A... Hunk #1 succeeded at 1. Hunk #2 succeeded at 856. Hunk #3 succeeded at 883. Hunk #4 succeeded at 891. Hunk #5 succeeded at 929. Hunk #6 succeeded at 943. Hunk #7 succeeded at 968. Hunk #8 succeeded at 976. Hmm... The next patch looks like a new-style context diff to me...

This output goes on for hundreds of lines. What happens if you make a mistake and try

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

33

again?
$ gunzip < /cd0/x11r5/fix23.gz | patch -p &> patch.log This file doesnt appear to be the public-patch-22 version--patch anyway? [n] y bad choice... Reversed (or previously applied) patch detected! Assume -R? [y] RETURN pressed Reversed (or previously applied) patch detected! Assume -R? [y] RETURN pressed Reversed (or previously applied) patch detected! Assume -R? [y] C$

The rst message is printed because patch didnt nd the text public-patch-22 in the le (in the previous step, patch changed it to read public-patch-23). This message also appears in patch.log. Of course, in any normal application you should immediately stop and check whats gone wrong. In this case, I make the incorrect choice and go ahead with the patch. Worse still, I entered RETURN to the next two prompts. Finally, I came to my senses and hit CTRL-C, the interrupt character on my machine, to stop patch. The result of this is that patch removed the patches in the rst two les (the -R ag tells patch to behave as if the les were reversed, which has the same effect as removing already applied patches). I now have the rst two les patched to patch level 22, and the others patched to patch level 23. Clearly, I cant leave things like this. Two wrongs dont normally make a right, but in this case they do. We do it again, and what we get this time looks pretty much the same as the time before:
$ gunzip < /cd0/x11r5/fix23.gz | patch -p &> mit/patch.log Reversed (or previously applied) patch detected! Assume -R? [y] C$

In fact, this time things went right, as we can see by looking at patch.log:
|*** /tmp/,RCSt1006225 Tue Mar 9 14:40:48 1993 |--- mit/bug-report Tue Mar 9 14:37:04 1993 -------------------------Good. This file appears to be the public-patch-22 version. Patching file mit/bug-report using Plan A... Hunk #1 succeeded at 2. Hmm... The next patch looks like a new-style context diff to me... The text leading up to this was: -------------------------|*** /tmp/,RCSt1005203 Tue Mar 9 13:45:42 1993 |--- mit/lib/X/Imakefile Tue Mar 9 13:45:45 1993 -------------------------Patching file mit/lib/X/Imakefile using Plan A... Hunk #1 succeeded at 1. (lots of hunks succeed) Hmm... The next patch looks like a new-style context diff to me... The text leading up to this was: -------------------------|*** /tmp/d03300 Tue Mar 9 09:16:46 1993 |--- mit/lib/X/Ximp/XimpLCUtil.c Tue Mar 9 09:16:41 1993 -------------------------Patching file mit/lib/X/Ximp/XimpLCUtil.c using Plan A... Reversed (or previously applied) patch detected! Assume -R? [y]

This time the rst two les have been patched back to patch level 23, and we stop before

5 February 2005 02:09

34

doing any further damage.

Hunk #3 failed
Patch makes an implicit assumption that the patch was created from an identical source tree. This is not always the caseyou may have changed something in the course of the port. The differences frequently dont cause problems if they are an area unrelated to the patch. In this example, well look at how things can go wrong. Lets consider the following situation: during a previous port of X11R5 pl 22,* you ran into some problems in mit/lib/Xt/Selection.c and xed them. The original text read:
if (XtWindow(widget) == window) XtAddEventHandler(widget, mask, TRUE, proc, closure); else { Widget w = XtWindowToWidget(dpy, window); RequestWindowRec *requestWindowRec; if (w != NULL && w != widget) widget = w; if (selectWindowContext == 0) selectWindowContext = XUniqueContext();

You had problems with this section, so you commented out a couple of lines:
if (XtWindow(widget) == window) XtAddEventHandler(widget, mask, TRUE, proc, closure); else { /* This doesnt make any sense at all - ignore * Widget w = XtWindowToWidget(dpy, window); */ RequestWindowRec *requestWindowRec; /* if (w != NULL && w != widget) widget = w; */ if (selectWindowContext == 0) selectWindowContext = XUniqueContext();

Back in the present, you try to apply patch 24 to this le:

$ gunzip < /cd0/x11r5/fix24.gz | patch -p &> mit/patch.log $

So far so good. But in patch.log we nd

|*** /tmp/da4854 Mon May 17 18:19:57 1993 |--- mit/lib/Xt/Selection.c Mon May 17 18:19:56 1993 -------------------------Patching file mit/lib/Xt/Selection.c using Plan A... Hunk #1 succeeded at 1. Hunk #2 succeeded at 70. Hunk #3 failed at 361. Hunk #4 succeeded at 1084. Hunk #5 succeeded at 1199. 1 out of 5 hunks failed--saving rejects to mit/lib/Xt/Selection.c.rej

What does this mean? Theres nothing for it but to look at the les concerned. In x24 we nd
* The abbreviation pl is frequently used to mean patch level.

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

*** /tmp/da4854 Mon May 17 18:19:57 1993 --- mit/lib/Xt/Selection.c Mon May 17 18:19:56 1993 *************** *** 1,4 **** this must be hunk 1 ! /* $XConsortium: Selection.c,v 1.74 92/11/13 17:40:46 converse Exp $ */ /*********************************************************** Copyright 1987, 1988 by Digital Equipment Corporation, Maynard, Massachusetts, --- 1,4 ---! /* $XConsortium: Selection.c,v 1.78 93/05/13 11:09:15 converse Exp $ */ /*********************************************************** Copyright 1987, 1988 by Digital Equipment Corporation, Maynard, Massachusetts, *************** *** 70,75 **** --- 70,90 ---this must be hunk 2 Widget w; /* unused */ *************** *** 346,359 **** and this must be hunk 3, the one that failed { Display *dpy = req->ctx->dpy; Window window = req->requestor; ! Widget widget = req->widget; ... etc *************** *** 1068,1073 **** --- 1084,1096 ---hunk 4 *************** *** 1176,1181 **** --- 1199,1213 ---and hunk 5--at least the count is correct

35

patch put the rejects in Selection.c.rej. Lets look at it:

*************** *** 346,359 **** { Display *dpy = req->ctx->dpy; Window window = req->requestor; ! Widget widget = req->widget; if (XtWindow(widget) == window) XtAddEventHandler(widget, mask, TRUE, proc, closure); else { Widget w = XtWindowToWidget(dpy, window); RequestWindowRec *requestWindowRec; if (w != NULL && w != widget) widget = w; if (selectWindowContext == 0) selectWindowContext = XUniqueContext(); if (XFindContext(dpy, window, selectWindowContext,

! -

5 February 2005 02:09

36
--- 361,375 ---{ Display *dpy = req->ctx->dpy; Window window = req->requestor; ! Widget widget = XtWindowToWidget(dpy, window); + + + ! if (widget != NULL) req->widget = widget; else widget = req->widget; if (XtWindow(widget) == window) XtAddEventHandler(widget, mask, False, proc, closure); else { RequestWindowRec *requestWindowRec; if (selectWindowContext == 0) selectWindowContext = XUniqueContext(); if (XFindContext(dpy, window, selectWindowContext,

The characters + and - at the beginning of the lines in this hunk identify it as a unied context diff. Well look at them in more detail in Chapter 10, Where to go from here, page 147. Not surprisingly, they are the contents of hunk 3. Because of our x, patch couldnt nd the old text and thus couldnt process this hunk. In this case, the easiest thing to do is to perform the x by hand. To do so, we need to look at the partially xed le that patch created, mit/lib/Xt/Selection.c. The line numbers have changed, of course, but since hunk 3 wasnt applied, we nd exactly the same text as in mit/lib/Xt/Selection.c.orig, only now it starts at line 366. We can effectively replace it by the after text in Selection.c.rej, remembering of course to remove the indicator characters in column 1.

Cant find file to patch

Sometimes youll see a message like:
$ patch -p <hotstuff.diff &>patch.log Enter name of file to patch:

One of the weaknesses of the combination of diff and patch is that its easy to get the le names out of sync. What has probably happened here is that the le names dont agree with your source tree. There are a number of ways for this to go wrong. The way that patch treats the le names in diff headers depends on the -p ag, the so-called pathname strip count: If you omit the -p ag, patch strips all directory name information from the le names and leaves just the lename part. Consider the following diff header:
*** config/sunos4.h --- config/sunos4.h Wed Feb 29 07:13:57 1992 Mon May 17 18:19:56 1993

Relative to the top of the source tree, the le is in the directory cong. If you omit the -p ag, patch will look for the le sunos4.h, not cong/sunos4.h, and will not nd it. If you specify -p, patch keeps the complete names in the headers. If you specify -pn, patch will remove the rst n directory name components in the pathname. This is useful when the diffs contain incorrect base path names. For example, you

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

37

may nd a diff header which looks like:

*** /src/freesoft/gcc-patches/config/sunos4.h Wed Feb 29 07:13:57 1992 --- /src/freesoft/gcc-patches/config/sunos4.h Mon May 17 18:19:56 1993

Unless your source tree also happens to be called /src/freesoft/gcc-patches, patch wont be able to nd the les if you use the -p ag with no argument. Assuming that you are in the root directory of the package (in other words, the parent directory of cong), you really dont want to know about the /src/freesoft/gcc-patches/ component. This pathname consists of four parts: the leading / making the pathname absolute, and the three directory names src, freesoft and gcc-patches. In this case, you can enter
$ patch -p4 <hotstuff.diff &>patch.log

The -p4 tells patch to ignore the rst four pathname components, so it would read thes lenames just as cong/sunos4.h and cong/sunos4.h. In addition to the problem of synchronizing the path names, you may run into broken diffs which dont specify pathnames, even though the les belong to different directories. Well see how easy it is to make this kind of mistake in Chapter 10, Where to go from here, page . For example, you may nd that the diff headers look like:
*** sunos4.h --- sunos4.h Wed Feb 29 07:13:57 1992 Mon May 17 18:19:56 1993

This kind of diff is a real nuisance: you at least need to search for the le sunos4.h, and if youre unlucky youll nd more than one and have to examine the patches to gure out which one is intended. Then you need to give this name to the prompt, and patch should perform the patches. Unfortunately, in a large collection of diffs, this can happen dozens of times.

I cant seem to find a patch in there

Sometimes you will get what looks like a perfectly good unied context diff, but when you run patch against it, you get a message:
$ patch <diffs Hmm... I cant seem to find a patch in there anywhere. $

Some versions of patch dont understand unied diffs, and since all versions skip anything they dont understand, this could be the result. The only thing for it is to get a newer version of patchsee Appendix E, Where to get sources, for details.

Malformed patch
If patch nds the les and understands the headers, you could still run into problems. One of the most common is really a problem in making the diffs:
$ patch <diffs Hmm... Looks like a unified diff to me... The text leading up to this was: --------------------------

5 February 2005 02:09

38
|--- real-programmers.ms Wed Dec 7 13:17:47 1994 |+++ real-programmers.ms Wed Dec 7 14:53:19 1994 -------------------------Patching file real-programmers.ms using Plan A... Hunk #1 succeeded at 1. Hunk #2 succeeded at 54. patch: **** malformed patch at line 398: No newline at end of file

Well, it tells you what happened: diff will print this message if the last character in a le is not \n. Most versions of patch dont like the message. You need to edit the diff and remove the offending line.

Debris left behind by patch

At the end of a session, patch leaves behind a number of les. Files of the form lename.orig are the original versions of patched les. The corresponding lenames are the patched versions. The length of the sufx may be a problem if you are using a le system with a limited lename length; you can change it (perhaps to the emacs standard sufx ) with the -b ag. In some versions of patch, is the default. If any patches failed, you will also have les called lename.rej (for rejected). These contain the hunks that patch could not apply. Another common sufx for rejects is #. Again, you can change the sufx, this time with the -r ag. If you have any .rej les, you need to look at them and nd out what went wrong. Its a good idea to keep the .orig les until youre sure that the patches have all worked as indicated.

Pruning the tree

Making clean distribution directories is notoriously difcult, and there is frequently irrelevant junk in the archive. For example, all emacs distributions for at least the last 6 years have included a le etc/COOKIES. As you might guess from the name, this le is a recipe for cookies, based on a story that went round Usenet years ago. This le is not just present in the source tree: since the whole subdirectory etc gets installed when you install emacs, you end up installing this recipe as well. This particular directory contains a surprising number of les, some of them quite amusing, which dont really have much to do with emacs. This is a rather extreme case of a common problem: you dont need some of the les on the distribution, so you could delete them. As far as I know, emacs works just as well without the cookie recipe, but in many cases, you cant be as sure. In addition, you might run into other problems: the GNU General Public License requires you to be prepared to distribute the complete contents of the source tree if so requested. You may think that its an accident that the cookie recipe is in the source tree, but in fact its a political statement*, and you are required by the terms of the GNU General Public License to keep the le in order to give it to anybody who might want it.
* To quote the beginning of the le: Someone sent this in from California, and we decided to extend our campaign against information hoarding to recipes as well as software. (Recipes are the closest thing, not involving computers, to software.)

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

39

This is a rather extreme example, but you might nd any of the following in overgrown trees: Old objects, editor backups and core dumps from previous builds. They may or may not go away with a make clean. Test programs left behind by somebody trying to get the thing to work on his platform. These probably will not go away with a make clean. Formatted documentation. Although the Makele should treat documents like objects when cleaning the tree, a surprising number of packages format and install documentation, and then forget about it when it comes to tidying it away again. Old mail messages, only possibly related to the package. I dont know why this is, but mail messages seem to be the last thing anybody wants to remove, and so they continue to exist for years in many trees. This problem seems to be worse in proprietary packages than in free packages.

The old objects are denitely the worst problem: make cant tell that they dont belong to this conguration, and so they just prevent the correct version of the object being built. Depending on how different the architectures are, you may even nd that the bogus objects fool the linker, too, and you run into bizarre problems when you try to execute.

Save the cleaned archive

If you had to go to any trouble (patches or cleanup) to get to a clean starting point for the port, save the cleaned archive. You wont need it again, of course, but Murphys law will ensure that if you dont save it, you will need it again.

Handling trees on CD-ROM

Its convenient to have your source tree on CD-ROM: you save disk space, and you can be sure that you dont accidentally change anything. Unfortunately, you also cant deliberately change anything. Normal Makeles expect to put their objects in the source tree, so this complicates the build process signicantly. In the next two sections, well look at a couple of techniques that address this problem. Both use symbolic links.

Link trees
You can simulate a writeable tree on disk by creating symbolic links to the sources on CDROM. This way, the sources remain on the CD-ROM, but the objects get written to disk. From your viewpoint, it looks as if all the les are in the same directory. For example, assume you have a CD-ROM with a directory /cd0/src/nd containing the sources to nd:
$ ls -FC /cd0/src/find COPYING Makefile COPYING.LIB Makefile.in ChangeLog NEWS config.status* lib/ configure* locate/ configure.in man/

5 February 2005 02:09

40
INSTALL README find/ xargs/

The / at the end of the le names indicate that these les are directories; the * indicates that they are executables. You could create a link tree with the following commands:
$ cd /home/mysrc/find put the links here $ for i in /cd0/src/find/*; do > ln -s $i . > done $ ls -l see what we got total 16 lrwxrwxrwx COPYING -> /cd0/src/find/COPYING lrwxrwxrwx COPYING.LIB -> /cd0/src/find/COPYING.LIB lrwxrwxrwx ChangeLog -> /cd0/src/find/ChangeLog lrwxrwxrwx INSTALL -> /cd0/src/find/INSTALL lrwxrwxrwx Makefile -> /cd0/src/find/Makefile lrwxrwxrwx Makefile.in -> /cd0/src/find/Makefile.in lrwxrwxrwx NEWS -> /cd0/src/find/NEWS lrwxrwxrwx README -> /cd0/src/find/README lrwxrwxrwx config.status -> /cd0/src/find/config.status lrwxrwxrwx configure -> /cd0/src/find/configure lrwxrwxrwx configure.in -> /cd0/src/find/configure.in lrwxrwxrwx find -> /cd0/src/find/find lrwxrwxrwx lib -> /cd0/src/find/lib lrwxrwxrwx locate -> /cd0/src/find/locate lrwxrwxrwx man -> /cd0/src/find/man lrwxrwxrwx xargs -> /cd0/src/find/xargs

I omitted most of the information that is printed by ls -l in order to get the information on the page: what interests us here is that all the les, including the directories, are symbolic links. In some cases, this is what we want: we dont need to create copies of the directories on the hard disk when a single link to a directory on the CD-ROM does it just as well. In this case, unfortunately, thats not the way it is: our sources are in the directory nd, and thats where we will have to write our objects. We need to do the whole thing again for the subdirectory nd:
$ cd mysource/find change to the source directory on disk $ rm find get rid of the directory symlink $ mkdir find and make a directory $ cd find and change to it $ for i in /cd0/src/find/find/*; do > ln -s $i . > done $ ls -l total 18 lrwxrwxrwx Makefile -> /cd0/src/find/find/Makefile lrwxrwxrwx Makefile.in -> /cd0/src/find/find/Makefile.in lrwxrwxrwx defs.h -> /cd0/src/find/find/defs.h lrwxrwxrwx find -> /cd0/src/find/find/find lrwxrwxrwx find.c -> /cd0/src/find/find/find.c lrwxrwxrwx fstype.c -> /cd0/src/find/find/fstype.c lrwxrwxrwx parser.c -> /cd0/src/find/find/parser.c lrwxrwxrwx pred.c -> /cd0/src/find/find/pred.c lrwxrwxrwx tree.c -> /cd0/src/find/find/tree.c

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

lrwxrwxrwx util.c -> /cd0/src/find/find/util.c lrwxrwxrwx version.c -> /cd0/src/find/find/version.c

41

Yes, this tree really does have a directory called nd/nd/nd, but we dont need to worry about it. Our sources and our Makele are here. We should now be able to move back to the top-level directory and perform the make:
$ cd .. $ make

This is a relatively simple example, but it shows two important aspects of the technique: You dont need to create a symlink for every single le. Although symlinks are relatively small in this case, less than 100 bytesthey occupy up to 1024 bytes of disk space per link, and you can easily nd yourself taking up a megabyte of space just for the links. On the other hand, you do need to make all the directories where output from the build process is stored. You need to make symlinks to the existing les in these directories.

An additional problem with this technique is that many tools dont test whether they have succeeded in creating their output les. If they try to create les on CD-ROM and dont notice that they have failed, you may get some strange and misleading error messages later on.

Object links on CD-ROM

Some CD-ROMs, notably those derived from the Berkeley Net/2 release, have a much better idea: the CD-ROM already contains a symlink to a directory where the object les are stored. For example, the FreeBSD 1.1 CD-ROM version of nd is stored on /cd0/lesys/usr/src/usr.bin/nd and contains:
total 106 drwxrwxr-x 2 bin 2048 Oct 28 1993 . drwxrwxr-x 153 bin 18432 Nov 15 23:28 .. -rw-rw-r-1 bin 168 Jul 29 1993 Makefile -rw-rw-r-1 bin 3157 Jul 29 1993 extern.h -rw-rw-r-1 bin 13620 Sep 7 1993 find.1 -rw-rw-r-1 bin 5453 Jul 29 1993 find.c -rw-rw-r-1 bin 4183 Jul 29 1993 find.h -rw-rw-r-1 bin 20736 Sep 7 1993 function.c -rw-rw-r-1 bin 3756 Oct 17 1993 ls.c -rw-rw-r-1 bin 3555 Jul 29 1993 main.c -rw-rw-r-1 bin 3507 Jul 29 1993 misc.c lrwxrwxr-x 1 root 21 Oct 28 1993 obj -> /usr/obj/usr.bin/find -rw-rw-r-1 bin 7766 Jul 29 1993 operator.c -rw-rw-r-1 bin 4657 Jul 29 1993 option.c -rw-rw-r-1 root 2975 Oct 28 1993 tags

All you have to do in this case is to create a directory called /usr/obj/usr.bin/nd. The Makeles are set up to compile into that directory.

5 February 2005 02:09

42

Tracking changes to the tree

The most obvious modication that you make to a source tree is the process of building: the compiler creates object les* and the loader creates executables. Documentation formatters may produce formatted versions of the source documentation, and possibly other les are created as well. Whatever you do with these les, you need to recognize which ones you have created and which ones you have changed. Well look at these aspects in the following sections.

Timestamps
Its easy enough to recognize les that have been added to the source tree since its creation: since they are all newer than any le in the original source tree, the simple command ls -lt (probably piped into more or less) will display them in the reverse order in which they were created (newest rst) and thus separate the new from the old. Every UNIX le and directory has three timestamps. The le system represents timestamps in the time_t format, the number of seconds elapsed since January 1, 1970 UTC. See Chapter 16, Timekeeping, page 270, for more details. The timestamps are: The last modication timestamp, updated every time the le or directory is modied. This is what most users think of as the le timestamp. You can display it with the ls -l command. The last access timestamp, updated every time a data transfer is made to or from the le. You can display it with the ls -lu command. This timestamp can be useful in a number of different places. The status change timestamp (at least, thats what my header les call it). This is a sort of kludged last modication timestamp for the inode, that part of a le which stores information about the le. The most frequent changes which dont affect the other timestamps are change in the number of links or the permissions, which normally isnt much use to anybody. On the other hand, the inode also contains the other timestamps, so if this rule were enforced rigidly, a change to another timestamp would also change the status change timestamp. This would make it almost completely useless. As a result, most implementations suppress the change to the status change timestamp if only the other timestamps are modied. If you want, you can display the status change timestamp with the ls -lc command.

Whichever timestamp you choose to display with ls -l, you can cause ls to sort by it with the -t ag. Thus, ls -lut displays and sorts by the last access timestamp. Of these three timestamps, the last modication timestamp is by far the most important. There are a number of reasons for this:
* To be pedantic, usually the assembler creates the object les, not the compiler. A kludge is a programming short cut, usually a nasty, untidy one. The New Hackers Dictionary goes to a lot of detail to explain the term, including why it should be spelt kluge and not kludge.

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

43

make relies on the last modication timestamp to decide what it needs to compile. If you move the contents of a directory with cp, it changes all the modication timestamps to the time when the copy was performed. If you then type make, you will perform a signicant amount of needless compilation. Its frequently important to establish if two les are in fact the same, in other words, if they have identical content. In the next section well see some programmatic tools that help us with this, but as a rst approximation we can assume that two les with the same name, length and modication timestamp have an identical content, too. The modication timestamp is the most important of these three: you can change the name, but if length and timestamp are the same, theres still a good chance its the same le. If you change the timestamp, you cant rely on the two les being the same just because they have the same name and length. As we have seen above, the last modication timestamp is useful for sorting when you list directories. If youre looking for a le you made the week before last, it helps if it is dated accordingly.

Keeping timestamps straight

Unfortunately, its not as easy to keep timestamps straight. Here are some of the things that can go wrong: If you copy the le somewhere else, traditional versions of cp always set the modication timestamp to the time of copying. ln does not, and neither does mv if it doesnt need to make a physical copy, so either of these are preferable. In addition, more modern versions of cp offer the ag -p (preserve), which preserves the modication timestamp and the permissions. When extracting an archive, cpios default behaviour is to set the modication timestamp to the time of extraction. You can avoid this with the -m ag to cpio. Editing the le changes the modication timestamp. This seems obvious, but you frequently nd that you make a modication to a le to see if it solves a problem. If it doesnt help, you edit the modication out again, leaving the le exactly as it was, except for the modication timestamp, which points to right now. A better strategy is to save the backup le, if the editor keeps one, or otherwise to rename the original le before making the modications, then to rename it back again if you decide not to keep the modications. In a network, its unusual for times to be exactly the same. UNIX machines are not very good at keeping the exact time, and some gain or lose as much as 5 minutes per day. This can cause problems if you are using NFS. You edit your les on one machine, where the clocks are behind, and compile on another, where the clocks are ahead. The result can be that objects created before the last edit still have a modication timestamp that is more recent, and make is fooled into believing that it doesnt need to recompile. Similar problems can occur when one system is running with an incorrect time zone setting.

5 February 2005 02:09

44

cmp
A modication timestamp isnt infallible, of course: even if EOF, timestamp and name are identical, there still can be a lingering doubt as to whether the les really are identical. This doubt becomes more pronounced if you seee something like:
$ ls -l total 503 -rw-rw-rw-rw-rw-rw-rw-rw-rw-rw-rw-rw-rw-rw-rw... etc

1 1 1 1 1

grog grog grog grog grog

wheel wheel wheel wheel wheel

1326 28871 4259 4515 33690

May May May May May

1 1 1 1 1

01:00 01:00 01:00 01:00 01:00

a29k-pinsn.c a29k-tdep.c a68v-nat.c alpha-nat.c alpha-tdep.c

Its a fairly clear bet that somebody has done a touch on all the les, and their modication timestamps have all been set to midnight on May 1.* The cmp program can give you certainty:
$ cmp foo.c ../orig/foo.c compare with the original $ echo $? show exit status 0 0: all OK $ cmp bar.c ../orig/bar.c bar.c ../orig/bar.c differ: char 1293, line 39 $ echo $? show exit status 1 1: they differ

Remember you can tell the shell to display the exit status of the previous command with the shell variable $?. In the C shell, the corresponding variable is called $status. If the contents of the les are identical, cmp says nothing and returns an exit status 0. If they are, it tells you where they differ and returns 1. You can use the exit status in a shell script. For example, the following Bourne shell script (it doesnt work with csh) compares les that are in both the current tree (which is the current working directory) and the original tree (../orig) and makes a copy of the ones that have changed in the directory ../changed.
$ for i in *; do > if [ -f ../orig/$i ]; then > cmp $i ../orig/$i 2>&1 >/dev/null > if [ $? -ne 0 ]; then > cp -p $i ../changed > fi > fi > done check all files in the directory it is present in the orig tree compare them theyre different make a copy

There are a couple of points to note about this example: Were not interested in where the les differ, or in even seeing the message. We just want to copy the les. As a result, we copy both stdout and stderr of cmp to /dev/null, the UNIX bit bucket.

* Midnight? That looks like 1 a.m. But remember that UNIX timestamps are all in UTC, and thats 1 a.m. in my time zone. This example really was done with touch.

5 February 2005 02:09

Chapter 3: Care and feeding of source trees

45

When copying, we use -p to ensure that the timestamps dont get changed again.

An example updating an existing tree

Chances are that before long you will have an old version of gcc on your system, but that you will want to install a newer version. As we saw on page 29, the gzipped archive for gcc is around 6 MB in size, whereas the patches run to 10 KB or 15 KB, so we opt to get diffs from prep.ai.mit.edu to update version 2.6.1 to 2.6.3. Thats pretty straightforward if you have enough disk space: we can duplicate the complete source tree and patch it. Before doing so, we should check the disk space: the gcc source tree with all objects takes up 110 MB of disk space.
$ cd /porting/srcmove to the parent directory $ mkdir gcc-2.6.3 make a directory for the new tree $ cd gcc-2.6.1 move to the old directory $ tar cf - . | (cd ../gcc-2.6.3;tar xf -) and copy all files* $ cd ../gcc-2.6.3 move to new directory $ make clean and start off with a clean slate $ gunzip < /C/incoming/gcc-2.6.1-2.6.2.tar.gz | patch -p | tee patch.log Hmm... Looks like a new-style context diff to me... The text leading up to this was: -------------------------|Changes for GCC version 2.6.2 from version 2.6.1: | |Before applying these diffs, go to the directory gcc-2.6.1. Remove all |files that are not part of the distribution with the command | | make distclean | |Then use the command | | patch -p1 | |feeding it the following diffs as input. Then rename the directory to |gcc-2.6.2, re-run the configure script, and rebuild the compiler. | |diff -rc2P -x c-parse.y -x c-parse.c -x c-parse.h -x c-gperf.h -x cexp.c -x bi-parser.c -x objc-parse.y -x objc-parse.c |-x TAGS -x gcc.?? -x gcc.??s -x gcc.aux -x gcc.info* -x cpp.?? -x cpp.??s -x cpp.aux -x cpp.info* -x cp/parse.c -x cp/pa |rse.h gcc-2.6.1/ChangeLog gcc-2.6.2/ChangeLog |*** gcc-2.6.1/ChangeLog Tue Nov 1 21:32:40 1994 |--- gcc-2.6.2/ChangeLog Sat Nov 12 06:36:04 1994 -------------------------File to patch:

Oops, these patches contain the directory name as well. As the diff header indicates, we can solve this problem by supplying the -p1 ag to patch. We can also solve the problem by
* When moving directories with tar, it may not seem to be important whether you say tar c . or tar c *--but it is. If you say *, you will miss out any le names starting with . (period).

5 February 2005 02:09

46

moving up one level in the directory hierarchy, since we have stuck to the same directory names. This message also reminds us that patch is very verbose, so this time we enter:
$ gunzip < /C/incoming/gcc-2.6.1-2.6.2.tar.gz | patch -p1 -s | tee patch.log 1 out of 6 hunks failed--saving rejects to cccp.c.rej $

What went wrong here? Lets take a look at cccp.c.rej and cccp.c.orig. According to the hunk, line 3281 should be
if (ip->macro != 0)

The hunk wants to change it to

if (output_marks)

However, line 3281 of cccp.orig is

if (output_marks)

In other words, we had already applied this change, probably from a message posted in gnu.gcc.bugs. Although the patch failed, we dont need to worry: all the patches had been applied. Now we have a gcc-2.6.2 source tree in our directory. To upgrade to 2.6.3, we need to apply the next patch:
$ gunzip < /C/incoming/gcc-2.6.2-2.6.3.diff.gz | patch -p1 -s | tee -a patch.log

We use the -a option to patch here to keep both logspossibly overkill in this case. This time there are no errors. After patching, there will be a lot of original les in the directory, along with the one .rej le. We need to decide when to delete the .orig les: if something goes wrong compiling one of the patched les, its nice to have the original around to compare. In our case, though, we have a complete source tree of version 2.6.2 on the same disk, and it contains all the original les, so we can remove the .orig les:
$ find . -name "*.orig" -print | xargs rm

We use xargs instead of -exec rm {} \; because its faster: -exec rm starts a rm process for every le, whereas xargs will put as many le names onto the rm command line as possible. After cleaning up the tree, we back it up. Its taken us a while to create the tree, and if anything goes wrong, wed like to be able to restore it to its initial condition as soon as possible.

5 February 2005 02:09

4
Package conguration
Programs dont run in a vacuum: they interface with the outside world. The view of this outside world differs from location to location: things like host names, system resources, and local conventions will be different. Theoretically, you could change the program sources every time you install a package on a new system, but besides being a pain, its very errorprone. All modern packages supply a method of conguration, a simplied way of adapting the sources to the environment in which the program will run. In this chapter, well look at common conguration conventions. We can divide system differences into one of three categories: The kind of hardware the package will run on. A compiler needs to generate the correct machine instructions, and an X server needs to know how to transfer data to the display hardware. Less well-written programs have hardware dependencies that could have been avoided with some forethought. Well look at this in more detail in Chapter 11, Hardware dependencies. The system software with which it will interact. Differences between UNIX systems are signicant enough that it will be necessary to make certain decisions depending on the system avour. For example, a communications program will need to know what kind of network interface your system has. Programs that come from other systems may need signicant rewriting to conform with UNIX library calls. Well look at these dependencies in part 2 of this book, from Chapter 12, Kernel dependencies to Chapter 21, Object les and friends. The local conguration. These may include obvious things like the system name, aspects of program behaviour, information about tools used locally, or local system conventions.

In this chapter, well look at what local conguration entails, and how we tell the package about our chosen conguration.

47

5 February 2005 02:09

48

Installation paths
Your system conguration may place constraints on where you can install the software. This is not normally a problem for individual systems, but on a large, heterogeneous network it could require more consideration. Traditionally, non-system software has been installed in the hierarchy /usr/local. This is not an sthetically pleasing location: the hierarchy can become quite large, and in a network many systems might share the directory. One of the best thought-out descriptions of a modern le system structure is in the UNIX System V Application Binary Interface, which is also similar to structures used by SunOS and the newer BSD variants. In brief, it species the following top-level directories: / The root directory. /dev The directory tree containing device les. /etc Directory for machine-specic conguration les. /opt Directory for add-on software. /usr This directory used to be the other le system on a UNIX machine. In the System V ABI it has lost most of its importance. The ABI states uses only for /usr/bin and /usr/share, and the name /usr has lost its original meaning: the ABI species /usr only as a location for system les that users may wish to access. /usr/bin is intended for Utility programs and commands for the use of all applications and users. In practice, its better to use this directory for system programs only. /usr/share The System V ABI states that /usr/share is intended for architecture-independent shareable les. In practice, those versions of System V that still have man pages put them in /usr/share/man, and terminfo data are stored in /usr/share/lib/terminfo. The rest of the directory may contain a few other odds and ends, but these two directories make up over 99% of the content. The choice of the location /usr/share is not a happy choice: rstly, it is frequently a separate le system, but it must be mounted on a non-root le system, and secondly the man pages arent really architecture-independent. The choice makes more sense from the point of view of the Unix Systems Group, who are concerned only with pure System V: the man pages are mainly independent of hardware architecture. However, in a real-world net you probably have two or three different operating systems, each with their own man pages. /var This directory contains les that are frequently modied. Typical subdirectories are /var/tmp for temporary les and /var/spool for printer output, uucp and news. The System V ABI does not say anything about where to store user les. The Seventh Edition typically stored them as a subdirectory of /usr, but newer systems have tended to store them in a directory called /home.

5 February 2005 02:09

Chapter 4: Package conguration

49

The /opt hierarchy resembles that of /usr. A typical structure is: /opt/bin for executables. /opt/man for man pages not /opt/share/man, unlike the structure in /usr. /opt/lib for additional les used by executables. In particular, this directory could contain library archives for compilers, as well as the individual passes of the compilers. /opt/<pkg> This is where the System V ABI places individual package data. Not many other systems follow it. /opt/lib/<pkg> This is where most packages place private data. Using the /opt hierarchy has one disadvantage: you may not want to have a separate le system. In modern systems, the solution is simple enough: place the directory where you want it, and create a symbolic link /opt that points to it. This works only if your system has symbolic links, of course, so I have come to a compromise: I use /opt on systems with symbolic links, and /usr/local on systems without symbolic links. Many packages compile pathnames into the code, either because its faster that way, or because its easier. As a result, you should set the path names before compilationdont put off this task until youre ready to install, or you may run into problems where the packages are all nicely installed in the correct place and look for data in the wrong directories.

Preferred tools
Many of the most popular software packages are alternative tools. Free software such as gcc, emacs and perl have become so popular that they are frequently supplied with proprietary system releases, and many other systems have ported them and use them as standard tools. If you want to use such programs, you need to tell the conguration routines about them. Depending on the tools you use, you may also need to change the ags that you pass to them. For example, if you compile with gcc, you may choose to include additional compiler ags such as -fstrength-reduce, which is specic to gcc.

Conveying configuration information

The goal of conguration is to supply the conguration information to the program sources. A good conguration mechanism will hide this from you, but its helpful to understand what its doing. In this section, well look under the covers you can skip it if it looks too technical. There are a number of possible ways to use conguration information: for example, the package may have separate communication modules for STREAMS and sockets, and the conguration routines may decide which of the two modules to compile. More typically, however, the conguration routines convey conguration information to the package by dening preprocessor variables indicating the presence or absence of a specic feature. Many packages provide this information in the make variable CFLAGSfor example, when you make bash, the GNU Bourne Again Shell, you see things like

5 February 2005 02:09

50
$ make gcc -DOS_NAME="FreeBSD" -DProgram=bash -DSYSTEM_NAME="i386" \ -DMAINTAINER="bug-bash@prep.ai.mit.edu" -O -g -DHAVE_SETLINEBUF -DHAVE_VFPRINTF \ -DHAVE_UNISTD_H -DHAVE_STDLIB_H -DHAVE_LIMITS_H -DHAVE_GETGROUPS \ -DHAVE_RESOURCE -DHAVE_SYS_PARAM -DVOID_SIGHANDLER -DOPENDIR_NOT_ROBUST \ -DINT_GROUPS_ARRAY -DHAVE_WAIT_H -DHAVE_GETWD -DHAVE_DUP2 -DHAVE_STRERROR \ -DHAVE_DIRENT -DHAVE_DIRENT_H -DHAVE_STRING_H -DHAVE_VARARGS_H -DHAVE_STRCHR \ -DHAVE_STRCASECMP -DHAVE_DEV_FD -D"i386" -D"FreeBSD" -DSHELL -DHAVE_ALLOCA \ -I. -I. -I././lib/ -c shell.c

The -D arguments pass preprocessor variables that dene the conguration information. An alternative method is to put this information in a le with a name like cong.h. Taylor uucp does it this way: in cong.h you will nd things like:
/* If your compiler supports prototypes, set HAVE_PROTOTYPES to 1. */ #define HAVE_PROTOTYPES 1 /* Set ECHO_PROGRAM to a program which echoes its arguments; if echo is a shell builtin you can just use "echo". */ #define ECHO_PROGRAM "echo" /* The following macros indicate what header files you have. Set the macro to 1 if you have the corresponding header file, or 0 if you do not. */ #define HAVE_STDDEF_H 1 /* <stddef.h> */ #define HAVE_STDARG_H 1 /* <stdarg.h> */ #define HAVE_STRING_H 1 /* <string.h> */

I prefer this approach: you have all the conguration information in one place, it is documented, and its more reliable. Assuming that the Makele dependencies are correct, any change to cong.h will cause the programs to be recompiled on the next make. As we will see in Chapter 5, Building the package, page 68, this usually doesnt happen if you modify the Makele. Typically, conguration information is based on the kind of operating system you run and the kind of hardware you use. For example, if you compile for a Sparc II running SunOS 4.1.3, you might dene sparc to indicate the processor architecture used and sunos4 to indicate the operating system. Since SunOS 4 is basically UNIX, you might also need to dene unix. On an Intel 486 running UnixWare you might need to dene i386 for the processor architecture,* and SVR4 to indicate the operating system. This information is then used in the source les as arguments to preprocessor #ifdef commands. For example, the beginning of each source le, or a general conguration le, might contain:
#ifdef i386 #include "m/i386.h" #endif #ifdef sparc #include "m/sparc.h" #endif

* Why not i486? The processor is an Intel 486, but the architecture is called the i386 architecture. You also use i386 when compiling for a Pentium.

5 February 2005 02:09

Chapter 4: Package conguration

#ifdef sunos4 #include "s/sunos4.h" #endif #ifdef SVR4 #include "s/usg-4.0.h" #endif

51

You can get yourself into real trouble if you dene more than one machine architecture or more than one operating system. Since conguration is usually automated to some extent, the likelihood of this is not very great, but if you end up with lots of double denitions when compiling, this is a possible reason. Conguration through the preprocessor works nicely if the hardware and software both exactly match the expectations of the person who wrote the code. In many cases, this is not the case: looking at the example above, note that the le included for SVR4 is s/usg-4.0.h, which suggests that it is intended for UNIX System V release 4.0. UnixWare is System V release 4.2. Will this work? Maybe. It could be that the conguration mechanism was last revised before System V.4.2 came out. If you nd a le s/usg-4.2.h, its a good idea to use it instead, but otherwise its a matter of trial and error. Most software uses this approach, although it has a number of signicant drawbacks: The choices are not very detailed: for example, most packages dont distinguish between Intel 386 and Intel 486, although the latter has a oating point coprocessor and the former doesnt. There is no general consensus on what abbreviations to use. For UnixWare, you may nd that the correct operating system information is determined by USG (USG is the Unix Systems Group, which, with some interruption,* is responsible for System V), SYSV, SVR4, SYSV_4, SYSV_4_2 or even SVR3. This last can happen when the conguration needed to be updated from System V.2 to System V.3, but not again for System V.4. The choice of operating system is usually determined by just a couple of differences. For example, base System V.3 does not have the system call rename, but most versions of System V.3 that you will nd today have it. System V.4 does have rename. A software writer may use #ifdef SVR4 only to determine whether the system has the rename system call or not. If you are porting this package to a version of System V.3.2 with rename, it might be a better idea to dene SVR4, and not SVR3. Many aspects attributed to the kernel are in fact properties of the system library. As we will see in the introduction to Part 2 of this book, there is a big difference between kernel functionality and library functionality. The assumption is that a specic kernel uses the library with which it is supplied. The situation is changing, though: many companies sell systems without software development tools, and alternative libraries such as the GNU C library are becoming available. Making assumptions about the library based on the kernel was never a good ideanow its completely untenable. For example, the GNU C

* The rst USG was part of AT&T, and was superseded by UNIX Systems Laboratories (USL). After the sale of USL to Novell, USL became Novells UNIX Systems Group.

5 February 2005 02:09

52

library supplies a function rename where needed, so our previous example would fail even on a System V.3 kernel without a rename system call if it uses the GNU C library. As you can imagine, many packages break when compiled with the GNU C library, through their own fault, not that of the library. In the example above, it would make a whole lot more sense to dene a macro HAS_RENAME which can be set if the rename function is present. Some packages use this method, and the GNU project is gradually working towards it, but the majority of packages base their decisions primarily on the combination of machine architecture and operating system. The results of incorrect conguration can be far-reaching and subtle. In many cases, it looks as if there is a bug in the package, and instead of reconguring, you can nd yourself making signicant changes to the source. This can cause it to work for the environment in which it is compiled, but to break it for anything else.

What do I need to change?

A good conguration mechanism should be able to decide the hardware and software dependencies that interest the package, but only you can tell it about the local preferences. For example, which compiler do you use? Where do you want to install the executables? If you dont know the answers to these questions, theres a good chance that youll be happy with the defaults chosen by the conguration routines. On the other hand, you may want to use gcc to compile the package, and to install the package in the /opt hierarchy. In all probability, youll have to tell the conguration routines about this. Some conguration routines will look for gcc explicitly, and will take it if they nd it. In this case, you may have a reason to tell the conguration routines not to use gcc. Some packages have a number of local preferences: for example, do you want the package to run with X11 (and possibly fail if X isnt running)? This sort of information should be in the README le.

Creating configuration information

A number of conguration methods exist, none of them perfect. In most cases you dont get a choice: you use the method that the author of the package decided upon. The rst signicant problem can arise at this point: what method does he use? This is not always easy to gure out it should be described in a le called README or INSTALL or some such, but occasionally you just nd cryptic comments in the Makele. In the rest of this chapter well look at conguration via multiple Makele targets, manual conguration, shell scripts, and imake, the X11 conguration mechanism. In addition, the new BSD make system includes a system of automatic conguration: once it is set up, you dont have to do anything, assuming you already have a suitable Makele. Well look at this method in more detail in Chapter 19, Make, page 323.

5 February 2005 02:09

Chapter 4: Package configuration

53

Multiple Makefile targets

Some packages anticipate every possibility for you and supply a customized Makele. For example, when building unzip, a free uncompression utility compatible with the DOS package PK-ZIP, you would nd:
$ make If youre not sure about the characteristics of your system, try typing "make generic". If the compiler barfs and says something unpleasant about "timezone redefined," try typing "make clean" followed by "make generic2". One of these actions should produce a working copy of unzip on most Unix systems. If you know a bit more about the machine on which you work, you might try "make list" for a list of the specific systems supported herein. And as a last resort, feel free to read the numerous comments within the Makefile itself. Note that to compile the decryption version of UnZip, you must obtain the full versions of crypt.c and crypt.h (see the "Where" file for ftp and mail-server sites). Have an excruciatingly pleasant day.

As the comments suggest, typing make generic should work most of the time. If it doesnt, looking at the Makele reveals a whole host of targets for a number of combined hardware/software platforms. If one of them works for you, and you can nd which one, then this might be an easy way to go. If none does, you might nd yourself faced with some serious Makele rewriting. This method has an additional disadvantage that it might compile with no problems and run into subtle problems when you try to execute itfor example, if the program expects System V sigpause and your system supplies BSD sigpause,* the build process may complete without detecting any problems, but the program will not run correctly, and you might have a lot of trouble nding out why.

Manual configuration
Modifying the Makele or cong.h manually is a better approach than multiple Makele targets. This seemingly arduous method has a number of advantages: You get to see what is being changed. If you have problems with the resultant build, its usually relatively easy to pin-point them. Assuming that the meanings of the parameters are well documented, it can be easier to modify them manually than run an automated procedure that hides much of what it is doing. If you nd you do need to change something, you can usually do it fairly quickly. With an automated script, you may need to go through the whole script to change a single minor parameter.

On the down side, manual conguration requires that you understand the issues involved: you cant do it if you dont understand the build process. In addition, you may need to repeat it every time you get an update of the package, and it is susceptible to error.
* See Chapter 13, Signals, pages 190 and 192 for further information.

5 February 2005 02:09

54

Configuration scripts
Neither multiple Makele targets nor manual modication of the Makele leave you with the warm, fuzzy feeling that everything is going to work correctly. It would be nice to have a more mechanized method to ensure that the package gets the correct information about the environment in which it is to be built. One way to do this is to condense the decisions you need to make in manual conguration into a shell script. Some of these scripts work very well. A whole family of conguration scripts has grown up in the area of electronic mail and news. Heres part of the conguration script for C news, which for some reason is called build:
$ cd conf $ build This interactive command will build shell files named doit.root, doit.bin, doit.news, and again.root to do all the work. It will not actually do anything itself, so feel free to abort and start again. C News wants to keep most of its files under a uid which preferably should be all its own. Its programs, however, can and probably should be owned by another user, typically the same one who owns most of the rest of the system. (Note that on a system running NFS, any program not owned by "root" is a gaping security hole.) What user id should be used for news files [news]? RETURN pressed What group id should be used for news files [news]? RETURN pressed What user id should be used for news programs [bin]? RETURN pressed What group id should be used for news programs [bin]? RETURN pressed Do the C News sources belong to bin [yes]? no You may need to do some of the installation procedures by hand after the software is built; doit.bin assumes that it has the power to create files in the source directories and to update the news programs. It would appear that your system is among the victims of the 4.4BSD / SVR4 directory reorganization, with (e.g.) shared data in /usr/share. Is this correct [yes]? RETURN pressed This will affect where C News directories go. We recommend making the directories wherever they have to go and then making symbolic links to them under the standard names that are used as defaults in the following questions. Should such links be made [yes]? no

We chose not to use the symbolic links: the script doesnt say why this method is recommended, they dont buy us anything, and symbolic links mean increased access time. The conguration script continues with many more questions like this. Well pick it up at various places in the book. The exibility of a shell script is an advantage when checking for system features which are immediately apparent, but most of them require that you go through the whole process from start to nish if you need to modify anything. This can take up to 10 minutes on each occasion, and they are often interactive, so you cant just go away and let it do its thing.

5 February 2005 02:09

Chapter 4: Package configuration

55

GNU package configuration

Most GNU project packages supply another variety of conguration script. For more details, see Programming with GNU Software, by Mike Loukides. GNU conguration scripts sometimes expect you to know the machine architecture and the operating system, but they often attempt to guess if you dont tell them. The main intention of the conguration utility is to gure out which features are present in your particular operating system port, thus avoiding the problems with functions like rename discussed on page 51. Taylor uucp uses this method:
$ sh configure checking how to run the C preprocessor checking whether -traditional is needed checking for install checking for ranlib checking for POSIXized ISC checking for minix/config.h checking for AIX checking for -lseq checking for -lsun checking whether cross-compiling checking for lack of working const checking for prototypes checking if #! works in shell scripts checking for echo program checking for ln -s

see page 351 the install program, page 128 see page Interactive POSIX extensions? MINIX specific IBM UNIX libseq.a needed? libsun.a? see page 339 does the compiler understand function prototypes? is echo a program or a builtin? do we have symbolic links? (page 218)

This method makes life a whole lot easier if the package has already been ported to your particular platform, and if you are prepared to accept the default assumptions that it makes, but can be a real pain if not: You may end up having to modify the conguration scripts, which are not trivial. Its not always easy to congure things you want. In the example above, we accepted the default compiler ags. If you want maximum optimization, and the executables should be installed in /opt/bin instead of the default /usr/local/bin, running congure becomes signicantly more complicated:*
$ CFLAGS="-O3 -g" sh configure --prefix=/opt

The scripts arent perfect. You should really check the resultant Makeles, and you will often nd that you need to modify them. For example, the conguration scripts of many packages, including the GNU debugger, gdb, do not allow you to override the preset value of CFLAGS. In other cases, you can run into a lot of trouble if you do things that the script didnt expect. I once spent a couple of hours trying to gure out the behaviour of the GNU make conguration script when porting to Solaris 2.4:

* This example uses the feature of modern shells of specifying environment variables at the beginning of the command. The program being run is sh, and the denition of CFLAGS is exported only to the program being started.

5 February 2005 02:09

56
$ CFLAGS="O3 -g" configure --prefix=/opt creating cache ./config.cache checking for gcc... gcc checking whether we are using GNU C... yes checking how to run the C preprocessor... gcc -E checking whether cross-compiling... yes

Although this was a normal port, it claimed I was trying to cross-compile. After a lot of experimentation, I discovered that the conguration script checks for cross-compilation by compiling a simple program. If this compilation fails for any reason, the script assumes that it should set up a cross-compilation environment. In this case, I had mistakenly set my CFLAGS to O3 -gof course, I had meant to write -O3 -g. The compiler looked for a le O3 and couldnt nd it, so it failed. The conguration script saw this failure and assumed I was cross-compiling. In most cases, you need to re-run the conguration script every time a package is updated. If the script runs correctly, this is not a problem, but if you need to modify the Makele manually, it can be a pain. For example, gdb creates 12 Makeles. If you want to change the CFLAGS, you will need to modify each of them every time you run congure. Like all conguration scripts, the GNU scripts have the disadvantage of only conguring things they know about. If your man program requires pre-formatted man pages, you may nd that there is no way to congure the package to do what you want, and you end up modifying the Makele after you have built it.

Modifying automatically build Makeles is a pain. An alternative is to modify Makele.in, the raw Makele used by congure. That way, you will not have to redo the modications after each run of congure.

imake
imake is the X11 solution to package conguration. It uses the C preprocessor to convert a number of conguration les into a Makele. Here are the standard les for X11R6: Imake.tmpl is the main conguration le that is passed to the C preprocessor. It is responsible for including all the other conguration les via the preprocessor #include directive. Imake.cf determines the kind of system upon that imake is running. This may be based on preprocessor variables supplied by default to the preprocessor, or on variables compiled in to imake. site.def describes local preferences. This is one of the few les that you should normally consider modifying. As its name implies, <vendor>.cf has a different name for each platform. Imake.tmpl decides which le to include based on the information returned by Imake.cf. For example, on BSD/OS the le bsdi.cf will be included, whereas under SunOS 4 or Solaris 2 the le sun.cf will be included.

5 February 2005 02:09

Chapter 4: Package conguration

57

Imake.rules contains preprocessor macros used to dene the individual Makele targets. Imakele is part of the package, not the imake conguration, and describes the package to imake.

You dont normally run imake directly, since it needs a couple of pathname parameters: instead you have two possibilities: Run xmkmf, which is a one-line script that supplies the parameters to imake. Run make Makele. This assumes that some kind of functinoal Makele is already present in the package.

Strangely, make Makele is the recommended way to create a new Makele. I dont agree: one of the most frequent reasons to make a new Makele is because the old one doesnt work, or because it just plain isnt there. If your imake conguration is messed up, you can easily remove all traces of a functional Makele and have to restore the original version from tape. xmkmf always works, and anyway, its less effort to type. Once you have a Makele, you may not be nished with conguration. If your package contains subdirectories, you may need to create Makeles in the subdirectories as well. In general, the following sequence will build most packages:
$ $ $ $ $ xmkmf make Makefiles make depend make make install run imake against the Imakefile create subordinate Makefiles run makedepend against all Makefiles make the packages install the packages

These commands include no package-dependent parametersthe whole sequence can be run as a shell script. Well, yes, there are minor variations: make Makeles fails if there are no subordinate Makeles to be made, and sometimes you have targets like a make World instead of make or make all, but in general its very straightforward. If your imake conguration les are set up correctly, and the package that you are porting contains no obscenities, this is all you need to know about imake, which saves a lot of time and is good for your state of mind. Otherwise, check Software Portability with imake, by Paul DuBois, for the gory details.

5 February 2005 02:09

5
Building the package
Now we have congured our package and were ready to build. This is the Big Moment: at the end of the build process we should have a complete, functioning software product in our source tree. In this chapter, well look at the surprises that make can have in store for you. You can nd the corresponding theoretical material in Chapter 19, Make.

Preparation
If youre unlucky, a port can go seriously wrong. The rst time that error messages appear thick and fast and scroll off the screen before you can read them, you could get the impression that the packages were built this way deliberately to annoy you. A little bit of preparation can go a long way towards keeping you in control of whats going on. Here are some suggestions:

Make sure you have enough space

One of the most frequent reasons of failure of a build is that the le system lls up. If possible, ensure that you have enough space before you start. The trouble is, how much is enough? Hardly any package will tell you how much space you need, and if it does it will probably be wrong, since the size depends greatly on the platform. If you are short on space, consider compiling without debugging symbols (which take up a lot of space). If you do run out of space in the middle of a build, you might be able to save the day by stripping the objects with strip, in other words removing the symbols from the le.

Use a windowing system

The sheer size of a complicated port can be a problem. Like program development, porting tends to be an iterative activity. You edit a le, compile, link, test, go back to edit the le, and so on. Its not uncommon to nd yourself having to compare and modify up to 20 different les in 5 different directories, not to mention running make and the debugger. In adddition, a single line of output from make can easily be 5000 or 10000 characters long, many times the screen capacity of a conventional terminal.
59

5 February 2005 02:09

60

All of these facts speak in favour of a windowing system such as X11, preferably with a highresolution monitor. You can keep your editor (or editors, if they dont easily handle multiple les) open all the time, and run the compiler and debugger in other windows. If multiple directories are involved, its easier to maintain multiple xterms, one per directory, than to continually change directories. A correctly set up xterm will allow you to scroll back as far as you want I nd that 250 lines is adequate.

Keep a log file

Sooner or later, youre going to run into a bug that you cant x immediately: you will have to experiment a bit before you can x the problem. Like nding your way through a labyrinth, the rst time through you will probably not take the most direct route, and its nice to be able to nd your way back again. In the original labyrinth, Theseus used a ball of string to nd his way both in and out. The log le, a text le describing what youve done, is the computer equivalent of the ball of string, so you should remember to roll it up again. If youre running an editor like emacs, which can handle multiple les at a time, you can keep the log in the editor buffer and remove the notes again when you back out the changes. In addition to helping you nd your way out of the labyrinth, the log will also be of use later when you come to install an updated version of the software. To be of use like this, it helps to keep additional information. For example, here are some extracts from a log le for the gcc:
Platform: Revision: Date ported: Ported by: Compiler used: Library: SCO UNIX System V.3.2.2.0 2.6.0 25 August 1994 Greg Lehey, LEMIS rcc, gcc-2.6.0 SCO

0. configure i386-unknown-sco --prefix=/opt. It sets local_prefix to /usr/local anyway, and wont listen to --local_prefix. For some reason, config decides that it should be cross-compiling. 1. function.c fails to compile with the message function.c: 59: no space. Compile this function with ISC gcc-2.5.8. 2. libgcc.a was not built because config decided to cross-compile. Re-run config with configure i386-*-sco --prefix=/opt, and do an explicit make libgcc.a. 3. crtbegin.o and crtend.o were not built. Fix configure: --- configure Tue Jul 12 01:25:53 1994 +++ configure Sat Aug 27 13:09:27 1994 @@ -742,6 +742,7 @@ else tm_file=i386/sco.h tmake_file=i386/t-sco + extra_parts="crtbegin.o crtend.o"

5 February 2005 02:09

Chapter 5: Building the package

fi truncate_target=yes ;;

61

Keeping notes about problems you have with older versions helps a lot: this example represents the results of a considerable time spent debugging the make procedure. If you didnt have the log, youd risk tripping over this problem every time.

Save make output

Typically, to build a package, after you have congured it, you simply type
$ make

Then the reworks start. You can sit and watch, but it gets rather boring to watch a package compile for hours on end, so you usually leave it alone once you have a reasonable expectation that it will not die as soon as you turn your back. The problem is, of course, that you may come back and nd a lot of gobbldegook on the screen, such as:
make[5]: execve: ../../config/makedepend/makedepend: No such file or directory make[5]: *** [depend] Error 127 make[5]: Leaving directory /cdcopy/SOURCE/X11/X11R6/xc/programs/xsetroot depending in programs/xstdcmap... make[5]: Entering directory /cdcopy/SOURCE/X11/X11R6/xc/programs/xstdcmap checking ../../config/makedepend/makedepend over in ../../config/makedepend first... make[6]: Entering directory /cdcopy/SOURCE/X11/X11R6/xc/config/makedepend gcc -DNO_ASM -fstrength-reduce -fpcc-struct-return -fwritable-strings -O \ -I../../config/imake -I../.. OSDefines -DSYSV -DSYSV386 -c include.c gcc: OSDefines: No such file or directory In file included from include.c:30: def.h:133: conflicting types for getline /opt/include/stdio.h:505: previous declaration of getline Broken pipe

This is from a real life attempt to compile X11R6, normally a fairly docile port. The target makedepend failed to compile, but why? The reason has long since scrolled off the screen.* You can have your cake and eat it too if you use tee to save your output:
$ make 2>&1 | tee -a Make.log

This performs the following actions: It copies error output (le descriptor 2) to standard output (le descriptor 1) with the expression 2>&1. It pipes the combined standard output to the program tee, which echos it to its standard output and also copies it to the le Make.log.

* Well, there is a clue, but its very difcult to see unless you have been hacking X11 congurations longer than is good for your health. OSDefines is a symbol used in X11 conguration. It should have been replaced by a series of compiler ags used to dene the operating system to the package. In this case, the X11 conguration was messed up, and nothing dened OSDefines, so it found its way to the surface.

5 February 2005 02:09

62

In this case, I specied the -a option, which tells tee to append to any existing Make.log. If I dont supply this ag, it will erase any previous contents. Depending on what youre doing, you may or may not want to use this ag.

If youre not sure what your make is going to do, and especially if the Makele is complicated, consider using the -n option. This option tells make to perform a dry run: it prints out the commands that it would execute, but doesnt actually execute them. These comparatively simple conventions can save a lot of pain. I use a primitive script called Make which contains just the single line:
make 2>&1 $* | tee -a Make.log

Its a good idea to always use the same name for the log les so that you can nd them easily.

Standard targets
Building packages consists of more than just compiling and linking, and by convention many Makeles contain a number of targets with specic meanings. In the following sections well look at some of the most common ones.

make depend
make depend creates a list of dependencies for your source tree, and usually appends it to the Makele. Usually it will perform this task with makedepend, but sometimes you will see a depend target that uses gcc with the -M ag or cpp. depend should be the rst target to run, since it inuences which other commands need to be executed. Unfortunately, most Makeles dont have a depend target. Its not difcult to write one, and it pays off in the reduction of strange, unaccountable bugs after a rebuild of the package. Heres a starting point:
depend: makedepend *.[ch]

This will work most of the time, but to do it correctly you need to analyze the structure of the package: it might contain les from other languages, or some les might be created by shell scripts or special conguration programs. Hopefully, if the package is this complicated, it will also have a depend target. Even if you have a depend target, it does not always work as well as you would hope. If you make some really far-reaching changes, and things dont work the way you expect, its worth starting from scratch with a make clean to be sure that the make still works.

make all
make all is the normal way to perform the build. Frequently, it is the default target (the rst target in the Makele), and you just need to enter make. This target typically rebuilds the package but does not install it.

5 February 2005 02:09

Chapter 5: Building the package

63

make install
make install installs the compiled package into the local system environment. The usage varies considerably; well look at this target in more detail in Chapter 9, Installation, page 126.

make clean
make clean normally removes everything that make all has madethe objects, executables and possibly auxiliary les. You use it after deciding to change a compiler, for example, or to save space after you have nished an installation. Be careful with make clean: there is no complete agreement about exactly what it removes, and frequently you will nd that it doesnt remove everything it should, or it is too eager and removes lots of things it shouldnt. make clean should remove everything that make all can make again the intermediate and installable les, but not the conguration information that you may have taken days to get right.

make stamp-halfway
Occasionally you see a target like make stamp-halfway. The commands perform a lot of other things, and at the end just create an empty le called stamp-halfway. This is a short cut to save lots of complicated dependency checking: the presence of this le is intended to indicate that the rst half of the build is complete, and that a restart of make can proceed directly to the second half. Good examples of this technique can be found in the Makele for the GNU C compiler, and in the X11 source tree, which uses the name DONE for the stamp le.

Problems running make

Ideally, running make should be simple:
$ make all lots of good messages from make

Things dont always go this smoothly. You may encounter a number of problems: You may not be able to nd a Makele, or the targets dont work the way you expect. make may not be able to make any sense of the Makele. The Makele may refer to non-existent les or directories. make seems to run, but it doesnt rebuild things it should, or it rebuilds things it shouldnt. You cant nd anything thats wrong, but make still produces obscure error messages.

In the following sections well look at each of these problems. Heres an overview of the

5 February 2005 02:09

64

types of error message well consider:

Table 51: Problems running make

Problem Argument list too long "$! nulled, predecessor circle" "Circular dependency dropped" "Commands commence before rst target" Comments in command lists "Graph cycles through target Incorrect continuation lines Incorrect dependencies make forgets the current directory "Missing separator - stop" Missing targets No dependency on Makele No Makele Nonsensical targets Problems with make clean Problems with subordinate makes Prompts in Makeles Subordinate makes Syntax errors from the shell Trailing blanks in variables Unable to stop make Wrong avour of make Wrong Makele

page 74 71 71 70 69 71 73 68 70 70 66 68 64 71 72 68 74 72 71 69 71 66 66

Missing Makefile or targets

Sometimes make wont even let you in the doorit prints a message like:
$ make all Dont know how to make all. Stop.

The rst thing to check here is whether there is a Makele. If you dont nd Makele or makele, check for one under a different name. If this is the case, the author should have documented where the Makele comes fromcheck the README les and other documentation that came with the package. You may nd that the package uses separate Makeles for different architectures. For example, Makele may be correct only if you are compiling in a BSD environment. If you want to compile for a System V machine, you may need to specify a different Makele:

5 February 2005 02:09

Chapter 5: Building the package

$ make -f Makefile.sysv

65

This is a pain because its so easy to make a mistake. In extreme cases the compiler will successfully create objects, but they will fail to link. Other possibilities include: The Makele is created by the conguration process, and you havent congured yet. This would be the case if you nd an Imakele (from which you create a Makele with xmkmfsee Chapter 4, Package conguration, page 57), or Makele.in (GNU conguresee page 55). The directory you are looking at doesnt need a Makele. The Makele in the parent directory, also part of the source tree, could contain rules like:
foo/foo: foo/*.c ${CC} foo/*.c -o foo/foo

In other words, the executable is made automatically when you execute make foo/foo in the parent directory. As a rule, you start building in the root directory of a package, and perform explicit builds in subdirectories only if something is obviously wrong. The author of the package doesnt believe in Makeles, and has provided a shell script instead. You often see this with programs that originated on platforms that dont have a make program. There is really nothing to build the package: the author is used to doing the compilation manually. In this case, your best bet is to write a Makele from scratch. The skeleton in Example 5-1 will get you a surprisingly long way. The empty targets are to remind you what you need to ll in:

Example 51:
SRCS = OBJS = ${SRCS:.c=.o} CC=gcc CFLAGS=-g -O3 LDFLAGS=-g BINDIR=/opt/bin LIBDIR=/opt/lib MANDIR=/opt/man MAN1DIR=man1 INFODIR=/opt/info PROGRAM= name of all: list of C source files corresponding object files file name of compiler flags for compiler flags for linker

finished program

$(PROGRAM) ${CC} ${LDFLAGS} -o ${PROGRAM} ${OBJS}

man: doc: install: all

5 February 2005 02:09

66 Example 51: (continued)

depend: makedepend ${SRCS} clean: rm -f \#* * core $(PROGRAM) *.o

Missing targets
Another obvious reason for the error message might be that the target all doesnt exist: some Makeles have a different target name for each kind of system to which the Makele has been adapted. The README le should tell you if this is the case. One of the more unusual examples is gnuplot. You need to enter
$ make All $ make x11 TARGET=Install

The better ones at least warn yousee Chapter 4, Package conguration, page 53, for an example. I personally dont like these solutions: its so much easier to add the following line at the top of the Makele:
BUILD-TARGET = build-bsd

The rst target would then be:

all: ${BUILD-TARGET}

If you then want to build the package for another architecture, you need only change the single line dening BUILD-TARGET.

make doesnt understand the Makefile

Sometimes make produces messages that make no sense at all: the compiler tries to compile the same le multiple times, each time giving it a different object name, or it claims not to be able to nd les that exist. One possible explanation is that various avours of make have somewhat different understandings of default rules. In particular, as we will see in Chapter 19, Make, there are a number of incompatibilities between BSD make and GNU make. Alternatively, make may not even be tryining to interpret the Makele. Somebody could have hidden a le called makele in the source tree. Most people today use the name Makele for makes description le, probably because its easier to see in an ls listing, but make always looks for a le called makele (with lower case m) rst. If you are using GNU make, it rst looks for a le called GNUmakele before checking for makele and Makele.

5 February 2005 02:09

Chapter 5: Building the package

67

make refers to non-existent files

Building a package refers to a large number of les, and one of the most frequent sources of confusion is a le that cant be found. There are various avours of this, and occasionally the opposite happens, and you have trouble with a le that make nds, but you cant nd. To analyse this kind of problem, its helpful to know how make is referring to a le. Here are some possibilities: make may be looking for a dependent le, but it cant nd it, and it cant nd a rule to build it. In this case you get a message like:
$ make make: *** No rule to make target config.h. Stop.

make may not be able to locate a program specied in a command. You get a message like:
$ make foo.o /bin/cc -c foo.o -o foo.c make: execve: /bin/cc: No such file or directory make: *** [foo.o] Error 127

The compilers and other programs started by make also access les specied in the source. If they dont nd them, youll see a message like
$ make foo.o gcc -c foo.c -o foo.o foo.c:1: bar.h: No such file or directory make: *** [foo.o] Error 1

No matter where the le is missing, the most frequent reasons why it is not found are: The package has been congured incorrectly. This is particularly likely if you nd that the package is missing a le like cong.h. The search paths are incorrect. This could be because you congured incorrectly, but it also could be that the conguration programs dont understand your environment. For example, its quite common to nd Makeles with contents like:
AR AS CC LD = = = = /bin/ar /bin/as /bin/cc /bin/cc

Some older versions of make need this, since they dont look at the PATH environment variable. Most modern versions of make do look at PATH, so the easiest way to x such a Makele is to remove the directory component of the denitions.

5 February 2005 02:09

68

Problems with subordinate makes

Occasionally while building, the compiler complains about a le that doesnt seem to be there. This can be because the make is running in a subdirectory: large projects are frequently split up into multiple subdirectories, and all the top level Makele does is to run a number of subordinate makes. If it is friendly, it also echos some indication of where it is at the moment, and if it dies you can nd the le. Newer versions of GNU make print messages on entering and leaving a directory, for example:
make[1]: Entering directory /cdcopy/SOURCE/Core/glibc-1.08.8/assert make[1]: Nothing to be done for subdir_lib. make[1]: Leaving directory /cdcopy/SOURCE/Core/glibc-1.08.8/assert

If neither of these methods work, you have the option of searching for the le:
$ find . -name foo.c -print

or modifying the Makele to tell you whats going on.

make doesnt rebuild correctly

One of the most insidious problems rebuilding programs occurs when make doesnt rebuild programs correctly: theres no easy way to know that a module has been omitted, and the results can be far-reaching and time-consuming. Lets look at some possible causes of this kind of problem.

Incorrect dependencies
One weakness of make is that you have to tell it the interdependencies between the source les. Unfortunately, the dependency specications are very frequently incorrect. Even if they were correct in the source tree as delivered, changing conguration ags frequently causes other header les to be included, and as a result the dependencies change. Make it a matter of course to run a make depend after reconguring, if this target is suppliedsee page 62 for details on how to make one.

No dependency on Makefile
What happens if you change the Makele? If you decide to change a rule, for example, this could require recompilation of a program. To put it in make terms: all generated les depend on the Makele. The Makele itself is not typically included in the dependency list. It really should be, but that would mean rebuilding everything every time you change the Makele, and in most cases its not needed. On the other hand, if you do change your Makele in the course of a port, its a good idea to save your les, do a make clean and start all over again. If everything is OK, it will build correctly without intervention.

5 February 2005 02:09

Chapter 5: Building the package

69

Other errors from make

The categories we have seen above account for a large proportion of the error messages you will see from make, but there are many others as well. In this section, well look at other frequent problems.

Trailing blanks in variables

You dene a make variable with the syntax:
NAME = Definition # optional comment

The exact Denition starts at the rst non-space character after the = and continues to the end of the line or the start of the comment, if there is one. You can occasionally run into problems with things like:
MAKE = /opt/bin/make # in case something else is in the path

When starting subsidiary makes, make uses the value of the variable MAKE as the name of the program to start. In this case it is /opt/bin/make it has trailing blanks, and the exec call fails. If youre lucky, you get:
$ make make: dont know how to make make . stop.

This message does give you a clue: there shouldnt be any white space between the name of the target and the following period. On the other hand, GNU make is friendly and tidies up trailing blanks, so it says:
$ make /opt/bin/make subdir note the space before the target name "subdir" make: execve: /opt/bin/make: No such file or directory make: *** [suball] Error 127

The only clue you have here is the length of the space on the rst line. Its relatively easy to avoid this sort of problem: avoid comments at the end of denition lines.

Comments in command lists

Some versions of make, notably XENIX, cant handle rules of the form
doc.dvi: doc.tex tex doc.tex # do it again to get the references right tex doc.tex # same thing again

The rst comment causes make to think that the rule is completed, and it stops. When you x this problem by removing the comment, you run into a second one: it doesnt understand the second comment either. This time it produces an error message. Again, you need to remove the comment.

5 February 2005 02:09

70

make forgets the current directory

Occasionally, it looks as if make has forgotten what you tell it. Consider the following rule:
docs: cd doc ${ROFF} ${RFLAGS} doc.ms > doc.ps

When you run it, you get:

$ make docs cd doc groff -ms doc.ms >doc.ps gtroff: fatal error: cant open doc.ms: No such file or directory make: *** [docs] Error 1

So you look for doc.ms in doc, and its there. Whats going on? Each command is run by a new shell. The rst one executes the cd doc and then exits. The second one tries to execute the groff command. Since the cd command doesnt affect the parent environment, it has no further effect, and youre still in the original directory. To do this correctly, you need to write the rule as:
docs: cd doc; \ ${ROFF} ${RFLAGS} doc.ms > doc.ps

This causes make to consider both lines as a single line, which is then passed to a single shell. The semicolon after the cd is necessary, since the shell sees the command as a single line.

Missing separator - stop

This strange message is usually made more complicated because it refers to a line that looks perfectly normal. In all probability it is trying to tell you that you have put leading spaces instead of a tab on a command line. BSD make expects tabs, too, but it recovers from the problem, and the message it prints if they are missing is much more intelligible:
"Makefile", line 21: warning: Shell command needs a leading tab

Commands commence before first target

This message, from System V make, is trying to tell you that you have used a tab character instead of spaces at the beginning of the denition of a variable. GNU make does not have a problem with thisit doesnt even mention the fact so you might see this in a Makele written for GNU make when you try to run it with System V make. BSD make cannot handle tabs at the beginning of denitions either, and produces the message:
"Makefile", line 3: Unassociated shell command "CC=gcc" Fatal errors encountered -- cannot continue

5 February 2005 02:09

Chapter 5: Building the package

71

Syntax errors from the shell

Many Makeles contain relatively complicated shell script fragments. As we have seen, these are constrained to be on one line, and most shells have rather strange relationship between new line characters and semicolons. Heres a typical example:
if test -d $(texpooldir); then exit 0; else mkdir -p $(texpooldir); fi

This example is all on one line, but you can break it anywhere if you end each partial line with a backslash (\). The important thing here is the placement of the semicolons: a rule of thumb is to put a semicolon where you would otherwise put a newline, but not after then or else. For more details, check your shell documentation.

Circular dependency dropped

This message comes from GNU make. In System V make, it is even more obscure:
$! nulled, predecessor circle

BSD make isnt much more help:

Graph cycles through docs

In each case, the message is trying to tell you that your dependencies are looping. This particular example was caused by the dependencies:
docs: man-pages

man-pages: docs

In order to resolve the dependency docs, make rst needs to resolve man-pages. But in order to resolve man-pages, it rst needs to resolve docsa real Catch 22 situation. Real-life loops are, of course, usually more complex.

Nonsensical targets
Sometimes the rst target in the Makele does nothing useful: you need to explicitly enter make all in order to make the package. There is no good reason for this, and every reason to x itsend the mods back to the original author if possible (and be polite).

Unable to stop make

Some Makeles start a number of second and third level Makeles with the -k option, which tells make to continue if the subsidiary Makele dies. This is quite convenient if you want to leave it running overnight and collect all the information about numerous failures the next morning. It also makes it almost impossible to stop the make if you want to: hitting the QUIT key (CTRL-C or DEL on most systems) kills the currently running make, but the top-level make just starts the next subsidiary make. The only thing to do here is to identify the top-level make and stop it rst, not an easy thing to do if you have only a single screen.

5 February 2005 02:09

72

Problems with make clean

make clean is supposed to put you back to square one with a build. It should remove all the les you created since you rst typed make. Frequently, it doesnt achieve this result very accurately: It goes back further than that, and removes les that the Makele doesnt know how to make.* Other Makeles remove conguration information when you do a make clean. This isnt quite as catastrophic, but you still will not appreciate it if this happens to you after you have spent 20 minutes answering conguration questions and xing incorrect assumptions on the part of the conguration script. Either way: before running a make clean for the rst time, make sure that you have a backup. make clean can also start off by doing just the opposite: in early versions of the GNU C library, for example, it rst compiled some things in order to determine what to clean up. This may work most of the time, but is still a Bad Idea: make clean is frequently used to clean up after some catastrophic mess, or when restarting the port on a different platform, and it should not be able to rely on being able to compile anything. Yet another problem with make clean is that some Makeles have varying degrees of cleanliness, from clean via realclean all the way to squeakyclean. There may be a need for this, but its confusing for casual users.

Subordinate makes
Some subordinate makes use a different target name for the subsidiary makes: you might write make all, but make might start the subsidiary makes with make subdirs. Although this cannot always be avoided, it makes it difcult to debug the Makele. When modifying Makeles, you may frequently come across a situation where you need to modify the behaviour of only one subsidiary make. For example, in many versions of System V, the man pages need to be formatted before installation. Its easy to tell if this applies to your system: if you install BSD-style unformatted man pages, the man program will just display a lot of hard-to-read nroff source. Frequently, xing the Makele is more work than you expect. A typical Makele may contain a target install that looks like:
install: for dir in ${SUBDIRS}; do \ echo making $@ in $$dir; \ cd $$dir; ${MAKE} ${MDEFINES} $@; \ cd ..; \ done

make $@ expands to make install. One of these subdirectories is the subdirectory doc,
* If this does happen to you, dont despair just yet. Check rst whether this is just simple-mindedness on the part of the Makelemaybe there is a relatively simple way to recreate the les. If not, and you forgot to make a backup of your source tree before you started, then you can despair.

5 February 2005 02:09

Chapter 5: Building the package

73

which contains the documentation and requires special treatment for the catman pages: they need to be formatted before installation, whereas the man pages are not formatted until the rst time they are referencedsee Chapter 7, Documentation, page 99 for further information. The simplest solution is a different target that singles out the doc and makes a different target, say install-catman. This is untidy and requires some modications to the variable SUBDIRS to exclude doc. A simpler way is to create a new target, install-catman, and modify all Makeles to recognize it:
install-catman install-manman: for dir in ${SUBDIRS}; do \ echo making $@ in $$dir; \ cd $$dir; ${MAKE} ${MDEFINES} $@; \ cd ..; \ done

In the Makeles in the subdirectories, you might then nd targets like

install-catman: ${MANPAGES} for i in $<; do ${NROFF} -man $$i > ${CATMAN}/$i; done install-manman: ${MANPAGES} for i in $<; do cp $$i > ${MANMAN}/$i; done

The rule in the top-level Makele is the same for both targets: you just need to know the name to invoke it with. In this example we have also renamed the original install target so that it doesnt get invoked accidentally. By removing the install target altogether, you need to make a conscious decision about what kind of man pages that your system wants. Were not done yet: we now have exactly the situation we were complaining about on page 66: it is still a nuisance to have to remember make install-catman or make install-manman. We can get round this problem, too, with
INSTALL_TYPE=install-catman install: ${INSTALL_TYPE}

After this, you can just enter make install, and the target install performs the type of installation specied in the variable INSTALL_TYPE. This variable needs to be modied from time to time, but it makes it easier to avoid mistakes while porting.

Incorrect continuation lines

Makeles frequently contain numerous continuation lines ending with \. This works only if it is the very last character on the line. A blank or a tab following the backslash is invisible to you, but it really confuses make. Alternatively, you might continue something you dont want to. Consider the following Makele fragment, taken from an early version of the Makele for this book:
PART1 = part1.ms config.ms imake.ms make.ms tools.ms compiler.ms obj.ms \ documentation.ms testing.ms install.ms epilogue.ms

5 February 2005 02:09

74

At some point I decided to change the sequence of chapters, and removed the le tools.ms. I was not completely sure I wanted to do this, so rather than just changing the Makele, I commented out the rst line and repeated it in the new form:
# PART1 = part1.ms config.ms imake.ms make.ms tools.ms compiler.ms obj.ms \ PART1 = part1.ms config.ms imake.ms make.ms compiler.ms obj.ms \ documentation.ms testing.ms install.ms epilogue.ms

This works just neat rst. In fact, it turns out that make treats all three lines as a comment, since the comment nished with a \ character. As a result, the variable PART1 remained undened. If you comment out a line that ends in \, you should also remove the \.

Prompts in Makefiles
If you do the Right Thing and copy your make output to a log le, you may nd that make just hangs. The following kind of Makele can cause this problem:
all: checkclean prog

checkclean: @echo -n "Make clean first? " @read reply; if [ "$$reply" = y ]; then make clean; fi

If you run make interactively, you will see:

$ make Make clean first?

If you copy the output to a le, of course, you dont see the prompt, and it looks as if make is hanging. This doesnt mean its a bad idea to save your make output: its generally a bad idea to put prompts into Makeles. There are some exceptions, of course. The Linux conguration program is a Makele, and to interactively congure the system you enter make cong.

Arg list too long

Sometimes make fails with this message, especially if you are running a System V system. Many versions of System V limit the argument list to 5120 byteswell look at this in more detail in Chapter 12, Kernel dependencies, page 169. Modern versions of System V allow you to rebuild the kernel with a larger parameter list: modify the tuneable parameter ARG_MAX to a value in the order of 20000. If you cant do this, there are a couple of workarounds: The total storage requirement is the sum of the length of the argument strings and the environment strings. Its very possible that you have environment variables that arent needed in this particular situation (in fact, if youre like me, you probably have environment variables that you will never need again). If you remove some of these from your shell startup le, you may get down below the limit. You might be able to simplify expressions. For example, if your Makele contains a line like

5 February 2005 02:09

Chapter 5: Building the package

clean: rm -rf *.o *.a *.depend * core ${INTERMEDIATES}

75

you can split it into

clean: rm -rf *.o rm -rf *.a *.depend * core ${INTERMEDIATES}

In most large trees, the *.o lenames constitute the majority of the arguments, so you dont need more than two lines. Even after the previous example, you might nd that the length of the *.o parameters is too long. In this case, you could try naming the objects explicitly:
clean: rm rm rm rm -rf -rf -rf -rf [a-f]*.o [g-p]*.o [r-z]*.o *.a *.depend * core ${INTERMEDIATES}

Alternatively, you could specify the names explicitly in the Makele:

OBJ1S = absalom.o arthur.o ... fernand.o OBJ2S = gerard.o guillaume.o ... pierre.o OBJ3S = rene.o roland.o ... zygyszmund.o OBJS = ${OBJ1S} ${OBJ2S} ${OBJ3S} clean: rm -rf ${OBJ1S} rm -rf ${OBJ2S} rm -rf ${OBJ3S}

Yet another method involves the use of the xargs program. This has the advantage of not breaking after new les have been added to the lists:
clean: find . -name "*.o" -print | xargs rm -f

This chops up the parameter list into chunks that wont overow the system limits.

Creating executable files

The xargs method is not much help if you want to build an executable le. If the command that fails looks like
${PROG}: ${CC} ${ALLOBJS} -o ${PROG}

there are some other possibilities. You might be able to shorten the pathnames. If you are building in a directory /next-release/SOURCE/sysv/SCO/gcc-2.6.0, and every le name in ALLOBJS is absolute, its much easier to exceed the limit than if the directory name was, say, /S. You could use a symbolic link to solve this problem, but most systems that dont support ARG_MAX also dont have symbolic links.*

5 February 2005 02:09

76

If this doesnt work, you could place the les in a library, possibly using xargs:
${PROG}: rm libkludge.a echo ${ALLOBJS} | xargs ar cruv libkludge.a ${CC} libkludge.a -o ${PROG}

This looks strange, since theres no object le, but it works: by the time it nds the name libkludge.a, the linker has already loaded the object le crt0.o (see Chapter 21, Object les and friends, page 368), and is looking for a symbol main. It doesnt care whether it nds it in an object le or a library le.

Modifying Makefiles
Frequently enough, you nd that the Makele is inadequate. Targets are missing, or some error occurs that is almost untraceable: you need to x the Makele. Before you do this, you should check whether you are changing the correct Makele. Some packages build a new Makele every time you run make. In particular, you frequently see Makeles that start with text like
# Makefile generated by imake - do not edit!

You can follow this advice or not: it depends on you and what you are doing: If you are just trying to gure out what the Makele is trying (and presumably failing) to do, its nice to know that you can subsequently delete your modied Makele and have it automatically remade. Once you have found out why the Makele is doing what it is, you need to x the source of the Makele. This is not usually too difcult: the input les to the Makele generation phase typically dont look too different from the nished Makele. For example, Makele.in in the GNU packages is a skeleton that is processed by m4, and except for the m4 parameters Makele.in looks very similar to the nished Makele. Finding the way back to the Imakele from the Makele requires a little more understanding of the imake process, but with a little practice its not that difcult.

* If you are on a network with other machines with more modern le systems, you could work around this problem by placing the les on the other system and accessing them via NFS.

5 February 2005 02:09

6
Running the compiler
In the previous chapter, we looked at building from the viewpoint of make. The other central program in the build process is the compiler, which in UNIX is almost always a C compiler. Like make, the compiler can discover a surprising number of problems in what ostensibly debugged source code. In this chapter, well look at these problems and how to solve them. In well look at how the compiler works and how the various avours of C differ. Although we restrict our attention to the C compiler, much of what we discuss relates to other compilers as well, particularly of course to C++. This chapter expects a certain understanding of the C language, of course, but dont be put of if youre still a beginner: this is more about living with C than writing it. Information from the compiler can come in a number of forms: The compiler may issue warnings, which are informational messages intended to draw attention to possible program errors. Their reliability and their value varies signicantly: some are a sure-re indication that something is wrong, while others should be taken with a pinch of salt. The compiler may issue error messages, indicating its conviction that it cannot produce a valid output module. This also usually means that the compiler will not create any output les, though you cant always rely on this. The compiler may fail completely, either because of an internal bug or because it realizes that it no longer understands the input sufciently to continue.

Compiler warnings
Its easy to make mistakes when writing programs, but it used to be even easier: nowadays, even the worst compilers attempt to catch dubious constructs and warn you about them. In this section, well look at what they can and cant do. Before compilers worried about coding quality, the program lint performed this task. lint is still around, but hardly anybody uses it any more, since it doesnt always match the compiler being used. This is a pity, because lint can catch a number of dubious situations that evade most compilers.

77

5 February 2005 02:09

78

Modern compilers can recognize two kinds of potential problems: Problems related to dubious program text, like
if (a = 1) return;

The rst line of this example is almost superuous: if I allocate the value 1 to a, I dont need an if to tell me what the result will be. This is probably a typo, and the text should have been
if (a == 1) return;

Problems related to program ow. These are detected by the ow analysis pass of the optimizer. For example:
int a; b = a;

The second line uses the value of a before it has been assigned a value. The optimizer notices this omission and may print a warning. In the following sections, well examine typical warning messages, how they are detected and how reliable they are. Ill base the sections on the warning messages from the GNU C compiler, since the it has a particularly large choice of warning messages, and since it is also widely used. Other compilers will warn about the same kind of problems, but the messages may be different. Table 6-1 gives an overview of the warnings well see.
Table 61: Overview of warning messages

5 February 2005 02:09

Chapter 6: Running the compiler Table 61: Overview of warning messages (continued)

79

Kind of warning Changing non-volatile automatic variables Character subscripts to arrays Dequalifying types Functions with embedded extern denitions Implicit conversions between enums Implicit return type Incomplete switch statements Inconsistent function returns Increasing alignment requirements Invalid keyword sequences in declarations Long indices for switch Missing parentheses Nested comments Signed comparisons of unsigned values Trigraphs Uninitialized variables

page 82 80 81 84 82 79 82 79 81 83 82 83 83 80 83 80

Implicit return type

K&R C allowed programs like
main () { printf ("Hello, World!\n"); }

ANSI C has two problems with this program: The function name main does not specify a return type. It defaults to int. Since main is implicitly an int function, it should return a value. This one does not.

Both of these situations can be caught by specifying the -Wreturn-type option to gcc. This causes the following messages:
$ gcc -c hello.c -Wreturn-type hello.c:2: warning: return-type defaults to int hello.c: In function main: hello.c:4: warning: control reaches end of non-void function

Inconsistent function returns

The following function does not always return a dened value:

5 February 2005 02:09

80
foo (int x) { if (x > 3) return x - 1; }

If x is greater than 3, this function returns x - 1. Otherwise it returns with some uninitialized value, since there is no explicit return statement for this case. This problem is particularly insidious, since the return value will be the same for every invocation on a particular architecture (possibly the value of x), but this is a by-product of the way the compiler works, and may be completely different if you compile it with a different compiler or on some other architecture.

Uninitialized variables
Consider the following code:
void foo (int x) { int a; if (x > 5) a = x - 3; bar (a); ... etc

Depending on the value of x, a may or may not be initialized when you call bar. If you select the -Wuninitialized compiler option, it warns you when this situation occurs. Some compilers, including current versions of gcc place some limitations on this test.

Signed comparisons of unsigned values

Occasionally you see code of the form
int foo (unsigned x) { if (x >= 0) ... etc

Since x is unsigned, its value is always >= 0, so the if is superuous. This kind of problem is surprisingly common: system header les may differ in opinion as to whether a value is signed or unsigned. The option -W causes the compiler to issue warnings for this and a whole lot of other situations.

Character subscripts to arrays

Frequently, the subscript to an array is a character. Consider the following code:
char iso_translate [256] = /* translate table for ISO 8859-1 to LaserJet */ { codes for the first 160 characters

5 February 2005 02:09

Chapter 6: Running the compiler

0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0xa6, 0xa7, 0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, ... etc }; #define xlate(x) iso_translate [x]; char *s; for (*s = buf; *s; s++) *s = xlate (*s); /* pointer in buf */

81

The intention of xlate is to translate text to a form used by older model HP LaserJet printers. This code works only if the char *s is unsigned. By default, the C char type is a signed value, and so the characters 0x80 to 0xff represent a negative array offset, and the program attempts (maybe successfully) to access a byte outside the table iso_translate. gcc warns about this if you set the option -Wchar-subscripts.

Dequalifying types
The following code fragment can cause problems:
char *profane; void foo (const char *holy) { profane = holy;

The assignment of holy to profane loses the qualier const, and the compiler complains about the fact. On the other hand, this is valid:
profane = (char *) holy;

This doesnt make it a better idea: holy is supposed to be unchangeable, and here you are removing this qualier. If you specify the -Wcast-qual option to gcc, it complains if you use a cast to remove a type qualier such as const.

Increasing alignment requirements

Many processors require that specic data types be aligned on specic boundaries, and the results can be spectacular if they are notsee Chapter 11, Hardware dependencies, page 158, for more details. We can easily outsmart the C compiler with code like:
void foo (char *x) { int *ip = (int *) x;

In this case, there is a good chance that the int * pointer ip requires a specic alignment and is not allowed to point at any address in memory the way the char pointer x is allowed to do. If you specify the -Wcast-align option to gcc, it warns you of such assignments.

5 February 2005 02:09

82

Implicit conversions between enums

One of the advantages of enums is that they make type checking easierwell look at that in more detail in Chapter 20, Compilers, page 339. If you specify the -Wenum-clash option to gcc, and youre compiling C++, it warns about sloppy use of enums.

Incomplete switch statements

A frequent cause of error in a switch statement is that the index variable (the variable that decides which case is chosen) may assume a value for which no case has been specied. If the index variable is an int of some kind, there is not much you can do except include a default clause. If the index variable is an enum, the compiler can check that case clauses exist for all the possible values of the variable, and warns if they do not. It also warns if case clauses exist for values that are not dened for the type of the index variable. Specify the -Wswitch option for these warnings.

long indices for switch

In some dialects of pre-ANSI C, you could write things like
foo (x) long x; { switch (x) { ... etc

This is no longer allowed in ANSI C: indices for switch must evaluate to an int, even if int and long have the same length. gcc issues a warning about long indices in switch unless you specify the -traditional option.

Changing non-volatile automatic variables

Under certain circumstances, a signal handler might modify a local automatic variable if the function has called setjmpsee Chapter 13, Signals, page 200 for more details. gcc options this situation as a warning if you specify the -W option. This is a complicated problem: It can occur only during an optimizing compilation, since the keyword volatile has meaning only in these circumstances. In addition, the situation is recognized only by the optimizer. The optimizer cannot recognize when a longjmp could be performed. This depends on semantics outside the scope of the optimizer. As a result, it could issue this warning when there is, in fact, no danger.

5 February 2005 02:09

Chapter 6: Running the compiler

83

Invalid keyword sequences in declarations

Currently, it is permissible to write declarations like
int static bad_usage;

Here the storage class specier static comes after the type specier int. The ANSI Standard still permits this, but declares the usage to be obsolescent. gcc issues a warning when it encounters this and the option -W has been set.

Trigraphs
Trigraphs (see Chapter 20, Compilers, page 342) are no error, at least according to the ANSI Standard. The Free Software Foundation makes no bones about their opinion of them, and so gcc supplies the option -Wtrigraphs, which prints a warning if any trigraphs occur in the source code. Since this works only if the option -trigraphs is used to enable them, it is not clear that this is of any real use.

Nested comments
Occasionally you see code like
void foo (int x) { int y; y = bar (); if (y == 4) ... etc

/* state information /* initialize y */

The code looks reasonable, and it is syntactically correct C, but in fact the comment after the declaration of y is not terminated, so it includes the whole of the next line, which is almost certainly not the intention. gcc recognizes this if it nds the sequence /* in a comment, and warns of this situation if you specify the -Wcomment option.

Missing parentheses
What value does the following code return?
int a = 11 << 4 & 7 << 2 > 4;

The result is 0, but the real question is: in what order does the compiler evaluate the expression? You can nd the real answer on page 53 of K&R, but you dont want to do that all the time. We can re-write the code as
int a = (11 << 4) & ((7 << 2) > 4);

This makes it a lot clearer what is intended. gcc warns about what it considers to be missing parentheses if you select the -Wparentheses option. By its nature, this option is subjective, and you may nd that it complains about things that look ne to you.

5 February 2005 02:09

84

Functions with embedded extern definitions

K&R C allowed you to write things like
int datafile; foo (x) { extern open (); datafile = open ("foo", 0777); }

The extern declaration was then valid until the end of the source le. In ANSI C, the scope of open would be the scope of foo: outside of foo, it would no longer be known. gcc issues a warning about extern statements inside a function denition unless you supply the -traditional option. If you are using -traditional and want these messages, you can supply the -Wnested-externs option as well.

Compiler errors
Of course, apart from warnings, you frequently see error messages from the compilerthey are the most common reason for a build to fail. In this section, well look at some of the more common ones.

Undefined symbols
This is one of the most frequent compiler error messages you see during porting. At rst sight, it seems strange that the compiler should nd undened symbols in a program that has already been installed on another platform: if there are such primitive errors in it, how could it have worked? In almost every case, you will nd one of the following problems: The denition you need may have been #ifdefed out. For example, in a manually congured package, if you forget to specify a processor architecture, the package may try to compile with no processor denitions, which is sure to give rise to this kind of problem. The symbol may have been dened in a header le on the system where it was developed. This header le is different on your system, and the symbol you need is never dened. You may be looking at the wrong header les. Some versions of gcc install xed copies of the system header les in their own private directory. For example, under BSD/386 version 1.1, gcc version 2.6.3 creates a version of unistd.h and hides it in a private directory. This le omits a number of denitions supplied in the BSDI version of unistd.h. You can conrm which header les have been included by running gcc with the -H option. In addition, on page 86 we look at a way to check exactly what the preprocessor did.

The second problem is surprisingly common, even on supposedly identical systems. For

5 February 2005 02:09

Chapter 6: Running the compiler

85

example, in most versions of UNIX System V.4.2, the system header le link.h denes information and structures used by debuggers. In UnixWare 1.0, it denes information used by some Novell-specic communications protocols. If you try to compile gdb under UnixWare 1.0, you will have problems as a result: the system simply does not contain the denitions you need. Something similar happens on newer System V systems with POSIX.1 compatibility. A program that seems formally correct may fail to compile with an undened symbol O_NDELAY. O_NDELAY is a ag to open, which species that the call to open should not wait for completion of the request. This can be very useful, for example, when the open is on a serial line and will not complete until an incoming call occurs. The ag is supported by almost all modern UNIX ports, but it is not dened in POSIX.1. The result is that the denition is carefully removed if you compile dening -D_POSIX_SOURCE. You might think that this isnt a problem, and that you can replace O_NDELAY with the POSIX.1 ag O_NONBLOCK. Unfortunately, the semantics of O_NONBLOCK vary from those of O_NDELAY: if no data is available, O_NONBLOCK returns -1, and O_NDELAY returns 0. You can make the change, of course, but this requires more modications to the program, and you have a strraighforward alternative: #undef _POSIX_SOURCE. If you do this, you may nd that suddenly other macros are undened, for example O_NOCTTY. System V.4 only denes this variable if _POSIX_SOURCE is set. Theres no simple solution to this problem. It is caused by messy programming style: the programmer has mixed symbols dened only by POSIX.1 with those that are not dened in POSIX.1. The program may run on your current system, but may stop doing so at the next release.

Conflicts between preprocessor and compiler variables

Occasionally youll see things that seem to make absolutely no sense at all. For example, porting gcc, I once ran into this problem:
gcc -c -DIN_GCC -g -O3 -I. -I. -I./config \ -DGCC_INCLUDE_DIR=\"/opt/lib/gcc-lib/i386--sysv/2.6.0/include\" \ -DGPLUSPLUS_INCLUDE_DIR=\"/opt/lib/g++-include\" \ -DCROSS_INCLUDE_DIR=\"/opt/lib/gcc-lib/i386--sysv/2.6.0/sys-include\" \ -DTOOL_INCLUDE_DIR=\"/opt/i386--sysv/include\" \ -DLOCAL_INCLUDE_DIR=\"/usr/local/include\" \ -DSTD_PROTO_DIR=\"/opt/lib/gcc-lib/i386--sysv/2.6.0\" \ ./protoize.c ./protoize.c:156: macro puts used without args

Looking at this part of protoize.c, I found lots of external denitions:

extern extern extern extern extern extern extern int int int int int int int fflush (); atoi (); puts (); fputs (); fputc (); link (); unlink ();

5 February 2005 02:09

86

Line 156 is, not surprisingly, the denition of puts. But this is a denition, not a call, and certainly not a macro. And why didnt it complain about all the other denitions? There were many more than shown here. In cases like this, its good to understand the way the compiler works well look at this in more detail in Chapter 20, Compilers, on page 348. At the moment, we just need to recall that programs are compiled in two stages: rst, the preprocessor expands all preprocessor denitions and macros, and then the compiler itself compiles the resultant output, which can look quite different. If you encounter this kind of problem, theres a good chance that the compiler is not seeing what you expect it to see. You can frequently solve this kind of riddle by examining the view of the source that the compiler sees, the output of the preprocessor. In this section, well look at the technique I used to solve this particular problem. All compilers will allow you to run the preprocessor separately from the compiler, usually by specifying the -E option see your compiler documentation for more details. In this case, I was running the compiler in an xterm*, so I was able to cut and paste the complete 8-line compiler invocation as a command to the shell, and all I needed to type was the text in bold face:
$ gcc -c -DIN_GCC -g -O3 -I. -I. -I./config \ -DGCC_INCLUDE_DIR=\"/opt/lib/gcc-lib/i386--sysv/2.6.0/include\" \ -DGPLUSPLUS_INCLUDE_DIR=\"/opt/lib/g++-include\" \ -DCROSS_INCLUDE_DIR=\"/opt/lib/gcc-lib/i386--sysv/2.6.0/sys-include\" \ -DTOOL_INCLUDE_DIR=\"/opt/i386--sysv/include\" \ -DLOCAL_INCLUDE_DIR=\"/usr/local/include\" \ -DSTD_PROTO_DIR=\"/opt/lib/gcc-lib/i386--sysv/2.6.0\" \ ./protoize.c -E -o junk.c $

If you dont have xterm, you can do the same sort of thing by editing the make log (see Chapter 5, Building the package, page 60), which will contain the invocation as well. junk.c starts with:
# 1 "./config.h" 1

# 1 "./config/i386/xm-i386.h" 1 40 empty lines # 1 "./tm.h" 1 19 empty lines # 1 "./config/i386/gas.h" 1 22 empty lines

This le seems to consist mainly of empty lines, and the lines that arent empty dont seem to be C! In fact, the # lines are C (see the line directive in Chapter 20, Compilers, page 344), except that in this case the keyword line has been omitted. The empty lines are where comments and preprocessor directives used to be. The error message referred to line 156 of protoize.c, so I searched for lines with protoize.c on them. I found a number of them:
* xterm is a terminal emulator program that runs under X11. If you dont use X11, you shouldfor example, it makes this particular technique much easier.

5 February 2005 02:09

Chapter 6: Running the compiler

$ grep protoize.c junk.c # 1 "./protoize.c" # 39 "./protoize.c" 2 # 59 "./protoize.c" 2 # 62 "./protoize.c" 2 # 63 "./protoize.c" 2 ... etc # 78 "./protoize.c" 2 # 222 "./protoize.c"

87

Clearly, the text was between lines 78 and 222. I positioned on the line after the marker for line 78 and moved down (156 - 78) or 78 lines. There I found:
extern int fflush (); extern int atoi (); extern int ((fputs(( ), stdout) || (( stdout )->__bufp < ( stdout )->__put_limit ? (int) (unsigned char) (*( stdout )->__bufp++ = (unsigned char) ( 0 )) :__flshfp (( stdout ), (unsigned char) ( 0 ))) == (-1) ) ? (-1) : 0) ; extern int fputs (); extern int fputc (); extern int link (); extern int unlink ();

Well, at any rate this made it clear why the compiler was complaining. But where did this junk come from? It can be difcult to gure this out. With gcc you can use the -dD option to keep the preprocessor denitionsunfortunately, the compiler still removes the other preprocessor directives. I used -dD as well, and found in junk.c:
# 491 "/opt/include/stdio.h" 2 25 lines missing extern int fputs (__const char *__s, FILE *__stream) ; /* Write a string, followed by a newline, to stdout. */ extern int puts (__const char *__s) ; #define puts(s) ((fputs((s), stdout) || __putc(0, stdout) == EOF) ? EOF : 0)

This looks strange: rst it declares puts as an external function, then it denes it as a macro. Looking at the original source of stdio.h, I found:
/* Write a string, followed by a newline, to stdout. */ extern int puts __P ((__const char *__s)); #ifdef __OPTIMIZE__ #define puts(s) ((fputs((s), stdout) || __putc(0, stdout) == EOF) ? EOF : 0) #endif /* Optimizing. */

No, this doesnt make sense its a real live bug in the header le. At the very least, the declaration of puts () should have been in an #else clause. But thats not the real problem: it doesnt worry the preprocessor, and the compiler doesnt see it. The real problem is that protoize.c is trying to do the work of the header les and dene puts again. There are many programs that try to out-guess header les: this kind of denition breaks them all. There are at least two ways to x this problem, both of them simple. The real question is, what is the Right Thing? System or library header les should be allowed to dene macros

5 February 2005 02:09

88

instead of functions if they want, and an application program has no business trying to do the work of the header les, so it would make sense to x protoize.c by removing all these external denitions: apart from this problem, theyre also incompatible with ANSI C, since they dont describe the parameters. In fact, I chose to remove the denition from the header le, since that way I only had to do the work once, and in any case, its not clear that the denition really would run any faster. Preprocessor output usually looks even more illegible than this, particularly if lots of clever nested #defines have been performed. In addition, youll frequently see references to nonexistant line numbers. Here are a couple of ways to make it more legible: Use an editor to put comments around all the #line directives in the preprocessor output, and then recompile. This will make it easier to nd the line in the preprocessor output to which the compiler or debugger is referring; then you can use the comments to follow it back to the original source. Run the preprocessor output through a program like indent, which improves legibility considerably. This is especially useful if you nd yourself in the unenviable position of having to modify the generated sources. indent is not guaranteed to maintain the same number of lines, so after indenting you should recompile.

Other preprocessors
There are many other cases in which the source le you use is not the source le that the compiler gets. For example, yacc and bison take a grammar le and make a (more or less illegible) .c le out of it; other examples are database preprocessors like Informix ESQL, which takes C source with embedded SQL statements and converts it into a form that the C compiler can compile. The preprocessors output is intended to be read by a compiler, not by humans. All of these preprocessors use lines beginning with # to insert information about the original line numbers and source les into their output. Not all of them do it correctly: if the preprocessor inserts extra lines into the source, they can become ambiguous, and you can run into problems when using symbolic debuggers, where you normally specify code locations by line number.

Syntax errors
Syntax errors in previously functional programs usually have the same causes as undened symbols, but they show their faces in a different way. A favourite one results from omitting /usr/include/sys/types.h. For example, consider bar.c:
#include <stdio.h> #ifdef USG #include <sys/types.h> #endif ushort num; int main (int argc, char *argv []) {

5 February 2005 02:09

Chapter 6: Running the compiler

num = atoi (argv [1]); printf ("First argument: %d\n", num); }

89

If you compile this under BSD/OS, you get:

$ gcc -o bar bar.c bar.c:6: parse error before num bar.c:6: warning: data definition has no type or storage class

Theres an error because ushort hasnt been dened. The compiler expected a type specier, so it reported a syntax error, not an undened symbol. To x it, you need to dene the type specied see Appendix A, Comparative reference to UNIX data types for a list of the more common type speciers.

Virtual memory exhausted

You occasionally see this message, particularly when youre using gcc, which has a particular hunger for memory. This may be due to unrealistically low virtual memory limits for your system by default, some systems limit total virtual memory per process to 6 MB, but gcc frequently requires 16 or 20 MB of memory space, and on occasion it can use up to 32 MB for a single compilation. If your system has less than this available, increase the limit accordingly. Dont forget to ensure that you have enough swap space! Modern systems can require over 100 MB of swap space. Sometimes this doesnt help. gcc seems to have particular difculties with large data denitions; bit map denitions in X11 programs are the sort of things that cause problems. xphoon, which displays a picture of the current phase of the moon on the root window, is a good example of a gcc-breaker.

Compiler limits exceeded

Some compilers have difculties with complicated expressions. This can cause cc1, the compiler itself, to fail with messages like expression too complicated or out of tree space. Fixing such problems can be tricky. Straightforward code shouldnt give the compiler indigestion, but some nested #denes can cause remarkable increases in the complexity of expressions: in some cases, a single line can expand to over 16K of text. One way to get around the problem is to preprocess the code and then break the preprocessed code into simpler expressions. The indent program is invaluable here: preprocessor output is not intended to be human-readable, and most of the time it isnt.

Running compiler passes individually

Typical compilers run four distinct passes to compile and link a programsee Chapter 20, Compilers, page 348, for more details. Sometimes running the passes separately can be useful for debugging a compilation:

5 February 2005 02:09

90

If you nd yourself with header les that confuse your preprocessor, you can run a different preprocessor, collect the output and feed it to your compiler. Since the output of the preprocessor is not machine-dependent, you could even do this on a different machine with different architecture, as long as you ensure that you use the correct system header les. By convention, the preprocessor output for foo.c would be called foo.isee Chapter 20, Compilers, page 348 for a list of intermediate le sufxes though it usually does no harm if you call it foo.c and pass it through the preprocessor again, since there should no longer be anything for the second preprocessor to do. If you want to report a compiler bug, its frequently a good idea to supply the preprocessor output: the bug might be dependent on some header le conict that doesnt exist on the system where the compiler development takes place. If you suspect the compiler of generating incorrect code, you can stop compilation after the compiler pass and collect the generated assembler output.

Incorrect code from compiler

Compilers sometimes generate incorrect code. Incorrect code is frequently difcult to debug because the source code looks (and might be) perfect. For example, a compiler might generate an instruction with an incorrect operand address, or it might assign two variables to a single location. About the only thing you can do here is to analyze the assembler output. One kind of compiler bug is immediately apparent: if the code is so bad that the assembler cant assemble it, you get messages from the assembler. Unfortunately, the message doesnt usually tell you that it comes from the assembler, but the line numbers change between the compiler and the assembler. If the line number seems completely improbable, either because it is larger than the number of lines in your source le, or because it seems to have nothing to do with the context of that line, there is a chance that the assembler produced the message. There are various ways to conrm which pass of the compiler produced the message. If youre using gcc, the simplest one is to use the -v option for the compiler, which announces each pass of compilation as it starts, together with the version numbers and parameters passed to the pass. This makes it relatively easy to gure out which pass is printing the error messages. Otherwise you can run the passes individually see Chapter 20, Compilers, page 348 for more details.

5 February 2005 02:09

7
Documentation
Ask any real guru a question, so the saying goes, and he will reply with a cryptic RTFM.* Cynics claim this is even the answer to the question Where can I nd the manual? All too often, programmers consider documentation a necessary (or even unnecessary) evil, and if it gets done at all, its usually the last thing that gets done. This is particularly evident when you look at the quality of documentation supplied with some free software packages (though many free packages, such as most of those from the Free Software Foundation, are very well documented). The quality and kind of the documentation in source packages varies wildly. In Chapter 2, Unpacking the goodies, page 25, we looked at the documentation that should be automatically supplied with the package to describe what it is and how to install it. In this chapter, well look at documentation that is intended for use after you have installed the package. The documentation you get with a package is usually in one of the following formats: man pages, the traditional on-line documentation for UNIX, which are formatted with nroff. info les, used with the GNU projects info on-line documentation reader. Unformatted roff, T X, or texinfo hardcopy documentation. E Preformatted documentation in PostScript or .dvi format, or occasionally in other formats such as HP LaserJet.

We know where we want to get tothe formatted documentationbut we dont always know where to start, so its easier to look at documentation in reverse order: rst, well look at the end result, then at the formatters, and nally at the input les.

Preformatted documentation
Occasionally you get documentation that has been formatted so that you can print it on just about any printer, but this doesnt happen very much: in order to achieve this, the text must be free of any frills and formatted so that any typewriter can print it. Nearly any printer
* Read The Manualthe F is usually silent. 91

5 February 2005 02:09

92

nowadays is capable of better results, so preformatted les are usually supplied in a format that can print high quality printout on a laser printer. The following three are about the only ones you will come across: PostScript is a specialized programming language for printers, and the printed data are in fact embedded in the program. This makes it an extremely exible format. .dvi is the format that is output by T X. In order to print it, you need a TEX driver. E Unlike PostScript and .dvi, the Hewlett-Packard LaserJet format is not portable: you need a LaserJet-compatible printer to print it. The LaserJet format is obsolescent: even many LaserJet printers made today also support PostScript, and there are programmatic ways to print PostScript on other laser printers, so there is little motivation for using the much more restrictive LaserJet format.

PostScript
PostScript is the current format of choice. Because it is a programming language, it is much more exible than conventional data formats. For example, it is easily scalable. You can take a le intended for a phototypesetter with a resolution of 2540 bpi and print it on a laser printer, and it will come out correctly.* In addition, better quality printers perform the formatting themselves, resulting in a considerable load reduction for the computer. A large number of printers and all modern phototypesetters can process PostScript directly. If your printer doesnt handle PostScript, you can use programs like ghostscript, which interpret PostScript programs and output in a multitude of other formats, including LaserJet, so even if you have a LaserJet, it can be a better idea to use PostScript format. ghostscript is distributed by the Free Software Foundation see Appendix E, Where to get sources. ghostscript can also display PostScript les on X displays. Most PostScript les are encoded in plain ASCII without any control characters except newline (though that doesnt make them easy to read). Even when you include special characters in your text, they appear in the PostScript document as plain ASCII sequences. Its usually pretty easy to recognize PostScript, even without the le program. Heres the start of a draft version of this chapter:
%!PS-Adobe-3.0 %%Creator: groff version 1.09 %%CreationDate: Thu Aug 18 17:34:24 1994 %%DocumentNeededResources: font Times-Bold

The data itself is embedded in parentheses between the commands. Looking at a draft of this text, we see things like
(It)79.8 273.6 Q 2.613(su)-.55 G .113 (sually pretty easy to recognize a PostScript program, e)-2.613 F -.15 (ve)-.25 G 2.614(nw).15 G .114(ithout the)-2.614 F F2(\214le)2.614 E F1 (program--here)79.8 285.6 Q 2.5(st)-.55 G(he start of a draft v)-2.5 E

* You may have to wait a while before a few megabytes of font information are transferred and processed, but eventually you get your document.

5 February 2005 02:09

Chapter 7: Documentation

93

Problems with PostScript

PostScript doesnt pose too many problems, but occasionally you might see one of these: Missing fonts PostScript documents include information about the fonts they require. Many fonts are built in to printers and PostScript display software, but if the fonts are not present, the system chooses a default value which may have little in common with the font which the document requested. The default font is typically Courier, which is fixed-width, and the results look terrible. If this happens, you can nd the list of required fonts with the following:
$ grep %%.* font mumble.ps %%DocumentNeededResources: font Garamond-BookItalic %%+ font Times-Roman %%+ font Garamond-Light %%+ font Garamond-LightItalic %%+ font Courier %%+ font Garamond-Book %%+ font Courier-Bold %%IncludeResource: font Garamond-BookItalic %%IncludeResource: font Times-Roman %%IncludeResource: font Garamond-Light %%IncludeResource: font Garamond-LightItalic %%IncludeResource: font Courier %%IncludeResource: font Garamond-Book %%IncludeResource: font Courier-Bold (%%DocumentNeededResources: font Times-Bold)131.711 327.378 S F1 1.281

This extracts the font requests from the PostScript le: in this case, the document requires Times Roman, Courier and Garamond fonts. Just about every printer and software package supplies Times Roman and Courier, but Garamond (the font in which this book is written) is less common. In addition, most fonts are copyrighted, so you probably wont be able to nd them on the net. If you have a document like this in PostScript format, your choices are: Reformat it with a different font if you have the source. Get the Garamond fonts.

Edit the le and change the name of the font to a font with similar metrics (in other words, with similar size characters). The results wont be as good, but if the font you nd is similar enough, they might be acceptable. For example, you might change the text Garamond to Times Roman. Wrong font type Most PostScript fonts are in plain ASCII. You may also come across Type 2 PostScript and display PostScript, both of which include binary data. Many printers cant understand the binary format, and they may react to it in an unfriendly way. For example, my National KX-P 4455 printer just hangs if I copy display PostScript to it. See the section format conversion below for ways to solve this dilemma.

5 February 2005 02:09

94

.dvi format
One of the goals of T X was to be able to create output for just about any printer. As we will E see, old versions of troff, the main competitor, were able to produce output only for a very limited number of phototypesetters. Even if you have one of them in your ofce, its unlikely that you will want to use it for printing out a draft of a 30-page paper. The T X solution, which was later adopted by troff in ditroff (device independent troff), was to E output the formatted data in a device-independent format, .dvi, and leave it to another program, a so-called driver, to format the les in a format appropriate to the output device. Unlike PostScript, .dvi contains large numbers of control characters and characters with the sign bit set, and is not even remotely legible. Most versions of le know about .dvi format.

Format conversion
Not so long ago your choice of documentation software determined your output format. For example, if you used T X, you would get .dvi output, and you would need a T X driver to print E E it. Nowadays, its becoming easier to handle le formats. GNU troff will output in .dvi format if you wish, and programs are available to convert from .dvi to PostScript and back again. Heres a list of conversions you might like to perform see Appendix E, Where to get sources for how to get software to perform them. A number of programs convert from .dvi to PostScriptfor example, dvips. Theres no good reason to want to convert from PostScript to .dvi, so there are no programs available. .dvi is not much use in itselfit needs to be tranformed to a nal printer form, and if you have PostScript output, you can do that directly with ghostscript (see below) without going via .dvi. To display .dvi les on an X display, use SeeTeX. To convert from .dvi to a printer output format, use one of the dvi2xxx programs. To convert from PostScript to a printer format, use ghostscript. To display PostScript on an X display, you can also use ghostscript, but ghostview gives you a better interface. To convert PostScript with binary data into ASCII, use t1ascii.

roff and friends

The original UNIX formatting program was called roff (for run-off). It is now completely obsolete, but it has a number of descendents: nroff is a comparatively simple formatter designed to produce output for plain ASCII displays and printers. troff is a more sophisticated formatter designed to produce output for phototypesetters. Many versions create output only for the obsolete APS-5 phototypesetter, and you need

5 February 2005 02:09

Chapter 7: Documentation

95

postprocessing software to convert this output to something that modern typesetters or laser printers understand. Fortunately, versions of troff that produce PostScript output are now available. ditroff (device independent troff) is a newer version of troff that produces output in a device-independent intermediate form that can then be converted into the nal form by a conversion program. This moves the problem of correct output format from troff to the conversion program. Despite the terminology, this device-independent format is not the same as .dvi format. groff is the GNU project troff and nroff replacement. In troff mode it can produce output in PostScript and .dvi format.

All versions of roff share the same source le syntax, though nroff is more restricted in its functionality than troff. If you have a usable version of troff, you can use it to produce properly formatted hardcopy versions of the man pages, for example. This is also what xman (the X11 manual browser) does.

formatting with nroff or troff

troff input bears a certain resemblance to the traces left behind when a y falls into an inkwell and then walks across a desk. The rst time you run troff against a le intended for troff, the results may be less than heartening. For example, consider the following passage from the documentation of the Revision Control System RCS. When correctly formatted, the output is:
Besides the operations ci and co, RCS provides the following commands: ident rcs rcsclean rcsdiff rcsfreeze rcsmerge rlog extract identication markers change RCS le attributes remove unchanged working les (optional) compare revisions record a conguration (optional) merge revisions read log messages and other information in RCS les

A synopsis of these commands appears in the Appendix. 2.1 Automatic Identication RCS can stamp source and object code with special identication strings, similar to product and serial numbers. To obtain such identication, place the marker $Id$ into the text of a revision, for instance inside a comment. The check-out operation will replace this marker with a string of the form $Id: lename revisionnumber date time author state locker $

To format it, you can try

$ troff rcs.ms >rcs.ps

This assumes the use of groff or another avour of troff that creates PostScript output (thus the

5 February 2005 02:09

96

name rcs.ps for the output le). If you do this, you get an output that looks like:
Besides the operations ci and co, RCS provides the following commands: tab(%); li l. ident%extract identication markers rcs%change RCS le attributes rcsclean%remove unchanged working les (optional) rcsdiff%compare revisions rcsfreeze%record a conguration (optional) rcsmerge%merge revisions rlog%read log messages and other information in RCS les A synopsis of these commands appears in the Appendix. Automatic Identication RCS can stamp source and object code with special identication strings, similar to product and serial numbers. To obtain such identication, place the marker Id into the text of a revision, for instance inside a comment. The check-out operation will replace this marker with a string of the form Id: lename revisionnumber date time author state locker

Most of the text seems to be there, but it hasnt been formatted at all (well, it has been right justied). What happened? Almost every troff or roff input document uses some set of macros. You can dene your own macros in the source, of course, but over time a number of standard macro packages have evolved. They are stored in a directory called tmac. In the days of no confusion, this was /usr/lib/tmac, but nowadays it might equally well be /usr/share/tmac (for systems close to the System V.4 ABIsee Chapter 4, Package conguration, page 48, for more details) or /usr/local/groff/tmac for GNU roff. The name is known to troff either by environment variables or by instinct (the path name is compiled into the program). troff loads specic macros if you specify the name of the le as an argument to the -m ag. For example, to specify the man page macros /usr/lib/tmac/an, you would supply troff with the parameter -man. man makes more sense than an, so these macros are called the man macros. The names of other macro packages also usually grow an m at the beginning. Some systems change the base name of the macros from, say, /usr/lib/tmac/an to /usr/lib/tmac/tmac.an. Most versions of troff supply the following macro packages: The man (tmac/an) and mandoc (tmac/andoc) packages are used to format man pages. The mdoc (tmac/doc) package is used to format hardcopy documents, including some man pages. The mm (tmac/m) macros, the so-called memorandum macros, are described in the documentation as macros to format letters, reports, memoranda, papers, manuals and books. It doesnt describe what you shouldnt use them for. The ms (tmac/s) macros were the original macros supplied with the Seventh Edition. They are now claimed to be obsolescent, but you will see them again and again. This book was formatted with a modied version of the ms macros. The me (tmac/e) macros are another, more recent set of macros which originated in Berkeley.

There is no sure-re way to tell which macros a le needs. Here are a couple of possibilities: The le name sufx might give a hint. For example, our le is called rcs.ms, so there is a very good chance that it wants to be formatted with -ms.

5 February 2005 02:09

Chapter 7: Documentation

97

The program grog, which is part of groff, examines the source and guesses the kind of macro set. It is frequently wrong. The only other way is trial and error. There arent that many different macro sets, so this might be a good solution.

In this case, our le name suggests that it should be formatted with the ms macros. Lets try that:
$ troff rcs.ms >rcs.ps

Now we get:
Besides the operations ci and co, RCS provides the following commands: tab(%); li l. ident%extract identication markers rcs%change RCS le attributes rcsclean%remove unchanged working les (optional) rcsdiff%compare revisions rcsfreeze%record a conguration (optional) rcsmerge%merge revisions rlog%read log messages and other information in RCS les A synopsis of these commands appears in the Appendix. 2.1 Automatic Identication RCS can stamp source and object code with special identication strings, similar to product and serial numbers. To obtain such identication, place the marker $Id$ into the text of a revision, for instance inside a comment. The check-out operation will replace this marker with a string of the form $Id: lename revisionnumber date time author state locker $

Well, it doesnt look quite as bad, but its still not where we want to be. What happened to that list of program names? troff does not do all the work by itself. The tabular layout of the program names in this example is done by the preprocessor tbl, which handles tables. Before we let troff at the document, we need to pass it through tbl, which replaces the code
.TS tab(%); li l. ident%extract identification markers rcs%change RCS file attributes rcsclean%remove unchanged working files (optional) rcsdiff%compare revisions rcsfreeze%record a configuration (optional) rcsmerge%merge revisions rlog%read log messages and other information in RCS files .TE

with a couple of hundred lines of complicated and illegible troff instructions to build the table. To get the desired results, we need to enter:
$ tbl rcs.ms | troff -ms >rcs.ps

nroff, troff and groff use a number of preprocessors to perform special functions. They are:

5 February 2005 02:09

98

soelim replaces .so statements (which correspond to C #include statements) with the contents of the le to which the line refers. The roff programs do this too, of course, but the other preprocessors dont, so if the contents of one of the les is of interest to another preprocessor, you need to run soelim rst. refer processes references. pic draws simple pictures. tbl formats data in tabular form. eqn formats equations.

Unless you know that the document youre formatting doesnt use any of these preprocessors, or formatting takes a very long time, its easier to use them all. There are two possible ways to do this: You can pipe from one processor to the next. This is the standard way:
$ soelim rcs.ms | refer | pic | tbl | eqn | troff -ms The soelim preprocessor reads in the document, and replaces any .so commands by the contents of the le to which they refer. It then passes the output to refer, which processes any textual references and passes it to pic, which processes any pictures it may nd, and passes the result to tbl. tbl processes any tables and passes its result to eqn, which processes any equations before passing the result to troff.

Some versions of troff invoke the preprocessors themselves if passed appropriate ags. For example, with groff:
Table 71: Starting preprocessors from groff Flag -e -t -p -s -R

Processor eqn tbl pic soelim refer

Starting the preprocessors from troff not only has the advantage of involving less typingit also ensures that the preprocessors are started in the correct sequence. Problems can arise if you run eqn before tbl, for example, when there are equations within tables. See Typesetting tables with tbl by Henry McGilton and Mary McNabb for further details.

Other roff-related programs

As you can see, the troff system uses a large number of programs. Once they were relatively small, and this was the UNIX way. Now they are large, but there are still a lot of them. Apart from the programs we have already seen, you could encounter the GNU variants, which can

5 February 2005 02:09

Chapter 7: Documentation

99

optionally be installed with a name beginning in gfor example, GNU eqn may be installed as geqn if the system already has a different version of eqn. indxbib and lookbib (sometimes called lkbib) process bibliographic references, and are available in the groff package if you dont have them. groff also includes a number of other programs, such as grops, and grotty, which you dont normally need to invoke directly.

Man pages
Almost from the beginning, UNIX had an on-line manual, traditionally called man pages. You can peruse man pages with the man program, or you can print them out as hardcopy documentation. Traditionally, man pages are cryptic and formalized: they were introduced at a time when disk storage was expensive, so they are short, and they were intended as a reference for people who already understand the product. More and more, unfortunately, they are taking on the responsibility of being the sole source of documentation. They dont perform this task very well.

man history
The UNIX man facility has had a long and varying history, and knowing it helps understand some of the strangenesses. The Seventh Edition of the Unix Programmers Manual was divided into nine sections. Section 9, which contained the quick reference cards, has since atrophied. Traditionally, you refer to man pages by the name of the item to which they refer, followed by the section number in parentheses, so the man page for the C compiler would be called cc(1). BSD systems have substantially retained the Seventh Edition structure, but System V has reorganized them. There are also differences of opinion about where individual man pages belong, so Table 7-2 can only be a guide:
Table 72: UNIX manual sections Seventh Edition Section 1 2 3 4 5 6 7 8 9

Contents

System V Section 1 2 3 7, 4 4, 5 6 7 1m

Commands (programs) System Calls (direct kernel interface) Subroutines (library functions in user space) Special les File Formats and Conventions Games Macro Packages and Language Conventions Maintenance Quick Reference cards

What distinguished the UNIX manual from that of other systems was that it was designed to

5 February 2005 02:09

100

be kept online. Each of these sections, except for the quick reference cards, was stored in nroff format in a directory called /usr/man/man<section>, where <section> was the section number. Each entry was (and is) called a man page, although nowadays some can run on for 100 pages or more. The manual was stored in nroff format in order to be independent of the display hardware, and because formatting the whole manual took such a long time. For these reasons it was chosen to format pages on an individual basis when they were accessed, which made access to the manual slower and thus less attractive to use. The speed problem was solved by saving the formatted copy of the man page in a second directory hierarchy, /usr/man/cat<section>, the rst time that the page was formatted. Subsequent accesses would then nd the formatted page and display that more quickly. This basic hierarchy has survived more or less intact to the present day. People have, of course, thought of ways to confuse it: As the manual got larger, it seemed reasonable to subdivide it further. Most users werent interested in system administration functions, so some systems put them into a separate directory, such as /usr/man/cat1m, or gave them a lename sufx such as m, so that the manual page for shutdown might end up being called /usr/man/cat1/shutdown.1m or /usr/man/man1m/shutdown.1m or something similar. Various commercial implementations reorganized the sequence of the sections in the printed manual, and reorganized the directories to coincide. For example, in System V the description of the le /etc/group is in section 4, but in the Seventh Edition and BSD it is in section 5. Even without the uncertainty of which section to search for a command, it was evident that section numbers were not very informative. Some implementations, such as XENIX and some versions of System V, chose to replace the uninformative numbers with uninformative letters. For example, ls(1) becomes ls(C) in XENIX. Some man programs have lost the ability to format the man pages, so you need to format them before installation. Youll nd this problem on systems where nroff is an add-on component. There is no longer a single directory where you can expect to put man pages: some System V versions put formatted man pages for users in a directory /usr/catman/u_man, and man pages for programmers in /usr/catman/p_man. Since most programmers are users, and the distinction between the use of the man pages is not always as clear as you would like, this means that man has to search two separate directory hierarchies for the man pages. As we saw in Chapter 4, Package conguration, page 48, System V.4 puts its man pages in /usr/share/man. Many System V.4 systems require formatted man pages, and some, such as UnixWare, dont provide a man program at all. Many man programs accept compressed input, either formatted or non-formatted. For some reason, the pack program still survives here, but other versions of man also understand man pages compressed with compress or gzip. We looked at all of these programs

5 February 2005 02:09

Chapter 7: Documentation

101

in Chapter 2, Unpacking the goodies, page 20. Different man programs place different interpretations on the sufx of the man page lename. They seldom document the meanings of the sufx. To keep up the tradition of incompatible man pages, BSD has changed the default macro set from man to mdoc. This means that older man page readers cant make any sense of unformatted BSD man pages.

This combination of affairs makes life difcult. For example, on my system I have a number of different man pages in different directories. The le names for the man pages for printf, which is both a command and a library function, are:
BSD printf command, formatted: /usr/share/man/cat1/printf.0 Solaris printf command, nroff: /pub/man/solaris-2.2/man1/printf.1 SVR4.2 printf command, formatted, compressed: /pub/man/svr4.2/cat1/printf.1.Z BSD printf function, formatted: /usr/share/man/cat3/printf.0 Solaris 2.2 printf function, nroff, standard: /pub/man/solaris-2.2/man3/printf.3s Solaris 2.2 printf function, nroff, BSD version: /pub/man/solaris-2.2/man3/printf.3b SunOS 4.1.3 printf function, nroff: /pub/man/sunos-4.1.3/man3/printf.3v SVR3 printf function, formatted, packed: /pub/man/catman/p_man/man3/printf.3s.z SVR4.2 printf function, formatted, compressed: /pub/man/svr4.2/cat3/printf.3s.Z SVR4.2 printf function, formatted, compressed, BSD version: /pub/man/svr4.2/cat3/printf.3b.Z XENIX printf function, nroff, packed: /pub/man/xenix-2.3.2/man.S/printf.S.z

Most packages assume that unformatted man pages will be installed in /usr/man. They usually accept that the path may be different, and some allow you to change the subdirectory and the le name sufx, but this is as far as they normally go. This lack of standardization can cause such problems that many people just give up and dont bother to install the man pages. This is a pityinstead, why not install a man program that isnt as fussy? A number of alternatives are available, including one for System V.4 from Walnut Creek and a number on various Linux distributions.

TeX
T X is Donald Knuths monument to the triumph of logic over convention. To quote Knuths E The TEX book,
Insiders pronounce the of T X as a Greek chi, not as an x, so that T X rhymes with the word E E blecchhh. Its the ch sound in Scottish words like loch or German words like ach; its a

5 February 2005 02:09

102 Spanish j and a Russian kh. When you say it correctly to your computer, the terminal may become slightly moist.

This is one of the more informative parts of The TEX book. It is, unfortunately, not a manual but a textbook, and most of the essential parts are hidden in exercises agged very difcult. If you just want to gure out how to format a T X document, Making TEX work, by Norman E Walsh, is a much better option. If troff input looks like a y having left an inkwell, T X input resembles more the attempts of a E drunken spider. Heres part of the le plain.tex which denes some of the things that any T X E macro package should be able to do:
\def\cases#1{\left\{\,\vcenter{\normalbaselines\m@th \ialign{$##\hfil$&\quad##\hfil\crcr#1\crcr}}\right.} \def\matrix#1{\null\,\vcenter{\normalbaselines\m@th \ialign{\hfil$##$\hfil&&\quad\hfil$##$\hfil\crcr \mathstrut\crcr\noalign{\kern-\baselineskip} #1\crcr\mathstrut\crcr\noalign{\kern-\baselineskip}}}\,}

More than anywhere else in porting, it is good for your state of mind to steer clear of TEX internals. The assumptions on which the syntax is based differ markedly from those of other programming languages. For example, identiers may not contain digits, and spaces are required only when the meaning would otherwise be ambiguous (to T X, not to you), so the E sequence fontsize300 is in fact the identier fontsize followed by the number 300. On the other hand, it is almost impossible to nd any good solid information in the documentation, so you could spend hours trying to solve a minor problem. I have been using T X freE quently for years, and I still nd it the most frustrating program I have ever seen.* Along with T X, there are a couple of macro packages that have become so important that they E are almost text processors in their own right:
AE L T X is a macro package that is not quite as painful as plain T X, but also not as powerE ful. It is normally built as a separate program when installing T X, using a technique of E dumping a running program to an object le that we will examine in Chapter 21, Object les and friends, page 376. AE BIBT X is an auxiliary program which, in conjuntion with L T X, creates bibliographic E references. Read all about it in Making TEX work. It usually takes three runs through the source les to create the correct auxiliary les and format the document correctly.

texinfo is a GNU package that supplies both online and hardcopy documentation. It uses T X to format the hardcopy documentation. Well look at it along with GNU info in the E next section.

* When I wrote this sentence, I wondered if I wasnt overstating the case. Mike Loukides, the author of Programming with GNU Software, reviewed the nal draft and added a single word: Amen.

5 February 2005 02:09

Chapter 7: Documentation

103

GNU Info and Texinfo

Its unlikely that youll break out in storms of enthusiasm about the documentation techniques weve looked at so far. The GNU project didnt, either, when they started, though their concerns were somewhat different: Man pages are straightforward, but the man program is relatively primitive. In particular, man does not provide a way to follow up on references in the man page. Man pages are intended to be stored on-line and thus tend to be cryptic. This makes them unsuited as hardcopy documentation. Making them longer and more detailed makes them less suited for online documentation. There is almost no link between man pages and hardcopy documentation, unless they happen to be the same thing for a particular package. Maintaining man pages and hardcopy documentation is double the work and opens you to the danger of omissions in one or the other document.

As in other areas, the GNU project started from scratch and came up with a third solution, info. This is a combined system of online and hardcopy documentation. Both forms of documentation are contained in the source le: you use makeinfo program to create info documents, which you read with the on-line browser info, and you use T X and the texinfo macro E set are used to format the documentation for printing. info is a menu-driven, tree-structured online browser. You can follow in-text references and then return to the original text. info is available both as a stand-alone program and as an emacs macro. If you have a package that supplies documentation in info format, you should use it. Even if some GNU programs, such as gcc and emacs, have both info and man pages, the info is much more detailled. Running texinfo is straightforward: run T X. The document reads in the le texinfo.tex, and E about the only problem you are likely to encounter is if it doesnt nd this le.

The World-Wide Web

The World-Wide Web (WWW) is not primarily a program documentation system, but it has a number of properties which make it suitable as a manual browser: as a result of the proliferation of the Internet, it is well known and generally available, it supplies a transparent crossreference system, and the user interface is easier to understand. Its likely that it will gain importance in the years to come. Hopefully it will do this without causing as much confusion as its predecessors.

5 February 2005 02:09

8
Testing the results
Finally make has run through to the end and has not reported errors. Your source tree now contains all the objects and executables. Youre done! After a brief moment of euphoria, you sit down at the keyboard and start the program:
$ xterm Segmentation fault - core dumped

Well, maybe youre not quite done after all. Occasionally the program does not work as advertised. What you do now depends on how much programming experience you have. If you are a complete beginner, you could be in troubleabout the only thing you can do (apart from asking somebody else) is to go back and check that you really did congure the package correctly. On the other hand, if you have even a slight understanding of programming, you should try to analyze the cause of the errorits easier than you think. Hold on, and try not to look down. There are thousands of possible reasons for the problems you encounter when you try to run a buggy executable, and lots of good books explain debugging techniques. In this chapter, we will touch only on aspects of debugging that relate to porting. First well attack a typical, if somewhat involved, real-life bug, and solve it, discussing the pros and cons on the way. Then well look at alternatives to traditional debuggers: kernel and network tracing. Before you even start your program, of course, you should check if any test programs are available. Some packages include their own tests, and separate test suites are available for others. For other packages there may be test suites that were not designed for the package, but that can be used with it. If there are any tests, you should obviously run them. You might also consider writing some tests and including them as a target test in the Makele.

What makes ported programs fail?

Ported programs dont normally fail for the same reasons as programs under development. A program under development still has bugs that prevent it from running correctly on any platform, while a ported program has already run reasonably well on some other platform. If it doesnt run on your platform, the reasons are usually:

105

5 February 2005 02:09

106

A latent bug has found more fertile feeding ground. For example, a program may read from a null pointer. This frequently doesnt get noticed if the data at address 0 doesnt cause the program to do anything unusual. On the other hand, if the new platform does not have any memory mapped at address 0, it will cause a segmentation violation or a bus error. Differences in the implementation of library functions or kernel functionality cause the program to behave differently in the new environment. For example, the function setpgrp has completely different semantics under System V and under BSD. See Chapter 12, Kernel dependencies, page 171, for more details. The conguration scripts have never been adequately tested for your platform. As a result, the program contains bugs that were not in the original versions.

A strategy for testing

When you write your own program with its own bugs, it helps to understand exactly what the program is trying to do: if you sit back and think about it, you can usually shorten the debugging process. When debugging software that you have just ported, the situation is different: you dont understand the package, and learning its internals could take months. You need to nd a way to track down the bug without getting bogged down with the specics of how the package works. You can overdo this approach, of course. It still helps to know what the program is trying to do. For example, when xterm dies, its nice to know roughly how xterm works: it opens a window on an X server and emulates a terminal in this window. If you know something about the internals of X11, this will also be of use to you. But its not time-effective to try to ght your way through the source code of xterm. In the rest of this chapter, well use this bug (yes, it was a real live bug in X11R6) to look at various techniques that you can use to localize and nally pinpoint the problem. The principle we use is the old GIGO principle garbage in, garbage out. Well subdivide the program into pieces which we can conveniently observe, and check which of them does not produce the expected output. After we nd the piece with the error, we subdivide it further and repeat the process until we nd the bug. The emphasis in this method is on convenient: it doesnt necessarily have to make sense. As long as you can continue to divide your problem area into between two and ve parts and localize the problem in one of the parts, it wont take long to nd the bug. So whats a convenient way to look at the problems? That depends on the tools you have at your disposal: If you have a symbolic debugger, you can divide your problem into the individual functions and examine what goes in and what goes out. If you have a system call trace program, such as ktrace or truss, you can monitor what the program says to the system and what the system replies.

5 February 2005 02:09

Chapter 8: Testing

107

If you have a communications line trace program, you can try to divide your program into pieces that communicate across this line, so you can see what they are saying to each other.

Of course, we have all these things. In the following sections well look at each of them in more detail.

Symbolic debuggers
If you dont have a symbolic debugger, get one. Now. Many people still claim to be able to get by without a debugger, and its horrifying how many people dont even know how to use one. Of course you can debug just about anything without a symbolic debugger. Historians tell us that you can build pyramids without wheelsthats a comparable level of technology to testing without a debugger. The GNU debugger, gdb, is available on just about every platform youre likely to encounter, and though its not perfect, it runs rings around techniques like putting printf statements in your programs. In UNIX, a debugger is a process that takes control of the execution of another process. Most versions of UNIX allow only one way for the debugger to take control: it must start the process that it debugs. Some versions, notably SunOS 4, but not Solaris 2, also allow the debugger to attach to a running process. Whichever debugger you use, there are a surprisingly small number of commands that you need. In the following discussion, well look at the command set of gdb, since it is widely used. The commands for other symbolic debuggers vary considerably, but they normally have similar purposes. A stack trace command answers the question, Where am I, and how did I get here?, and is almost the most useful of all commands. Its certainly the rst thing you should do when examining a core dump or after getting a signal while debugging the program. gdb implements this function with the backtrace command. Displaying data is the most obvious requirement: what is the current value of the variable bar? In gdb, you do this with the print command. Displaying register contents is really the same thing as displaying program data. In gdb, you display individual registers with the print command, or all registers with the info registers command. Modifying data and register contents is an obvious way of modifying program execution. In gdb, you do this with the set command. breakpoints stop execution of the process when the process attempts to execute an instruction at a certain address. gdb sets breakpoints with the break command. Many modern machines have hardware support for more sophisticated breakpoint mechanisms. For example, the i386 architecture can support four hardware breakpoints on instruction fetch (in other words, traditional breakpoints), memory read or memory write. These features are invaluable in systems that support them; unfortunately, UNIX usually

5 February 2005 02:09

108

does not. gdb simulates this kind of breakpoint with a so-called watchpoint. When watchpoints are set, gdb simulates program execution by single-stepping through the program. When the condition (for example, writing to the global variable foo) is fullled, the debugger stops the program. This slows down the execution speed by several orders of magnitude, whereas a real hardware breakpoint has no impact on the execution speed.* Jumping (changing the address from which the next instruction will be read) is really a special case of modifying register contents, in this case the program counter (the register that contains the address of the next instruction). This register is also sometimes called the instruction pointer, which makes more sense. In gdb, use the jump command to do this. Use this instruction with care: if the compiler expects the stack to look different at the source and at the destination, this can easily cause incorrect execution. Single stepping in its original form is supported in hardware by many architectures: after executing a single instruction, the machine automatically generates a hardware interrupt that ultimately causes a SIGTRAP signal to the debugger. gdb performs this function with the stepi command. You wont want to execute individual machine instructions until you are in deep trouble. Instead, you will execute a single line instruction, which effectively single steps until you leave the current line of source code. To add to the confusion, this is also frequently called single stepping. This command comes in two avours, depending on how it treats function calls. One form will execute the function and stop the program at the next line after the call. The other, more thorough form will stop execution at the rst executable line of the function. Its important to notice the difference between these two functions: both are extremely useful, but for different things. gdb performs single line execution omitting calls with the next command, and includes calls with the step command.

There are two possible approaches when using a debugger. The easier one is to wait until something goes wrong, then nd out where it happened. This is appropriate when the process gets a signal and does not overwrite the stack: the backtrace command will show you how it got there. Sometimes this method doesnt work well: the process may end up in no-mans-land, and you see something like:
Program received signal SIGSEGV, Segmentation fault. 0x0 in ?? () (gdb) bt abbreviation for backtrace #0 0x0 in ?? () nowhere (gdb)

Before dying, the process has mutilated itself beyond recognition. Clearly, the rst approach wont work here. In this case, we can start by conceptually dividing the program into a number of parts: initially we take the function main and the set of functions which main calls. By single stepping over the function calls until something blows up, we can localize the function in which the problem occurs. Then we can restart the program and single step through this
* Some architectures slow the overall execution speed slightly in order to test the hardware registers. This effect is negligible.

5 February 2005 02:09

Chapter 8: Testing

109

function until we nd what it calls before dying. This iterative approach sounds slow and tiring, but in fact it works surprisingly well.

Libraries and debugging information

Lets come back to our xterm program and use gdb to gure out what is going on. We could, of course, look at the core dump, but in this case we can repeat the problem at will, so were better off looking at the live program. We enter:
$ gdb xterm (political statement for the FSF omitted) (gdb) r -display allegro:0 run the program Starting program: /X/X11/X11R6/xc/programs/xterm/xterm -display allegro:0 Program received signal SIGBUS, Bus error. 0x3b0bc in _XtMemmove () (gdb) bt look back down the stack #0 0x3b0bc in _XtMemmove () all these functions come from the X toolkit #1 0x34dcd in XtScreenDatabase () #2 0x35107 in _XtPreparseCommandLine () #3 0x4e2ef in XtOpenDisplay () #4 0x4e4a1 in _XtAppInit () #5 0x35700 in XtOpenApplication () #6 0x357b5 in XtAppInitialize () #7 0x535 in main () (gdb)

The stack trace shows that the main program called XtAppInitialize, and the rest of the stack shows the program deep in the X Toolkit, one of the central X11 libraries. If this were a program that you had just written, you could expect it to be a bug in your program. In this case, where we have just built the complete X11 core system, theres also every possibility that it is a library bug. As usual, the library was compiled without debug information, and without that you hardly have a hope of nding it. Apart from size constraints, there is no reason why you cant include debugging information in a library. The object les in libraries are just the same as any others we discuss them in detail on page 369. If you want, you can build libraries with debugging information, or you can take individual library routines and compile them separately. Unfortunately, the size constraints are signicant: without debugging information, the le libXt.a is about 330 kB long and contains 53 object les. With debugging information, it might easily reach 20 MB, since all the myriad X11 global symbols would be included with each object le in the archive. Its not just a question of disk space: you also need virtual memory during the link phase to accommodate all these symbols. Most of these les dont interest us anyway: the rst one that does is the one that contains _XtMemmove. So we nd where it is and compile it alone with debugging information. Thats not as simple as it sounds: rst we need to nd the source le, and to do that we need to nd the source directory. We could read the documentation, but to do that we need to know that the Xt functions are in fact the X toolkit. If were using GNU make, or if our Makele

5 February 2005 02:09

110

documents directory changes, an alternative would be to go back to our make log and look for the text Xt. If we do this, we quickly nd
make[4]: Leaving directory /X/X11R6/xc/lib/Xext making Makefiles in lib/Xt... mv Makefile Makefile.bak make[4]: Entering directory /X/X11R6/xc/lib/Xt make[4]: Nothing to be done for Makefiles. make[4]: Leaving directory /X/X11R6/xc/lib/Xt

So the directory is /X/X11R6/xc/lib/Xt. The next step is to nd the le that contains XtMemmove. There is a possibility that it is called XtMemmove.c, but in this case there is no such le. Well have to grep for it. Some versions of grep have an option to descend recursively into subdirectories, which can be very useful if you have one available. Another useful tool is cscope, which is supplied with System V.
$ grep XtMemmove *.c Alloc.c:void _XtMemmove(dst, src, length) Convert.c: XtMemmove(&p->from.addr, from->addr, from->size); ... many more references to XtMemmove

So XtMemmove is in Alloc.c. By the same method, we look for the other functions mentioned in the stack trace and discover that we also need to recompile Initialize.c and Display.c. In order to compile debugging information, we add the compiler option -g. At the same time, we remove -O. gcc doesnt require this, but its usually easier to debug a non-optimized program. We have three choices of how to set the options: We can modify the Makele (make World, the main make target for X11, rebuilds the Makeles from the corresponding Imakeles, so this is not overly dangerous). If we have a working version of xterm, we can use its facilities: rst we start the compilation with make, but we dont need to wait for the compilation to complete: as soon as the compiler invocation appears on the screen, we abort the build with CTRL-C. Using the xterm copy function, we copy the compiler invocation to the command line and add the options we want:
$ rm Alloc.o Initialize.o Display.o remove the old objects $ make and start make normally rm -f Alloc.o gcc -DNO_ASM -fstrength-reduce -fpcc-struct-return -c -I../.. \ -DNO_AF_UNIX -DSYSV -DSYSV386 -DUSE_POLL Alloc.c C interrupt make with CTRL-C make: *** [Alloc.o] Interrupt copy the invocation lines above with the mouse, and paste below, then modify as shown in bold print $ gcc -DNO_ASM -fstrength-reduce -fpcc-struct-return -c -I../.. \ -DNO_AF_UNIX -DSYSV -DSYSV386 -DUSE_POLL Alloc.c -g

You can also use make -n, which just shows the commands that make would execute, rather than aborting the make, but you frequently nd that make -n prints out a whole lot of stuff you dont expect. When you have made Alloc.o, you can repeat the process

5 February 2005 02:09

Chapter 8: Testing

111

for the other two object les. We could change CFLAGS from the make command line. Our rst attempt doesnt work too well, though. If you compare the following line with the invocation above, youll see that a whole lot of options are missing. They were all in CFLAGS; by redening CFLAGS, we lose them all:
$ make CFLAGS=-g rm -f Alloc.o gcc -DNO_ASM -fstrength-reduce -fpcc-struct-return -c -g Alloc.c

CFLAGS included all the compiler options starting from -I/../.., so we need to write:
$ make CFLAGS=-g -c -I../.. -DNO_AF_UNIX -DSYSV -DSYSV386 -DUSE_POLL

When we have created all three new object les, we can let make complete the library for us. It will not try to remake these object les, since now they are newer than any of their dependencies:
$ make run make to build a new library rm -f libXt.a ar clq libXt.a ActionHook.o Alloc.o ArgList.o Callback.o ClickTime.o Composite.o \ Constraint.o Convert.o Converters.o Core.o Create.o Destroy.o Display.o Error.o \ Event.o EventUtil.o Functions.o GCManager.o Geometry.o GetActKey.o GetResList.o \ GetValues.o HookObj.o Hooks.o Initialize.o Intrinsic.o Keyboard.o Manage.o \ NextEvent.o Object.o PassivGrab.o Pointer.o Popup.o PopupCB.o RectObj.o \ Resources.o Selection.o SetSens.o SetValues.o SetWMCW.o Shell.o StringDefs.o \ Threads.o TMaction.o TMgrab.o TMkey.o TMparse.o TMprint.o TMstate.o VarCreate.o \ VarGet.o Varargs.o Vendor.o ranlib libXt.a rm -f ../../usrlib/libXt.a cd ../../usrlib; ln ../lib/Xt/libXt.a . $

Now we have a copy of the X Toolkit in which these three les have been compiled with symbols. Next, we need to rebuild xterm. Thats straightforward enough:
$ cd ../../programs/xterm/ $ pwd /X/X11R6/xc/programs/xterm $ make rm -f xterm gcc -DNO_ASM -fstrength-reduce -fpcc-struct-return -fwritable-strings -o xterm \ -L../../usrlib main.o input.o charproc.o cursor.o util.o tabs.o screen.o \ scrollbar.o button.o Tekproc.o misc.o VTPrsTbl.o TekPrsTbl.o data.o menu.o -lXaw \ -lXmu -lXt -lSM -lICE -lXext -lX11 -L/usr/X11R6/lib -lpt -ltermlib

Finally, we try again. Since the library is not in the current directory, we use the dir command to tell gdb where to nd the sources. Now we get:
$ gdb xterm (gdb) dir ../../lib/X11 set source paths Source directories searched: /X/X11/X11R6/xc/programs/xterm/../../lib/X11:$cdir:$cwd (gdb) dir ../../lib/Xt

5 February 2005 02:09

112
Source directories searched: /X/X11/X11R6/xc/programs/xterm/../../lib/Xt/X/X11/X11R6/xc/programs/xterm/../..\ /lib/X11:$cdir:$cwd (gdb) r and run the program Starting program: /X/X11/X11R6/xc/programs/xterm/xterm Program received signal SIGBUS, Bus error. 0x3ced6 in _XtMemmove (dst=0x342d8 "E 03", src=0x41c800 "", length=383) \ at Alloc.c:101 101 *dst++ = *src++; (gdb)

This shows a typical byte for byte memory move. About the only thing that could cause a bus error on that statement would be an invalid address, but the parameters show that they appear to be valid. There are at two possible gotchas here: The debugger may be lying. The parameters it shows are the parameters on the stack. If the code has been optimized, there is a very good chance that the source and destination addresses are stored in registers, and thus the value of dst on the stack is not up to date. The destination address may be in the text segment, in which case an attempt to write to it will cause some kind of error. Depending on the system it could be a segmentation violation or a bus error.

The most reliable way to nd out what is really going on is to look at the machine instructions being executed. First we tell the debugger to look at current instruction and the following ve instructions:
(gdb) x/6i $eip 0x3ced6 <_XtMemmove+74>: 0x3ced8 <_XtMemmove+76>: 0x3cedb <_XtMemmove+79>: 0x3cede <_XtMemmove+82>: 0x3cee0 <_XtMemmove+84>: 0x3cee1 <_XtMemmove+85>: list the movb incl incl jmp leave ret next 6 instructions %al,(%edx) 0xc(%ebp) 0x8(%ebp) 0x3cec2 <_XtMemmove+54>

The rst instruction is a byte move, from register al to the address stored in register edx. Lets look at the address in edx:
(gdb) p/x $edx $9 = 0x342d8

Well, this is our dst address alrightwhy cant it store there? It would be nice to be able to try to set values in memory and see if the debugger can do it:
(gdb) set *dst = Xb (gdb) p *dst $13 = 88 X

That looks writable enough. Unfortunately, you cant rely on the debugger to tell the truth. Debuggers must be able to write to the text segment. If the write had failed, you could have been sure that the address was not writable, but if the write succeeds, you cant be sure. What

5 February 2005 02:09

Chapter 8: Testing

113

we need to know are the exact segment limits. Some debuggers show you the segment limits, but current versions of gdb do not. An alternative is the size command:
$ size xterm text data 846204 56680 bss 23844 dec hex 926728 e2408 filename xterm

The text segment is 846204 decimal bytes long (0xce97c), and on this system (SCO UNIX) it starts at address 0, so the address is, indeed, in the text segment. But where did it come from? To nd an answer to that question, we need to look at the calling function. In gdb, we do this with the frame command:
(gdb) f 1 look at the calling function (frame 1) #1 0x35129 in _MergeOptionTables (src1=0x342d8, num_src1=24, src2=0x400ffe, num_src2=64, dst=0x7ffff9c0, num_dst=0x7ffff9bc) at Initialize.c:602 602 (void) memmove(table, src1, sizeof(XrmOptionDescRec) * num_src1 );

Thats funnylast time it died, the function was called from XtScreenDatabase,* not from _MergeOptionTables. Why? At the moment its difcult to say for sure, but its possible that this difference happened because we removed optimization. In any case, we still have a problem, so we should x this one rst and then go back and look for the other one if solving this problem isnt enough. In this case, the frame command doesnt help much, but it does tell us that the destination variable is called table, and implicitly that memmove has been dened as _XtMemmove in this source le. We could now look at the source le in an editor in a different X window, but its easier to list the instructions around the current line with the list command:
(gdb) l 597 598 599 600 601 602 603 604 605 606 enum {Check, NotSorted, IsSorted} sort_order = Check; *dst = table = (XrmOptionDescRec*) XtMalloc( sizeof(XrmOptionDescRec) * (num_src1 + num_src2) ); (void) memmove(table, src1, sizeof(XrmOptionDescRec) * num_src1 ); if (num_src2 == 0) { *num_dst = num_src1; return; }

So, the address is returned by the function XtMallocit seems to be allocating storage in the text segment. At this point, we could examine it more carefully, but lets rst be sure that were looking at the right problem. The address in table should be the same as the address in the parameter dst of XtMemmove. Were currently examining the environment of _MergeOptionTables, so we can look at it directly:
(gdb) p table $29 = (XrmOptionDescRec *) 0x41c800

That looks just ne. Where did this strange dst address come from? Lets set a breakpoint
* See frame 1 in the stack trace on page 109.

5 February 2005 02:09

114

on the call to memmove on line 602, and then restart the program:
Example 81:
(gdb) b 602 Breakpoint 8 at 0x35111: file Initialize.c, line 602. (gdb) r The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: /X/X11/X11R6/xc/programs/xterm/xterm Breakpoint 8, _MergeOptionTables (src1=0x342d8, num_src1=24, src2=0x400ffe, num_src2=64, dst=0x7ffff9c0, num_dst=0x7ffff9bc) at Initialize.c:602 602 (void) memmove(table, src1, sizeof(XrmOptionDe (gdb) p table look again, to be sure $31 = (XrmOptionDescRec *) 0x41c800 (gdb) s single step into memmove _XtMemmove (dst=0x342d8 "E 03", src=0x41c800 "", length=384) at Alloc.c:94 94 if (src < dst) {

This is really strange! table has a valid address in the data segment, but the address we pass to _XtMemmove is in the text segment and seems unrelated. Its not clear what we should look at next: The source of the function calls memmove, but after preprocessing it ends up calling _XtMemmove. memmove might simply be dened as _XtMemmove, but it might also be dened with parameters, in which case some subtle type conversions might result in our problem. If you understand the assembler of the system, it might be instructive to look at the actual instructions that the compiler produces.

Its denitely quicker to look at the assembler instructions than to ght your way through the thick undergrowth in the X11 source tree:
(gdb) x/8i $eip look at the next 0x35111 <_MergeOptionTables+63>: movl 0x35114 <_MergeOptionTables+66>: movl 0x35117 <_MergeOptionTables+69>: movl 0x3511a <_MergeOptionTables+72>: shll 0x3511d <_MergeOptionTables+75>: pushl 0x3511e <_MergeOptionTables+76>: pushl 0x35121 <_MergeOptionTables+79>: pushl 0x35124 <_MergeOptionTables+82>: call 8 instructions 0xc(%ebp),%edx %edx,0xffffffd8(%ebp) 0xffffffd8(%ebp),%edx $0x4,%edx %edx 0xfffffffc(%ebp) 0x8(%ebp) 0x3ce8c <_XtMemmove>

This isnt easy stuff to handle, but its worth understanding, so well pull it apart, instruction for instruction. Its easier to understand this discussion if you refer to the diagrams of stack structure in Chapter 21, Object les and friends, page 377. movl 0xc(%ebp),%edx takes the content of the stack word offset 12 in the current stack frame and places it in register edx. As we have seen, this is num_src1, the second

5 February 2005 02:09

Chapter 8: Testing

115

parameter passed to _MergeOptionTables. movl %edx,0xffffffd8(%ebp) stores the value of edx at offset -40 in the current stack frame. This is for temporary storage. movl 0xffffffd8(%ebp),%edx does exactly the opposite: it loads register edx from the location where it just stored it. These two instructions are completely redundant. They are also a sure sign that the function was compiled without optimization. shll $0x4,%edx shifts the contents of register edx left by 4 bits, multiplying it by 16. If we compare this to the source, its evident that the value of XrmOptionDescRec is 16, and that the compiler has taken a short cut to evaluate the third parameter of the call. pushl %edx pushes the contents of edx onto the stack. pushl 0xfffffffc(%ebp) pushes the value of the word at offset -4 in the current stack frame onto the stack. This is the value of table, as we can conrm by looking at the instructions generated for the previous line. pushl 0x8(%ebp) pushes the value of the rst parameter, src1, onto the stack. Finally, call _XtMemmove calls the function. Expressed in C, we now know that it calls
memmove (src1, table, num_src1 << 4);

This is, of course, wrong: the parameter sequence of source and destination has been reversed. Lets look at _XtMemmove more carefully:
(gdb) l _XtMemmove 89 #ifdef _XNEEDBCOPYFUNC 90 void _XtMemmove(dst, src, length) 91 char *dst, *src; 92 int length; 93 { 94 if (src < dst) { 95 dst += length; 96 src += length; 97 while (length--) 98 *--dst = *--src; 99 } else { 100 while (length--) 101 *dst++ = *src++; 102 } 103 } 104 #endif

Clearly the function parameters are the same as those of memmove, but the calling sequence has reversed them. Weve found the problem, but we havent found whats causing it. Aside: Debugging is not an exact science. Weve found our problem, though we still dont know whats causing it. But looking back at Example 8-1, we see that the address for src on entering _XtMemmove was the same as the address of table. That tells us as much as analyzing the machine code did. This will happen again and again: after you nd a problem, you

5 February 2005 02:09

116

discover you did it the hard way. The next thing we need to gure out is why the compiler reversed the sequence of the parameters. Can this be a compiler bug? Theoretically, yes, but its very unlikely that such a primitive bug should go undiscovered up to now. Remember that the compiler does not compile the sources you see: it compiles whatever the preprocessor hands to it. It makes a lot of sense to look at the preprocessor output. To do this, we go back to the library directory. Since we used pushd, this is easyjust enter pushd. In the library, we use the same trick as before in order to run the compiler with different options, only this time we use the options -E (stop after running the preprocessor), -dD (retain the text of the denitions in the preprocessor output), and -C (retain comments in the preprocessor output). In addition, we output to a le junk.c:
$ pushd $ rm Initialize.o $ make Initialize.o rm -f Initialize.o gcc -DNO_ASM -fstrength-reduce -fpcc-struct-return -c -g -I../.. \ -D_SVID -DNO_AF_UNIX -DSYSV -DSYSV386 -DUSE_POLL Initialize.c make: *** [Initialize.o] Interrupt hit CTRL-C ... copy the command into the command line, and extend: $ gcc -DNO_ASM -fstrength-reduce -fpcc-struct-return -c -g -I../.. \ -D_SVID -DNO_AF_UNIX -DSYSV -DSYSV386 -DUSE_POLL Initialize.c \ -E -dD -C >junk.c $

As you might have guessed, we now look at the le junk.c with an editor. Were looking for memmove, of course. We nd a denition in /usr/include/string.h, then later on we nd, in /X/X11/X11R6/xc/X11/Xfuncs.h,
#define memmove(dst,src,len) bcopy((char *)(src),(char *)(dst),(int)(len))

#define memmove(dst,src,len) _XBCOPYFUNC((char *)(src),(char *)(dst),(int)(len)) #define _XNEEDBCOPYFUNC

For some reason, the conguration les have decided that memmove is not dened on this system, and have replaced it with bcopy (which is really not dened on this system). Then they replace it with the substitute function _XBCOPYFUNC, almost certainly a preprocessor denition. It also denes the preprocessor variable _XNEEDBCOPYFUNC to indicate that _XtMemmove should be compiled. Unfortunately, we dont see what happens with _XNEEDBCOPYFUNC. The preprocessor discards all #ifdef lines. It does include #defines, however, so we can look for where _XBCOPYFUNC is denedits in IntrinsicI.h, as the last #line directive before the denition indicates.
#define _XBCOPYFUNC _XtMemmove

IntrinsicI.h also contains a number of denitions for XtMemmove, none of which are used in the current environment, but all of which have the parameter sequence (dst, src, count). bcopy has the parameter sequence (src, dst, count). Clearly, somebody has confused

5 February 2005 02:09

Chapter 8: Testing

117

something in this header le, and under certain rare circumstances the call is dened with the incorrect parameter sequence. Somewhere in here is a lesson to be learnt: this is a real bug that occurred in X11R6, patch level 3, one of the most reliable and most portable software packages available, yet here we have a really primitive bug. The real problem lies in the conguration mechanism: automated conguration can save a lot of time in normal circumstances, but it can also cause lots of pain if it makes incorrect assumptions. In this case, the environment was unusual: the kernel platform was SCO UNIX, which has an old-fashioned library, but the library was GNU libc. This caused the assumptions of the conguration mechanism to break down. Lets look more carefully at the part of Xfuncs.h where we found the denitions:
/* the new Xfuncs.h */ #if !defined(X_NOT_STDC_ENV) && (!defined(sun) || defined(SVR4)) /* the ANSI C way */ #ifndef _XFUNCS_H_INCLUDED_STRING_H #include <string.h> #endif #undef bzero #define bzero(b,len) memset(b,0,len) #else /* else X_NOT_STDC_ENV or SunOS 4 */ #if defined(SYSV) || defined(luna) || defined(sun) || defined(__sxg__) #include <memory.h> #define memmove(dst,src,len) bcopy((char *)(src),(char *)(dst),(int)(len)) #if defined(SYSV) && defined(_XBCOPYFUNC) #undef memmove #define memmove(dst,src,len) _XBCOPYFUNC((char *)(src),(char *)(dst),(int)(len)) #define _XNEEDBCOPYFUNC #endif #else /* else vanilla BSD */ #define memmove(dst,src,len) bcopy((char *)(src),(char *)(dst),(int)(len)) #define memcpy(dst,src,len) bcopy((char *)(src),(char *)(dst),(int)(len)) #define memcmp(b1,b2,len) bcmp((char *)(b1),(char *)(b2),(int)(len)) #endif /* SYSV else */ #endif /* ! X_NOT_STDC_ENV else */

This is hairy (and incorrect) stuff. It makes its decisions based on the variables X_NOT_STDC_ENV, sun, SVR4, SYSV, luna, __sxg__ and _XBCOPYFUNC. These are the decisions: If the variable is not dened, it assumes ANSI C, unless this is a pre-SVR4 Sun machine. Otherwise it checks the variables SYSV (for System V.3), luna, sun or __sxg__. If any of these are set, it includes the le memory.h and denes memmove in terms of bcopy. If _XBCOPYFUNC is dened, it redenes memmove as _XBCOPYFUNC, reversing the parameters as it goes. If none of these conditions apply, it assumes a vanilla BSD machine and denes the functions memmove, memcpy and memcmp in terms of the BSD functions bcopy and bcmp.

There are two errors here:

5 February 2005 02:09

118

The only way that _XBCOPYFUNC is ever dened is as _XtMemmove, which does not have the same parameter sequence as bcopyinstead, it has the same parameter sequence as memmove. We can x this part of the header by changing the denition line to
#define memmove(dst,src,len) _XBCOPYFUNC((char *)(dst),(char *)(src),(int)(len))

or even to
#define memmove _XBCOPYFUNC

There is no reason to assume that this system does not use ANSI C: its using gcc and GNU libc.a, both of them very much standard compliant. We need to examine this point in more detail:

Going back to our junk.c, we search for X_NOT_STDC_ENV and nd it dened at line 85 of /X/X11/X11R6/xc/X11/Xosdefs.h:
#ifdef SYSV386 #ifdef SYSV #define X_NOT_POSIX #define X_NOT_STDC_ENV #endif #endif

In other words, this bug is likely to occur only with System V.3 implementations on Intel architecture. This is a fairly typical way to make decisions about the system, but it is wrong: X_NOT_STDC_ENV relates to a compiler, not an operating system, but both SYSV386 and SYSV dene operating system characteristics. At rst sight it would seem logical to modify the denitions like this:
#ifdef SYSV386 #ifdef SYSV #ifndef __GNU_LIBRARY__ #define X_NOT_POSIX #endif #ifndef __GNUC__ #define X_NOT_STDC_ENV #endif #endif #endif

This would only dene the variables if the library is not GNU libc or the compiler is not gcc. This is still not correct: the relationship between __GNUC__ and X_NOT_STDC_ENV or __GNU_LIBRARY__ and X_NOT_POSIX is not related to System V or the Intel architecture. Instead, it makes more sense to backtrack at the end of the le:
#ifdef #undef #endif #ifdef #undef #endif __GNU_LIBRARY__ X_NOT_POSIX __GNUC__ X_NOT_STDC_ENV

Whichever way we look at it, this is a mess. Were applying cosmetic patches to a

5 February 2005 02:09

Chapter 8: Testing

119

conguration mechanism which is based in incorrect assumptions. Until some better conguration mechanism comes along, unfortunately, were stuck with this situation.

Limitations of debuggers
Debuggers are useful tools, but they have their limitations. Here are a couple which could cause you problems:

Cant breakpoint beyond fork

UNIX packages frequently start multiple processes to do the work on hand. Frequently enough, the program that you start does nothing more than to spawn a number of other processes and wait for them to stop. Unfortunately, the ptrace interface which debuggers use requires the process to be started by the debugger. Even in SunOS 4, where you can attach the debugger to a process that is already running, there is no way to monitor it from the start. Other systems dont offer even this facility. In some cases you can determine how the process was started and start it with the debugger in the same manner. This is not always possible for example, many child processes communicate with their parent. Unfortunately, SunOS trace doesnt support tracing through fork. truss does it better than ktrace. In extreme cases (like debugging a program of this nature on SunOS 4, where there is no support for trace through fork), you might nd it an advantage to port to a different machine running an operating system such as Solaris 2 in order to be able to test with truss. Of course, Murphys law says that the bug wont show up under Solaris 2.

Terminal logs out

The debugger usually shares a terminal with the program being tested. If the program changes the driver conguration, the debugger should change it back again when it gains control (for example, on hitting a breakpoint), and set it back to the way the program set it before continuing. In some cases, however, it cant: if the process has taken ownership of the terminal with a system call like setsid (see Chapter 12, Kernel dependencies, page 171), it will no longer have access to the terminal. Under these circumstances, most debuggers crawl into a corner and die. Then the shell in control of the terminal awakes and dies too. If youre running in an xterm, the xterm then stops; if youre running on a glass tty, you will be logged out. The best way out of this dilemma is to start the child process on a different terminal, if your debugger and your hardware conguration support it. To do this with an xterm requires starting a program which just sleeps, so that the window stays open until you can start your test program:
$ xterm -e sleep 100000& [1] 27013 $ ps aux|grep sleep grog 27025 3.0 0.0 264 132 p6 S+ root 27013 0.0 0.0 1144 740 p6 I grog 27014 0.0 0.0 100 36 p8 Is+ $ gdb myprog (gdb) r < /dev/ttyp8 > /dev/ttyp8

1:13PM 1:12PM 1:12PM

0:00.03 grep sleep 0:00.37 xterm -e sleep 100000 0:00.06 sleep 100000

5 February 2005 02:09

120

This example was done on a BSD machine. On a System V machine you will need to use ps -ef instead of ps aux. First, you start an xterm with sleep as controlling shell (so that it will stay there). With ps you grep for the controlling terminal of the sleep process (the third line in the example), and then you start your program with stdin and stdout redirected to this terminal.

Cant interrupt process

The ptrace interface uses the signal SIGTRAP to communicate with the process being debugged. What happens if you block this signal, or ignore it? Nothing the debugger doesnt work any more. Its bad practice to block SIGTRAP, of course, but it can be done. More frequently, though, youll encounter this problem when a process gets stuck in a signal processing loop and doesnt get round to processing the SIGTRAPprecisely one of the times when you would want to interrupt it. My favourite one is the program which had a SIGSEGV handler which went and retried the instruction. Unfortunately, the only signal to which a process in this state will still respond is SIGKILL, which doesnt help you much in nding out whats going on.

Tracing system calls

An alternative approach is to divide the program between system code and user code. Most systems have the ability to trace the parameters supplied to each system call and the results that they return. This is not nearly as good as using a debugger, but it works with all object les, even if they dont have symbols, and it can be very useful when youre trying to gure out why a program doesnt open a specic le. Tracing is a very system-dependent function, and there are a number of different programs to perform the trace: truss runs on System V.4, ktrace runs on BSD NET/2 and 4.4BSD derived systems, and trace runs on SunOS 4. They vary signicantly in their features. Well look briey at each. Other systems supply still other programsfor example, SGIs IRIX operating system supplies the program par, which offers similar functionality.

trace
trace is a relatively primitive tool supplied with SunOS 4 systems. It can either start a process or attach to an existing process, and it can print summary information or a detailed trace. In particular, it cannot trace the child of a fork call, which is a great disadvantage. Heres an example of trace output with a possibly recognizable program:
$ trace hello open ("/usr/lib/ld.so", 0, 040250) = 3 read (3, "".., 32) = 32 mmap (0, 40960, 0x5, 0x80000002, 3, 0) = 0xf77e0000 mmap (0xf77e8000, 8192, 0x7, 0x80000012, 3, 32768) = 0xf77e8000 open ("/dev/zero", 0, 07) = 4 getrlimit (3, 0xf7fff488) = 0 mmap (0xf7800000, 8192, 0x3, 0x80000012, 4, 0) = 0xf7800000

5 February 2005 02:09

Chapter 8: Testing
close (3) = 0 getuid () = 1004 getgid () = 1000 open ("/etc/ld.so.cache", 0, 05000100021) = 3 fstat (3, 0xf7fff328) = 0 mmap (0, 4096, 0x1, 0x80000001, 3, 0) = 0xf77c0000 close (3) = 0 open ("/opt/lib/gcc-lib/sparc-sun-sunos".., 0, 01010525) = 3 fstat (3, 0xf7fff328) = 0 getdents (3, 0xf7800108, 4096) = 212 getdents (3, 0xf7800108, 4096) = 0 close (3) = 0 open ("/opt/lib", 0, 056) = 3 getdents (3, 0xf7800108, 4096) = 264 getdents (3, 0xf7800108, 4096) = 0 close (3) = 0 open ("/usr/lib/libc.so.1.9", 0, 023170) = 3 read (3, "".., 32) = 32 mmap (0, 458764, 0x5, 0x80000002, 3, 0) = 0xf7730000 mmap (0xf779c000, 16384, 0x7, 0x80000012, 3, 442368) = 0xf779c000 close (3) = 0 open ("/usr/lib/libdl.so.1.0", 0, 023210) = 3 read (3, "".., 32) = 32 mmap (0, 16396, 0x5, 0x80000002, 3, 0) = 0xf7710000 mmap (0xf7712000, 8192, 0x7, 0x80000012, 3, 8192) = 0xf7712000 close (3) = 0 close (4) = 0 getpagesize () = 4096 brk (0x60d8) = 0 brk (0x70d8) = 0 ioctl (1, 0x40125401, 0xf7ffea8c) = 0 write (1, "Hello, World!0, 14) = Hello, World! 14 close (0) = 0 close (1) = 0 close (2) = 0 exit (1) = ?

121

Whats all this output? All we did was a simple write, but we have performed a total of 43 system calls. This shows in some detail how much the viewpoint of the world differs when youre on the other side of the system library. This program, which was run on a SparcStation 2 with SunOS 4.1.3, rst sets up the shared libraries (the sequences of open, read, mmap, and close), then initializes the stdio library (the calls to getpagesize, brk, ioctl, and fstat), and nally writes to stdout and exits. It also looks strange that it closed stdin before writing the output text: again, this is a matter of perspective. The stdio routines buffer the text, and it didnt actually get written until the process exited, just before closing stdout.

5 February 2005 02:09

122

ktrace
ktrace is supplied with newer BSD systems. Unlike the other trace programs, it writes unformatted data to a log le (by default, ktrace.out), and you need to run another program, kdump, to display the log le. It has the following options: It can trace the descendents of the process it is tracing. This is particularly useful when the bug occurs in large complexes of processes, and you dont even know which process is causing the problem. It can attach to processes that are already running. Optionally, it can also attach to existing children of the processes to which it attaches. It can specify broad subsets of system calls to trace: system calls, namei translations (translation of le name to inode number), I/O, and signal processing.

Heres an example of ktrace running against the same program:

$ ktrace hello Hello, World! $ kdump 20748 ktrace 20748 ktrace 20748 ktrace 20748 ktrace 20748 ktrace 20748 ktrace 20748 ktrace 20748 ktrace 20748 ktrace 20748 ktrace 20748 ktrace 20748 hello 20748 hello 20748 hello 20748 hello 20748 hello 20748 hello 20748 hello 20748 hello 20748 hello 20748 hello 20748 hello 20748 hello 20748 hello 20748 hello 20748 hello "Hello, " 20748 hello 20748 hello

RET CALL RET CALL RET CALL RET CALL RET CALL NAMI RET CALL RET CALL RET CALL RET CALL RET CALL RET CALL RET CALL GIO World!

ktrace 0 getpagesize getpagesize 4096/0x1000 break(0xadfc) break 0 break(0xaffc) break 0 break(0xbffc) break 0 execve(0xefbfd148,0xefbfd5a8,0xefbfd5b0) "./hello" execve 0 fstat(0x1,0xefbfd2a4) fstat 0 getpagesize getpagesize 4096/0x1000 break(0x7de4) break 0 break(0x7ffc) break 0 break(0xaffc) break 0 ioctl(0x1,TIOCGETA,0xefbfd2e0) ioctl 0 write(0x1,0x8000,0xe) fd 1 wrote 14 bytes

RET write 14/0xe CALL exit(0xe)

This display contains the following information in columnar format:

5 February 2005 02:09

Chapter 8: Testing

123

1. 2. 3.

The process ID of the process. The name of the program from which the process was started. We can see that the name changes after the call to execve. The kind of event. CALL is a system call, RET is a return value from a system call, NAMI is a system internal call to the function namei, which determines the inode number for a pathname, and GIO is a system internal I/O call. The parameters to the call.

4.

In this trace, run on an Intel 486 with BSD/OS 1.1, we can see a signicant difference from SunOS: there are no shared libraries. Even though each system call produces two lines of output (the call and the return value), the output is much shorter.

truss
truss, the System V.4 trace facility, offers the most features: It can print statistical information instead of a trace. It can display the argument and environment strings passed to each call to exec. It can trace the descendents of the process it is tracing. Like ktrace, it can attach to processes which are already running and optionally attach to existing children of the processes to which it attaches. It can trace specic system calls, signals, and interrupts (called faults in System V terminology). This is a very useful feature: as we saw in the ktrace example above, the C library may issue a surprising number of system calls.

Heres an example of truss output:

$ truss -f hello 511: execve("./hello", 0x08047834, 0x0804783C) argc = 1 511: getuid() = 1004 511: getuid() = 1004 511: getgid() = 1000 511: getgid() = 1000 511: sysi86(SI86FPHW, 0x80036058, 0x80035424, 0x8000E255) = 511: ioctl(1, TCGETA, 0x08046262) = 0 Hello, World! 511: write(1, " H e l l o , W o r l d".., 14) = 14 511: _exit(14)

[ 1004 ] [ 1004 ] [ 1000 ] [ 1000 ] 0x00000000

truss offers a lot of choice in the amount of detail it can display. For example, you can select a verbose parameter display of individual system calls. If were interested in the parameters to the ioctl call, we can enter:
$ truss -f -v ioctl hello ... 516: ioctl(1, TCGETA, 0x08046262)

= 0

5 February 2005 02:09

124
516: 516: iflag=0004402 oflag=0000005 cflag=0002675 lflag=0000073 line=0 cc: 177 003 010 030 004 000 000 000

In this case, truss shows the contents of the termio structure associated with the TCGETA request see Chapter 15, Terminal drivers, pages 241 and 258, for the interpretation of this information.

Tracing through fork

Weve seen that ktrace and truss can both trace the child of a fork system call. This is invaluable: as we saw on page 119, debuggers cant do this. Unfortunately, SunOS trace doesnt support tracing through fork. truss does it better than ktrace. In extreme cases (like debugging a program of this nature on SunOS 4, where there is no support for trace through fork), you might nd it an advantage to port to a different machine running an operating system such as Solaris 2 in order to be able to test with truss. Of course, Murphys law says that the bug wont show up under Solaris 2.

Tracing network traffic

Another place where we can trace is at the network interface. Many processes communicate across the network, and if we have tools to look at this communication, they may help us isolate the part of the package that is causing the problem. Two programs trace message ow across a network: On BSD systems, tcpdump and the Berkeley Packet Filter provide a exible means of tracing trafc across Internet domain sockets. See Appendix E, Where to get sources, for availability. trpt will print a trace from a socket marked for debugging. This function is available on System V.4 as well, though it is not clear what use it is under these circumstances, since System V.4 emulates sockets in a library module. On BSD systems, it comes in a poor second to tcpdump.

Tracing net trafc is an unusual approach, and we wont consider it here, but in certain circumstances it is an invaluable tool. You can nd all you need to know about tcpdump in TCP/IP Illustrated, Volume 1, by Richard Stevens.

5 February 2005 02:09

9
Installation
Finally the package has been built and tested, and it works. Up to this point, everything in the package has been in the private source tree where it has been built. Most packages are not intended to be executed from the source directory: before we can use them, we need to move the component parts to their intended directories. In particular: We need to put executables where a normal PATH environment variable will nd them. We need to place on-line documentation in the correct directories in a form that the document browser understands. The installed software needs to be given the correct permissions to do what it has to do: all executables need to have their execute permissions set, and some programs may need setuid or setgid bits set (see Chapter 12, Kernel dependencies, page ). In addition, software will frequently be installed in directories to which normal users have no access. In these cases, the install must be done by root. Library routines and conguration les need to be installed where the package expects them: the location could be compiled into the programs, or an environment variable could point to the location. If the package uses environment variables, you may also need to update .prole and .cshrc les to add or modify environment variables. Many packages for example, news transfer programscreate data in specic directories. Although initially there may be no data to install, the install process may need to create the directories. At some future date, you may want to remove the package again, or to install an updated version. The installation routines should make some provision for removing the package when you no longer want it.

Real-life packages differ signicantly in their ability to perform these tasks. Some Makeles consider that their job is done when the package has been compiled, and leave it to you do install the les manually. In some cases, as when there is only a single program, this is no hardship, but it does require that you understand exactly what you need to install. On the other hand, very few packages supply an uninstall target.
125

5 February 2005 02:09

126

In this chapter, well look at the following subjects: The way Makeles typically install software. Alternatives if the Makele doesnt do everything it should do. How to install documentation. How to keep track of installed software. How to remove installed software.

Installation is an untidy area. At the end of this chapter, youll probably be left with a feeling of dissatisfaction this area has been sadly neglected, and there just arent enough good answers.

make install
The traditional way to install a pre-compiled package is with make install. Typically, it performs the following functions: It creates the necessary directories if they are not there. It copies all necessary les to their run-time locations. It sets the permissions of the les to their correct values. This frequently requires you to be root when you install the package. If you dont have root access, you should at least arrange for access to the directories into which you want to install. It may strip debug information from executables.

Some other aspects of make install are less unied: make install may imply a make all: you cant install until you have made the package, and youll frequently see an install target that starts with
install: all installation commands

On the other hand, make install may not only expect the make all to be completedand fail if it is notbut remove the executables after installation. Sometimes this is due to the use of BSD install without the -c option see the section on the install program belowbut it means that if you want to make a change to the program after installation, you effectively have to repeat the whole build. Removing les from the tree should be left to make clean (see Chapter 5, Building the package, page 63). Some install targets install man pages or other on-line documentation, others leave it to a separate target with a name like install-man, and yet other Makeles completely ignore online documentation, even if the package supplies it.

5 February 2005 02:09

Chapter 9: Installation

127

Configuring the installed package

Some packages have run-time conguration les that need to be set up before the package will run. Also, its not always enough just to install the les in the correct place and with the correct permissions: you may need to modify the individual users environment before they can use the package. Here are some examples: sendmail, the Internet mail transport agent, has an extremely complicated conguration le sendmail.cf which needs to be set up to reect your network topology. A description of how to set up this le takes up hundreds of pages in sendmail, by Bryan Costales, Eric Allman and Neil Rickert. Many X11 clients have supplementary les that dene application defaults, which may or may not be suitable for your environment. They are intended to be installed in a directory like /usr/X11/lib/X11/app-defaults. Not all Imakeles perform this task. The path where the executables are installed should be in your PATH environment variable. If you install man pages, the path should be in your MANPATH environment variable. Many packages dene their own environment variables. For example, TEX denes the environment variables TEXCONFIG, TEXFONTS, TEXFORMATS, TEXINPUTS, and TEXPOOL to locate its data les. Some programs require a setup le in the home directory of each user who uses the program. Others do not require it, but will read it if it is present. Some programs will create links with other names. For example, if you install pax, the portable archive exchange program, you have the opportunity of creating links called tar and cpio. This is really a conguration option, but the Makele for pax does not account for it.

Typical Makeles are content with moving the les to where they belong, and leave such details to the user. Well see an alternative on page 138.

Installing the correct files

At rst, installation seems straightforward enough: you copy the les to where they belong, and thats that. In practice, a number of subtle problems can occur. Theres no hard and fast solution to them, but if you run into trouble it helps to understand the problem.

To replace or not to replace?

Throughout the build process, we have used make to decide whether to rebuild a target or not: if the target exists, and is newer than any of its dependencies, it will not be rebuilt. Traditionally, installation is different: the les are installed anyway, even if newer les are already present in the destination directory.

5 February 2005 02:09

128

The reasons for this behaviour are shrouded in time, but may be related to the fact that both install (which we will discuss below) and cp traditionally modify the time stamps of the les, so that the following scenario could occur: 1. 2. 3. 4. Build version 1 of a package, and install it. Start building version 2, but dont complete it. Make a modication to version 1, and re-install it. Complete version 2, and install it. Some of the le in version 2 were compiled before version 1 was re-installed, and are thus older than the installed les. As a result, they will not be installed, and the installed software will be inconsistent.

Its obviously safer to replace everything. But is that enough? Well look at the opposite problem in the next section.

Updating
Frequently you will install several versions of software over a period of time, as the package evolves. Simply installing the new version on top of the old version will work cleanly only if you can be sure that you install a new version of every le that was present in the old version: otherwise some les from the old version will remain after installation. For example, version 1.07.6 of the GNU libc included a le include/sys/bitypes.h, which is no longer present in version 1.08.6. After installing version 1.08.6, include/sys/bitypes.h is still present from the earlier installation. The correct way to handle this problem is to uninstall the old package before installation. For reasons we will investigate on page 133, this seldom happens.

install
install is a program that is used in installing software. It performs the tasks of creating any necessary directories, copying les, stripping executables, and setting permissions. install originated in Berkeley, and older System V systems dont support it. Its a fairly trivial program, of course, and many packages supply a script with a name like install.sh which performs substantially the same functions. The source is available in the 4.4BSD Lite distribution see Appendix E, Where to get sources. Although install is a relatively simple program, it seems that implementors have done their best to make it differ from one system to the next. The result is a whole lot of incompatible and just downright confusing options. System V.4 even supplies two different versions with conicting options, a BSD compatible one and a native one the one you get depends on your other preferences, as laid down in your PATH environment variable. System V.4 native install is sufciently different from the others that we need to look at it separately it can install only a single le. The syntax is:

5 February 2005 02:09

Chapter 9: Installation
install options file [dir dir ...]

129

If the dirs are specied, they are appended to the xed list of directories /bin, /usr/bin, /etc, /lib, and /usr/lib. install will search the resultant list of directories sequentially for a le with the name le. If it nds one, it will replace it and print a message stating in which directory it has installed the le. The -i option tells install to omit the standard directories and take only the list specied on the command line. Other versions of install have a syntax similar to mv and cp, except that they take a number of supplementary options:
install options file1 file2 install options file1 ... fileN dir

The rst form installs le1 as le2, the second form installs le1 through leN in the directory dir. Table 9-1 contains an overview of install options:
Table 91: install options

option
-c

Purpose In BSD, copy the le. If this option is not specied, the le is moved (the original le is deleted after copying). In GNU and System V.4 (BSD compatibility), this option is ignored. Files are always copied.

-c dir -d

System V.4 native: install the le in directory dir only if the le does not already exist. If the le exists already, exit with an error message. In GNU and SunOS, create all necessary directories if the target directory does not exist. Not available in BSD. This lets you create the directory with the command
install -d [-g group] [-m perm] [-o owner] dir

-f ags

In 4.4BSD, specify the targets le ags. This relates to the chags program introduced with 4.4BSDsee the man page usr.bin/chags/chags.1 in the 4.4BSD Lite distribution. System V.4 native: force the le to be installed in dir. This is the default for other versions. Set the group ownership to group. System V.4 native: ignore the default directory list (see below). This is not applicable with the -c or -f options. Set the le permissions to perm. perm may be in octal or symbolic form, as dened for chmod(1). By default, perm is 0755 (rwxr-xr-x).

-f dir -g group -i -m perm

5 February 2005 02:09

130 Table 91: install options (continued)

option
-n dir -o

Purpose System V.4 native: if le is not found in any of the directories, install it in dir. System V.4 native: if le is already present at the destination, rename the old version by prepending the letters OLD to the lename. The old le remains in the same directory. All except System V.4 native: change the owner to owner. System V.4 native: suppress error messages. All except System V.4 native: strip the nal binary. System V.4 native: change the owner to owner.

-o owner -s -s -u owner

Other points to note are: install attempts to prevent you from moving a le onto itself. Installing /dev/null creates an empty le. install exits with a return code of 0 if successful and 1 if unsuccessful.

System V.4 install is denitely the odd man out: if you can avoid it, do. Even Solaris 2 supplies only the BSD version of install. On the other hand, pure BSD install also has its problems, since it requires the -c option to avoid removing the original les.

Installing documentation
Installing man pages would seem to be a trivial exercise. In fact, a number of problems can occur. In this section, well look at problems you might encounter installing man pages and GNU info.

Man pages.
As we saw in Chapter 7, Documentation, page 99, there is not much agreement about naming, placing, or format of man pages. In order to install man pages correctly you need to know the following things: The name of the man directory. The naming convention for man les. As we saw, these are many and varied. Whether the man pages should be formatted or not. If the man pages should be formatted, which formatter should be used? Which macros should be used? This may seem like a decision to be made when building the package, but many Makeles put off this operation to the install phase. Whether the man pages should be packed, compressed or zipped.

5 February 2005 02:09

Chapter 9: Installation

131

Typically, this information is supplied in the Makele like this example from the electronic mail reader elm, which is one of the better ones:
FORMATTER MAN MANEXT CATMAN CATMANEXT TBL MANROFF SUFFIX PACKED PACKER = = = = = = = = = = /usr/ucb/nroff /opt/man/man1 .1 /opt/man/cat1 .1 /usr/ucb/tbl /usr/ucb/nroff .Z y /bin/compress

# List of installed man pages (except for wnemail.1 - handled differently) MAN_LIST = $(MAN)/answer$(MANEXT) \ $(MAN)/autoreply$(MANEXT) \ ...etc # List of installed catman pages (except for wnemail.1 - handled differently) CATMAN_LIST = $(CATMAN)/answer$(CATMANEXT)$(SUFFIX) \ $(CATMAN)/autoreply$(CATMANEXT)$(SUFFIX) \ ...etc # List of formatted pages for catman FORMATTED_PAGES_LIST = catman/answer$(CATMANEXT)$(SUFFIX) catman/autoreply$(CATMANEXT)$(SUFFIX) ...etc

\ \

# Targets all: @if $(TEST) $(CATMAN) != none; then $(MAKE) formatted_pages ; \ else true ; fi formatted_pages: catman $(FORMATTED_PAGES_LIST) catman: mkdir catman install: $(LIB_LIST) @if $(TEST) $(MAN) != none; then $(MAKE) install_man ; \ else true ; fi @if $(TEST) $(CATMAN) != none; then $(MAKE) install_catman ; \ else true ; fi install_man: $(MAN_LIST) $(MAN)/wnewmail$(MANEXT) install_catman: $(CATMAN_LIST) $(CATMAN)/wnewmail$(CATMANEXT)$(SUFFIX) # Dependencies and rules for installing man pages and lib files $(MAN)/answer$(MANEXT): answer.1 $(CP) $? $@ $(CHMOD) u=rw,go=r $@

5 February 2005 02:09

132
$(MAN)/autoreply$(MANEXT): autoreply.1 $(CP) $? $@ $(CHMOD) u=rw,go=r $@

This Makele is in the subdirectory doc, which is concerned only with documentation, so all the targets relate to the man pages. The target all makes the decision whether to format the pages or not based on the value of the make variable CATMAN. If this is set to the special value none, the Makele does not format the pages. The target install uses the same technique to decide which man pages to install: if the variable MAN is not set to none, the sources of the man pages are copied there, and if CATMAN is not set to none, the formatted pages are installed there. This Makele does not use install: it performs the operations with cp and chmod instead.

GNU info
Installing GNU info is somewhat more straightforward, but it is also not as clean as it could be: info is always formatted, so you need the formatter, a program called makeinfo, which is part of the texinfo package. Before you can run makeinfo, you need to port texinfo. Its not that big a job, but it needs to be done. Of course, in order to completely install texinfo, you need to format the documentation with makeinfoa vicious circle. The solution is to port the texinfo executables, then port makeinfo, and then format the texinfo documentation. All info les are stored in a single directory with an index le called dir. This looks like:
-*- Text -*This is the file /opt/info/dir, which contains the topmost node of the Info hierarchy. The first time you invoke Info you start off looking at that node, which is (dir)Top. File: dir Node: Top This is the top of the INFO tree This (the Directory node) gives a menu of major topics. Typing "d" returns here, "q" exits, "?" lists all INFO commands, "h" gives a primer for first-timers, "mTexinfo<Return>" visits Texinfo topic, etc. Note that the presence of a name in this list does not necessarily mean that the documentation is available. It is installed with the package in question. If you get error messages when trying to access documentation, make sure that the package has been installed. --- PLEASE ADD DOCUMENTATION TO THIS TREE. (See INFO topic first.) --* Menu: The list of major topics begins on the next line. * * * * Bash: (bash). Bfd: (bfd). Bison: (bison). CL: (cl). The GNU Bourne Again SHell. The Binary File Descriptor Library. The Bison parser generator. Partial Common Lisp support for Emacs Lisp.

5 February 2005 02:09

Chapter 9: Installation
...etc

133

The lines at the bottom of the example are menu entries for each package. They have a syntax which isnt immediately apparentin particular, the sequence * item: has a special signicance in emacs info mode. Programs that supply info documentation should supply such an entry, but many of them do not, and none of them install the line in diryou need to do this by hand.

Removing installed software

For a number of reasons, you may want to remove software that you have already installed: You may decide you dont need the software. You may want to replace it with a newer version, and you want to be sure that the old version is gone. You may want to install it in a different tree.

If you look for a remove or uninstall target in the Makele, chances are that you wont nd one. Packages that supply a remove target are very rare. If you want to remove software, and you didnt take any precautions when you installed it, you have to do it manually with the computer equivalent of an axe and a spear: ls and rm.

Removing software manually

In fact, its frequently not that difcult to remove software manually. The modication timestamps of all components are usually within a minute or two of each other, so ls with the -lt options will list them all together. For example, lets consider the removal of ghostscript. The rst step is to go back to the Makele and see what it installed:
prefix = /opt exec_prefix = $(prefix) bindir = $(exec_prefix)/bin datadir = $(prefix)/lib gsdatadir = $(datadir)/ghostscript mandir = $(prefix)/man/man1 ...skipping install: $(GS) -mkdir $(bindir) for f in $(GS) gsbj gsdj gslj gslp gsnd bdftops font2c \ ps2ascii ps2epsi; \ do $(INSTALL_PROGRAM) $$f $(bindir)/$$f ; done -mkdir $(datadir) -mkdir $(gsdatadir) for f in README gslp.ps gs_init.ps gs_dps1.ps gs_fonts.ps gs_lev2.ps \ gs_statd.ps gs_type0.ps gs_dbt_e.ps gs_sym_e.ps quit.ps Fontmap \ uglyr.gsf bdftops.ps decrypt.ps font2c.ps impath.ps landscap.ps \ level1.ps prfont.ps ps2ascii.ps ps2epsi.ps ps2image.ps pstoppm.ps\

5 February 2005 02:09

134
showpage.ps type1ops.ps wrfont.ps ; \ do $(INSTALL_DATA) $$f $(gsdatadir)/$$f ; done -mkdir $(docdir) for f in NEWS devices.doc drivers.doc fonts.doc hershey.doc \ history.doc humor.doc language.doc lib.doc make.doc ps2epsi.doc \ psfiles.doc readme.doc use.doc xfonts.doc ; \ do $(INSTALL_DATA) $$f $(docdir)/$$f ; done -mkdir $(mandir) for f in ansi2knr.1 gs.1 ; do $(INSTALL_DATA) $$f $(mandir)/$$f ; done -mkdir $(exdir) for f in chess.ps cheq.ps colorcir.ps golfer.ps escher.ps \ snowflak.ps tiger.ps ; \ do $(INSTALL_DATA) $$f $(exdir)/$$f ; done

One alternative is to make a remove target for this Makele, which isnt too difcult in this case: First, copy the install target and call it remove. Move the mkdir lines to the bottom and change them to rmdir. Youll notice that this Makele accepts the fact that mkdir can fail because the directory already exists (the - in front of mkdir). Well do the same with rmdir: if the directory isnt empty, rmdir fails, but thats OK. We replace $(INSTALL_PROGRAM) $$f and $(INSTALL_DATA) $$f with rm -f.

The result looks like:

remove: $(GS) for f in $(GS) gsbj gsdj gslj gslp gsnd bdftops font2c \ ps2ascii ps2epsi; \ do rm -f $(bindir)/$$f ; done for f in README gslp.ps gs_init.ps gs_dps1.ps gs_fonts.ps gs_lev2.ps \ gs_statd.ps gs_type0.ps gs_dbt_e.ps gs_sym_e.ps quit.ps Fontmap \ uglyr.gsf bdftops.ps decrypt.ps font2c.ps impath.ps landscap.ps \ level1.ps prfont.ps ps2ascii.ps ps2epsi.ps ps2image.ps pstoppm.ps\ showpage.ps type1ops.ps wrfont.ps ; \ do rm -f $(gsdatadir)/$$f ; done for f in NEWS devices.doc drivers.doc fonts.doc hershey.doc \ history.doc humor.doc language.doc lib.doc make.doc ps2epsi.doc \ psfiles.doc readme.doc use.doc xfonts.doc ; \ do rm -f $(docdir)/$$f ; done for f in ansi2knr.1 gs.1 ; do $(INSTALL_DATA) $$f $(mandir)/$$f ; done for f in chess.ps cheq.ps colorcir.ps golfer.ps escher.ps \ snowflak.ps tiger.ps ; do rm -f $(exdir)/$$f ; done -rmdir $(bindir) -rmdir $(datadir) -rmdir $(gsdatadir) -rmdir $(docdir) -rmdir $(mandir)

5 February 2005 02:09

Chapter 9: Installation
-rmdir $(exdir)

135

More frequently, however, you cant use this approach: the Makele isnt as easy to nd, or you have long since discarded the source tree. In this case, well have to do it differently. First, we nd the directory where the executable gs, the main ghostscript program, is stored:
$ which gs /opt/bin/gs

Then we look at the last modication timestamp of /opt/bin/gs:

$ ls -l /opt/bin/gs -rwxrwxr-x 1 root wheel 3168884 Jun 18 14:29 /opt/bin/gs

This is to help us to know where to look in the next step: we list the directory /opt/bin sorted by modication timestamp. Its a lot easier to nd what were looking for if we know the date. If you dont have which, or possibly even if you do, you can use the following script, called wh:
for j in $*; do for i in echo $PATH|sed s/:/ /g; do if [ -f $i/$j ]; then ls -l $i/$j fi done done

wh searches the directories in the current environment variable PATH for a specic le and lists all occurrences in the order in which they appear in PATH in ls -l format, so you could also have entered:
$ wh gs -rwxrwxr-x 1 root wheel 3168884 Jun 18 14:29 /opt/bin/gs

Once we know the date we are looking for, its easy to list the directory, page it through more and nd the time frame we are looking for.
$ ls -lt /opt/bin|more total 51068 -rw------1 root bin -rwxr-xr-x 1 grog lemis ...skipping lots of stuff -rw-rw-rw1 grog bin -rw-rw-rw1 grog bin -rw-rw-rw1 grog bin -rwxrwxrwx 1 grog wheel -rwxrwxr-x 1 root wheel -rwxrwxr-x 1 root wheel -rwxrwxr-x 1 root wheel -rwxrwxr-x 1 root wheel -rwxrwxr-x 1 root wheel -rwxrwxr-x 1 root wheel -rwxrwxr-x 1 root wheel -rwxrwxr-x 1 root wheel

294912 Sep 6 15:08 trn.old 106496 Sep 6 15:08 man 370 370 196 469 52 807 35 563 50 3168884 53 51 Jun Jun Jun Jun Jun Jun Jun Jun Jun Jun Jun Jun 21 21 21 18 18 18 18 18 18 18 18 18 17:24 17:22 17:22 15:19 14:29 14:29 14:29 14:29 14:29 14:29 14:29 14:29 prab parb parb tep font2c ps2epsi bdftops ps2ascii gslp gs gsdj gsbj

5 February 2005 02:09

136
-rwxrwxr-x 1 root -rwxrwxr-x 1 root -rwxr-xr-x 1 root -r-xr-xr-x 1 bin -r-xr-xr-x 1 bin ...more stuff follows wheel wheel bin bin bin 18 54 81165 249856 106496 Jun Jun Jun Jun Jun 18 18 18 17 17 14:29 14:29 12:41 17:18 15:50 gsnd gslj faxaddmodem faxinfo dialtest

Its easy to recognize the programs in this format: they were all installed in the same minute, and the next older le (faxaddmodem) is more than 90 minutes older, the next newer le (tep) is 50 minutes newer. The les we want to remove are, in sequence, font2c, ps2epsi, bdftops, ps2ascii, gslp, gs, gsdj, gsbj, gsnd and gslj. Were not done yet, of course: ghostscript also installs a lot of fonts and PostScript les, as we saw in the Makele. How do we nd and remove them? It helps, of course, to have the Makele, from which we can see that the les are installed in the directories /opt/bin, /opt/lib/ghostscript and /opt/man/man1 (see the Makele excerpt on page 133). If you dont have the Makele, all is not lost, but things get a little more complicated. You can search the complete directory tree for les modied between Jun 18 14:00 and Jun 18 14:59 with:
$ find /opt -rwxrwxr-x ...etc -rw-rw-r--rw-rw-r--rw-rw-r--rw-rw-r--rw-rw-r--rw-rw-r--rw-rw-r-...many more -follow -type f -print|xargs ls -l|grep "Jun 18 14:" 1 root wheel 35 Jun 18 14:29 /opt/bin/bdftops 1 root 1 root 1 root 1 root 1 root 1 root 1 root files wheel wheel wheel wheel wheel wheel wheel 910 10005 11272 22789 295 74791 13974 Jun Jun Jun Jun Jun Jun Jun 18 18 18 18 18 18 18 14:29 14:29 14:29 14:29 14:29 14:29 14:29 /opt/man/man1/ansi2knr.1 /opt/man/man1/gs.1 /opt/lib/ghostscript/Fontmap /opt/lib/ghostscript/bdftops.ps /opt/lib/ghostscript/decrypt.ps /opt/lib/ghostscript/doc/NEWS /opt/lib/ghostscript/doc/devices.doc

There are a couple of points to note here: We used GNU nd, which uses the -follow option to follow symbolic links. If your /opt hierarchy contains symbolic links, nd would otherwise not search the subdirectories. Other versions of nd may require different options. You cant use ls -lR here because ls -lR does not show the full pathnames: you would nd the les, but the name at the end of the line would just be the name of the le, and you wouldnt know the name of the directory. If the le is more than six months old, ls -l will list it in the form
-rwxrwxrwx 1 grog wheel 22 Feb 10 1994 xyzzy

This may be enough to differentiate between the les, but its less certain. GNU ls (in the leutils package) includes a option -full-time (note the two leading hyphens). This will always print the full time, regardless of the age of the le. With this option, the le above will list as:
$ ls --full-time -l xyzzy -rwxrwxrwx 1 grog wheel 22 Thu Feb 10 16:00:24 1994 xyzzy

5 February 2005 02:09

Chapter 9: Installation

137

Removing too much

None of these methods for removing installed software can handle one remaining serious problem: some programs install a modied version of a standard program, and if you remove the package, you remove all trace of this standard program. For example, GNU tar and GNU cpio both include the remote tape protocol program rmt. If you install both of these packages, and then decide to remove cpio, tar will not work properly either. Its not always enough to keep track of which packages depend on which programs: in some cases, a modied version of a program is installed by a package, and if you remove the package, you need to re-install the old version of the program.

Keeping track of installed software

All the methods weve seen so far smell strongly of kludge: They involve signicant manual intervention, which is prone to error. The remove or uninstall targets of a Makele are based on names not contents. If you stop using a package, and install a new one with some names that overlap the names of the old package, and then remove the old package, the les from your new package will go too. The manual method based on the dates does not discover conguration or data lesif you remove net news from a system, you will have to remember to remove the news spool area as well, because that certainly wont have the same modication timestamp as the installed software. Its almost impossible to safely and automatically remove modications to environment variables in .cshrc and .prole les.

We can come closer to our goal if we have a method to keep track of the les that were actually installed. This requires the maintenance of some kind of database with information about the relationship between packages and les. Ideally, It would contain a list of the les installed, including their sizes and modication timestamps. It would prevent modication to the package except by well-dened procedures. It would contain a list of the les that were modied, including diffs to be able to reverse them. It would keep track of the modications to the package as time went by: which les were created by the package, which les were modied.

This is an ideal, but the System V.4 pkgadd system comes reasonably close, and the concept is simple enough that we can represent the most important features as shell scripts. Well look at it in the next section.

5 February 2005 02:09

138

System V pkgadd
UNIX System V.4 is supplied as a number of binary packages* you can choose which to install and which not to install. You can even choose whether or not to install such seemingly essential components as networking support and man pages. Packages can be created in two formats: stream format for installation from serial data media like tapes, and le system format for installation from le systems. In many cases, such as diskettes, either form may be used. The program pkgtrans transforms one format into the other. In the following discussion, well assume le system format. The package tools offer a bewildering number of options, most of which are not very useful. Well limit our discussion to standard cases: in particular, we wont discuss classes and multipart packages. If you are using System V.4 and want to use other features, you should read the documentation supplied with the system. In the following sections well look at the individual components of the packages.

pkginfo
The le pkginfo, in the root directory of the package, contains general information about the package, some of which may be used to decide whether or not to install the package. For example, the pkginfo le for an installable emacs package might look like:
ARCH=i386 PKG=emacs VERSION=19.22 NAME=Emacs text editor CATEGORY=utilities CLASSES=none VENDOR=Free Software Foundation HOTLINE=LEMIS, +49-6637-919123, EMAIL=lemis@lemis.de the architecture for which the package is intended the name of the package the version number a brief description the kind of package class information the name of the owner Fax +49-6637-919122 who to call if you have trouble mail for HOTLINE

This information is displayed by pkgadd as information to the user before installation.

pkgmap
The le pkgmap is also in the root directory of the package. It contains information about the destination of the individual les. For example, from the same emacs package,
: 1 1 1 1 1 1 1 d d f d f f 37986 none /opt 0755 bin bin none /opt/README 0755 bin bin none /opt/README/emacs-19.22 0644 root sys 1518 59165 760094611 none /opt/bin 0755 bin bin none /opt/bin/emacs 0755 root sys 1452488 11331 760577316 none /opt/bin/etags 0755 root sys 37200 20417 760577318

* As used here, the term package is a collection of precompiled programs and data and information necessary to install themthis isnt the same thing as the kind of package we have been talking about in the rest of this book.

5 February 2005 02:09

Chapter 9: Installation
1 1 1 1 1 1 1 1 1 d f f f d d d d f none none none none none none none none none /opt/info 0755 bin bin /opt/info/cl.info 0644 root sys 3019 62141 760094526 /opt/info/dir 0644 root sys 2847 23009 760559075 /opt/info/emacs 0644 root sys 10616 65512 760094528 /opt/lib 0755 bin bin /opt/lib/emacs 0755 bin bin /opt/lib/emacs/19.22 0755 bin bin /opt/lib/emacs/19.22/etc 0755 bin bin /opt/lib/emacs/19.22/etc/3B-MAXMEM 0644 root sys 1913 18744 574746032

139

The rst line species that the package consists of a single part, and that it consists of 37986 512 byte blocks. The other lines describe les or directories: The rst parameter is the part to which the le belongs. The next parameter species whether the le is a plain le (f), a directory (d), a link (l) or a symbolic link (s). A number of other abbreviations are also used. The next parameter is the class of the le. Like most packages, this package does not use classes, so the class is always set to none. The following four parameters specify the name of the installed object, its permissions, the owner and the group. After this come the size of the le, a checksum and the modication time in naked time_t format. The checksum ensures that the package is relatively protected against data corruption or deliberate modication.

Package subdirectories
In addition to the les in the main directory, packages contain two subdirectories root and install: root contains the les that are to be installed. All the les described in pkgmap are present under the same names in root (for example, /opt/bin/emacs is called root/opt/bin/emacs in the package). The le install/copyright contains a brief copyright notice that is displayed on installation. pkgadd does not wait for you to read this, so it should really be brief. Optionally, there may be scripts with names like install/preinstall and install/postinstall which are executed before and after copying the les, respectively. preinstall might, for example, set up the directory structure /opt if it does not already exist. postinstall might update .cshrc and .prole les. In some cases, it may need to do more. For example, the ISO 9660 directory standard for CD-ROMs limits allows only eight nested directories (in other words, the directory /a/b/c/d/e/f/g/h/i is nested too deeply). gcc on a CD-ROM would violate this limitation, so some of the package has to be stored as a tar le, and the postinstall script extracts it to the correct position.

5 February 2005 02:09

140

pkgadd
With this structure, adding a package is almost childs play: you just have to enter
$

pkgadd emacs

Well, almost. The name emacs is the name of the package and not a le name. By default, pkgadd expects to nd it in /var/spool/pkg. If your package is elsewhere, you cant tell pkgadd simply by prepending the nameinstead, you need to specify it with the -d option:
$ pkgadd -d /cdrom emacs

This will install emacs from the directory cdrom.

Removing packages
One really nice thing about the System V.4 package system is the ease with which you can remove a package. Assuming that you have decided that vi is a better choice than emacs, or you just dont have the 19 MB that the emacs package takes up, you just have to type:
$ pkgrm emacs

and all the les will be removed.

Making installable packages

The discussion of pkgadd assumes that you already have an installable package. This is appropriate for System V.4, but if you have just ported a software package, you rst need to create an installable binary package from it. This is the purpose of pkgmk. It takes a number of input les, the most important of which is prototype: it describes which les should be installed. It is almost identical in format to the pkgmap le we discussed above. For example, the prototype le for the emacs example above looks like:
# i i i i # d d d d d d d d d # f f Prototype file created by /cdcopy/ETC/tools/mkmkpk on Wed Jan 19 18:24:41 WET 1994 pkginfo preinstall postinstall copyright Required directories none /opt 755 bin bin none /opt/bin 755 bin bin none /opt/README 755 bin bin none /opt/man 755 bin bin none /opt/lib 755 bin bin none /opt/lib/emacs 755 bin bin none /opt/lib/emacs/19.22 755 bin bin none /opt/lib/emacs/19.22/etc 755 bin bin none /opt/info 755 bin bin Required files none /opt/lib/emacs/19.22/etc/3B-MAXMEM 644 root sys none /opt/bin/emacs 755 root sys

5 February 2005 02:09

Chapter 9: Installation
f none /opt/info/emacs 644 root sys f none /opt/info/dir 644 root sys

141

This looks rather different from pkgmap: There are comment lines starting with #. The rst line indicates that this le was created by a script. Later on well see the kind of function mkmkpk might perform. The rst column (part number) and the last three columns (size, checksum and modication timestamp) are missing. Some lines start with the keyletter i. These describe installation les: we recognize the names from the discussion above. pkgmk copies these les into the directory tree as discussed above. What is not so immediately obvious is that pkginfo is placed in the main directory of the package, and the others are placed in the subdirectory install. It is also not obvious that some of these les are required: if they are not specied, pkgmk dies.

Making a prototype file

Theres still a gap between the original make install and building an installable package. We need a prototype le, but make install just installs software. The packaging tools include a program called pkgproto that purports to build prototype les. It searches a directory recursively and creates prototype entries for every le it nds. If you have just installed emacs, say, in your /opt directory, pkgproto will give you a prototype including every le in /opt, including all the packages which are already installed therenot what you want. There are a number of alternatives to solve this problem: You can install into a different directory. pkgproto supports this idea: you can invoke it with
$ pkgproto /tmp-opt=/opt

which will tell it to search the directory /tmp-opt and generate entries for /opt. The disadvantage of this approach is that you may end up building programs with the path /tmpopt hard coded into the executables, and though it may test just ne on your system, the executable les will not work on the target systemdenitely a situation to avoid. You rename /opt temporarily and install emacs in a new directory, which you can then rename. This virtually requires you to be the only user on the system. Before installing emacs, you create a dummy le stamp-emacs just about anywhere on the system. Then you install emacs, and make a list of the les you have just installed:
$ find /opt -follow -cnewer stamp-emacs -type f -print | xargs ls -l >info

This requires you to be the only person on the system who can write to the directory at the time. This is more not as simple as you might think. Mail and news can come in even if nobody else is using the system. Of course, they wont usually write in the same directories that youre looking in. Nevertheless, you should be prepared for a few surprises. For example, you might nd a le like this in your list:

5 February 2005 02:09

142
/opt/lib/emacs/lock/!cdcopy!SOURCE!Core!glibc-1.07!version.c

This is an emacs lock le: it is created by emacs when somebody modies a buffer (in this case, a le called /cdcopy/SOURCE/Core/glibc-1.07/version.c: emacs replaces the slashes in the le name by exclamation marks), and causes another emacs to warn the user before it, too, tries to modify the same le. It contains the pid of the emacs process that has the modied buffer. Obviously you dont want to include this le in your installable package. Once you have tidied up your list of les, you can generate a prototype le with the aid of a shell script or an editor.

Running pkgmk
Once you have a prototype le, youre nearly home. All you have to do is run pkgmk. We run into terminology problems here: throughout this book, we have been using the term package to refer to the software we are building. More properly, this is the software package. pkgmk refers to its output as a package toohere, well refer to it as the installable package. Unfortunately, pkgmk handles some pathnames strangely. You can read the man page (preferably several times), or use this method, which works: Before building the installable package, change to the root directory of the software package. Ignore path specications in the prototype le and specify the root path as the root le system: -r /. Specify the base directory as the root directory of the package: since thats the directory were in, just add -b pwd. Choose to overwrite any existing package: -o. Specify the destination path explicitly: -d /usr/pkg. pkgmk creates packages will as subdirectories in this directory: the package gcc would create a directory hierarchy /usr/pkg/gcc.

The resultant call doesnt change from one package to the next: it is
pkgmk -r / -b pwd -o -d /usr/pkg

There is a whole lot more to using pkgmk, of course, but if you have pkgmk, you will also have the man pages, and thats the best source of further information.

5 February 2005 02:09

Where to go from here

Finally its all over. The package is ported, youve installed the software, and it really does work. This time, were done! Well, we said that once before, before we started testing, and we were wrong. Were wrong here, too: In the course of the port, you may nd a bug or a misfeature and x it. If you do so, you have effectively created a new version of the package. You should send in information about these changes to the author. If this is a popular package, you might consider reporting the changes to the Usenet group that exists for the package. You no longer need the space on disk, so you can clean up the archive and write it to tape. Its a good idea to maintain enough documentation to be able to retrieve it again. Sometime, maybe very soon, somebody will come out with a x for a bug that will probably bite you some time, or with a feature that could really be of use to you. Your experience with this port will help you to port the new version.

None of this is much work now, and it will save you grief later on. Lets look at it in a little more detail.

Reporting modifications
Once you have the software running, you should report any changes to the author or maintainer of the software. In order for this to be of any use, you need to supply the following information: A description of the problems you ran into. Dont spare details here: remember the pain you went to to gure out what was going wrong, and you had an interest in solving the problem. If youre the rst person to run into the problem, it probably hasnt hurt anybody else, least of all the author. He probably gets lots of mail saying xfoo is broke, and he may not believe what you have to say until you prove it to him. How you xed them. Again, lots of detail. The author probably understands the package better than you do. If you explain the problem properly, he may come up with a better
143

5 February 2005 02:09

144

x. The xes themselves. diffs, lists of differences between the previous version and your versions, are the method of choice. Well look at them in the rest of this section.

diff
diff is a program that compares two related source les and outputs information about how to create the second le from the rst. You typically use it after making modications to a le in order to describe how the modied le differs from the original. The resultant output le is also called a diff. We saw the application of diffs in Chapter 3, Care and feeding of source trees, page 29. Here well look at how to make them. Its useful to recognize and understand diff formats, since you occasionally have to apply them manually. diff compares two source les and attempts to output a reasonably succinct list of the differences between them. In diff terminology, the output is grouped into hunks, information about a relatively local groups of differences. Like most useful programs, diff has grown in the course of time, and modern versions can output in a bewildering number of formats. Fortunately, almost all diffs nowadays use the context format. Well look at some others anyway so that you can recognize them. In the following examples, we compare the les eden.1:
A doctor, an architect, and a computer scientist were arguing about whose profession was the oldest. In the course of their arguments, they got all the way back to the Garden of Eden, whereupon the doctor said, "The medical profession is clearly the oldest, because Eve was made from Adams rib, as the story goes, and that was a simply incredible surgical feat." The architect did not agree. He said, "But if you look at the Garden itself, in the beginning there was chaos and void, and out of that, the Garden and the world were created. So God must have been an architect." The computer scientist, who had listened to all of this said, "Yes, but where do you think the chaos came from?"

and eden.2:
A doctor, an architect, and a computer scientist were arguing about whose profession was the oldest. In the course of their arguments, they came to discuss the Garden of Eden, whereupon the doctor said, "The medical profession is clearly the oldest, because Eve was made from Adams rib, as the story goes, and that was a simply incredible surgical feat." The architect did not agree. He said, "But if you look at the Garden itself, in the beginning there was chaos and void, and out of that, the Garden and the world were created. So God must have been an architect." The computer scientist, who had listened to all of

5 February 2005 02:09

Chapter 10: Where to go from here

this, said, "Yes, but where do you think the chaos came from?"

145

normal format diffs

As the name implies, the normal format is the default. You dont need to specify any format ags:
$ diff eden.1 eden.2 3,7c3,7 < course of their arguments, they got all the way back to the < Garden of Eden, whereupon the doctor said, "The medical < profession is clearly the oldest, because Eve was made from < Adams rib, as the story goes, and that was a simply < incredible surgical feat." --> course of their arguments, they came to discuss the Garden > of Eden, whereupon the doctor said, "The medical profession > is clearly the oldest, because Eve was made from Adams rib, > as the story goes, and that was a simply incredible surgical > feat." 13c13 < this said, "Yes, but where do you think the chaos came --> this, said, "Yes, but where do you think the chaos came

The rst line of each hunk species the line range: 3,7c3,7 means lines 3 to 7 of the rst le, lines 3 to 7 of the second le. 13c13 means line 13 of the rst le, line 13 of the second le, has changed (c). Instead of c you will also see d (lines deleted) and a (lines added). After this header line come the lines of the rst le, with a leading < character, then a divider (---) and the lines of the second le with a leading > character. This example has two hunks.

ed format diffs
ed format diffs have the dubious advantage that the program ed can process them. You can create them with the -e ag. In this example, we also use shell syntax to shorten the input line. Writing eden.[12] is completely equivalent to writing eden.1 eden.2.
$ diff -e eden.[12] 13c this, said, "Yes, but where do you think the chaos came . 3,7c course of their arguments, they came to discuss the Garden of Eden, whereupon the doctor said, "The medical profession is clearly the oldest, because Eve was made from Adams rib, as the story goes, and that was a simply incredible surgical feat." .

Just about everybody who has diff also has patch, and nowadays not everybody has ed. In addition, this format is extremely dangerous, since there is no information about the old

5 February 2005 02:09

146

content of the le: you cant be sure if the patch will be applied in the right place. As a result, you almost never see this form.

context diffs
You select a context diff with the ag -c:
$ diff -c eden.[12] *** eden.1 Tue May 10 14:21:47 1994 --- eden.2 Tue May 10 14:22:38 1994 *************** *** 1,14 **** A doctor, an architect, and a computer scientist were arguing about whose profession was the oldest. In the ! course of their arguments, they got all the way back to the ! Garden of Eden, whereupon the doctor said, "The medical ! profession is clearly the oldest, because Eve was made from ! Adams rib, as the story goes, and that was a simply ! incredible surgical feat." The architect did not agree. He said, "But if you look at the Garden itself, in the beginning there was chaos and void, and out of that, the Garden and the world were created. So God must have been an architect." The computer scientist, who had listened to all of ! this said, "Yes, but where do you think the chaos came from?" --- 1,14 ---A doctor, an architect, and a computer scientist were arguing about whose profession was the oldest. In the ! course of their arguments, they came to discuss the Garden ! of Eden, whereupon the doctor said, "The medical profession ! is clearly the oldest, because Eve was made from Adams rib, ! as the story goes, and that was a simply incredible surgical ! feat." The architect did not agree. He said, "But if you look at the Garden itself, in the beginning there was chaos and void, and out of that, the Garden and the world were created. So God must have been an architect." The computer scientist, who had listened to all of ! this, said, "Yes, but where do you think the chaos came

The output here gives us signicantly more information: the rst two line gives the name and modication timestamp of the les. Then the hunks start, with a row of * as a leader. The next line is line number information for the rst le (lines 1 to 14), after which come the lines themselves, surrounded by a number of lines of context, unchanged information. You can specify the number of lines of context, but by default diff includes 2 lines either side of the changes. The lines that have been modied are agged with an exclamation mark (!) at the beginning of the line. In this case, the le is so small that the two modications have been merged into one large one, and the whole le gets repeated, but in a larger le diff would include only the information immediately surrounding the changes. This format is more reliable than normal diffs: if the original source le has changed since the diff, the context

5 February 2005 02:09

Chapter 10: Where to go from here

147

information helps establish the correct location to apply the patch.

unified context diffs

unied diffs are similar to normal context diffs. They are created with the -u ag:
$ diff -u eden.[12] --- eden.1 Tue May 10 14:21:47 1994 +++ eden.2 Tue May 10 14:22:38 1994 @@ -1,14 +1,14 @@ A doctor, an architect, and a computer scientist were arguing about whose profession was the oldest. In the -course of their arguments, they got all the way back to the -Garden of Eden, whereupon the doctor said, "The medical -profession is clearly the oldest, because Eve was made from -Adams rib, as the story goes, and that was a simply -incredible surgical feat." +course of their arguments, they came to discuss the Garden +of Eden, whereupon the doctor said, "The medical profession +is clearly the oldest, because Eve was made from Adams rib, +as the story goes, and that was a simply incredible surgical +feat." The architect did not agree. He said, "But if you look at the Garden itself, in the beginning there was chaos and void, and out of that, the Garden and the world were created. So God must have been an architect." The computer scientist, who had listened to all of -this said, "Yes, but where do you think the chaos came +this, said, "Yes, but where do you think the chaos came from?"

As with context diffs, there is a header with information about the two les, followed by a hunk header specifying the line number range in each of the two les. Unlike a normal context diff, the following hunk contains the old text mingled with the new text. The lines prexed with the character - belong to the rst le, those prexed with + belong to the second le in other words, to convert the old le to the new le you remove the lines prexed with - and insert the lines prexed with +. There are still other formats offered by various avours of diff, but these are the only important ones.

What kind of diff?

As weve seen, ed style diffs are out of the question. You still have the choice between regular diffs, context diffs and unied context diffs. Its not that important which kind of diff you choose, but context diffs are easier to apply manually. Unied context diffs take up less space than regular context diffs, but there are still versions of patch out there that dont understand unied diffs. Until that changes, its probably best to settle for regular context diffs. You may have noticed that all the examples in Chapter 3, Care and feeding of source trees, were regular context diffs.

5 February 2005 02:09

148

Living with diff

Diff is a straightforward enough program, but you might run into a couple of problems: After a large port, it makes sense to make diffs of the whole directory hierarchy. This requires that you have copies of all the original les. You can use rcsdiff, part of the RCS package, but it only does diffs one at a time. I nd it easier to maintain a copy of the complete original source tree, and then run diff with the option -r (descend recursively into directories):
$ diff -ru /S/SCO/Base/gcc-2.6.3 /S/Base/Core/gcc-2.6.3 >SCO.diffs

This command will create a single le with all the diffs and a list of les which only exist in the rst directory. This can be important if you have added les, but it also means that you should do a make clean before running diff, or you will have entries of this kind for all the object les you create. Another problem that may occur is that one of the les does not have a newline character at the end of the last line. This does not normally worry compilers, but diff sees t to complain. This is particularly insidious, because patch doesnt like the message, and it causes patch to fail.

Saving the archive

Most of us have had the message Dont forget to make backups drummed into us since we were in elementary school, but nowhere does it make more sense than at the end of a port. Dont forget where you put it! After archiving your port of xfoo, you may not look at it again for three years. When the new version comes out, you try to port it, but all sorts of things go wrong. Now is the time to get out the old version and read your notesbut where is it? Its beyond the scope of this book to go into backup strategies, but you should do some thinking about the subject. One good idea is to keep separate (DAT or Exabyte) tapes of old ports, and just add additional archives at the end. That way you dont have to worry about overwriting them accidentally: the tapes are small and cheap enough that you can afford to keep their contents almost indenitely. If you dont choose this method (maybe because the media dont t into your QIC-150 tape drive), you need to think carefully about how to track the archives and when they are no longer needed.

Not done after all?

Of course, it may be that this optimistic nish is completely out of place. After what seems like months of frustration, you nally decide that you are never going to get this &%%$@# to work, and you give up. You can never rule out this possibilityas I said in Chapter 1, Introduction, I hope this book made it easier, but its not a magic scroll. Even if you do give up, you have some tidying up to do: you obviously cant send the author your bug xes, but you can at least report the bugs. What he does with them depends on his interest and his contractual obligations, but even with free software, which is free of obligations of this nature, the author may be interested enough to x the problem. One way or

5 February 2005 02:09

Chapter 10: Where to go from here

149

another, you should go to the trouble to report problems you experience, even if you cant x them and there is no support obligation. A nal word: if you give up on a port after getting this far, this book has failed for you. I dont want that to happen. Please contact me, too (grog@lemis.de, or via OReilly and Associates) and explain the problem. Like the authors of the software, I dont guarantee to do anything about it, but I might, and your experience might help to make the next edition of this book more useful.

5 February 2005 02:09

Platform dependencies
In the rst part of this book, we looked at the various activities needed to port and install software on a UNIX system. We carefully avoided getting too involved with the nitty-gritty of why we should need to go to so much trouble. In this part of the book, well look at those differences between platforms which require us to modify software. As we saw in Chapter 4, Package conguration, conguration can be required for local preferences, software dependencies and hardware dependencies. We looked at local preferences in Chapter 4. In this part of the book, well look at differences in hardware and software platforms.

Software Dependencies
Probably the biggest problem you will have with conguration will be with the underlying software platform. Even if you limit your scope to the various UNIX versions, 25 years of continuing (and mainly uncoordinated) evolution have left behind a plethora of marginally compatible versions. The only good thing about the situation is that porting between UNIX versions is still an order of magnitude easier than porting to or from a non-UNIX environment. Its easy to misjudge the effort required to port to a different platform. It helps to make the following very clear distinctions between the following kinds of functionality: Functionality that relies on system calls (section 2 of the UNIX manual). These calls interface directly with the kernel. If the kernel doesnt supply the functionality, you may have serious difculty in porting the product. Good examples are the System V function shmget, which allocates an area of shared memory, or the BSD system call symlink, which creates a symbolic link. Functionality dependent on system library calls (section 3 of the UNIX manual). If these do not rely on system calls, you may be able to port a corresponding call from another library. A good example of this is the function strcasecmp, which compares strings ignoring case. This function is supplied with later versions of the BSD library and also with GNU libc, but not with System V libraries. If you dont have it, its trivial to port.

151

5 February 2005 02:09

152

Functionality contained totally inside the package, like math routines that dont call external libraries. This should work on any platform.

Some systems, such as OSF, have merged sections 2 and 3 of the manual pages. While that has some advantages (if you know a function name, you dont have to go to two different places to look for them), it doesnt mean that there is no longer a difference. Kernel dependencies are signicantly more difcult to handle than library dependencies, since theres relatively little you can do about them. Well look at kernel-related problems in Chapter 12, Kernel dependencies, Chapter 13, Signals, Chapter 14, File systems, Chapter 15, Terminal drivers, and Chapter 16, Timekeeping. In Chapter 17 well look at header les, and in Chapter 18 well look at libraries. In addition to these program dependencies, two tools can differ signicantly: the make program and the C compiler. Well look at these aspects in Chapter 19, Make, and Chapter 20, Compilers. Finally, in Chapter 21, Object les and friends, well look at some of the more esoteric aspects of object les. When discussing differences between kernels and libraries, the big difference is usually between System V and BSD, with other systems such as SunOS taking a mideld position. System V.4 incorporates nearly everything in BSD. When programming, you have the choice between using the native System V development tools or the BSD tools. Some admixture is possible, but it can cause problems. When using BSD development tools, everything that is supported by BSD should also be supported by System V.4. On the other hand, System V.4 also includes some functionality that no other system provides. When, in the following chapters, I say that a function is supported by System V.4, I mean that it is supported by System V.4 using the standard development tools and libraries. If I state that it is supported by BSD, it also implies that it is supported by System V.4 using the BSD libraries.

5 February 2005 02:09

Hardware dependencies
The days are gone when moving a package from one hardware platform to another meant rewriting the package, but there are still a number of points that could cause you problems. In this chapter, well look at the most common causes.

Data types
All computers have at least two basic data types, characters and integers. While European languages can get by with a character width of 8 bits, integers must be at least 16 bits wide to be of any use, and most UNIX systems use 32 bit integers, as much storage as four characters. Problems can obviously arise if you port a package to a system whose int size is less than the author of the package expected.

Integer sizes
Data sizes arent the problem they used to betimes were when a machine word could be 8, 12, 16, 18, 24, 30, 32, 36, 48, 60, 64 or 72 bits long, and so were the primary integer data objects. Nowadays you can expect nearly every machine to have an int of 16, 32 or 64 bits, and the vast majority of these have a 32 bit int. Still, one of the biggest problems in ANSI C is the lack of an exact denition of data sizes. int is the most used simple data type, but depending on implementation it can vary between 16 and 64 bits long. short and long can be the same size as int, or they can be shorter or longer, respectively. There are advantages to this approach: the C compiler will normally choose an int which results in the fastest processing time for the processor on which the program will run. This is not always the smallest data size: most 32-bit machines handle 32 bit arithmetic operations faster than 16 bit operations. Problems dont arise until the choice of int is too small to hold the data that the program tries to store in it. If this situation arises, you have a number of options: You can go through the sources with an editor and replace all occurrences of the word int with long (and possibly short with int).*

* If you do this, be sure to check that you dont replace short int with int int! 153

5 February 2005 02:09

154

You can simplify this matter a little by inserting the following denition in a common header le:
#define int long

This has the disadvantage that you cant dene short as int, because preprocessor macros are recursive, and you will end up with both int and short dened as long. Some compilers, particularly those with 16-bit native ints, offer compiler ags to generate longer standard ints.

All these solutions have the problem that they do not affect library functions. If your system library expects 16-bit integers, and you write
int x = 123456; printf ("x is %d\n", x);

the library routine printf still assumes that the parameter x is 16 bits long, and prints out the value as a signed 16-bit value (-7616), not what you want. To get it to work, you need to either specify an alternate library, or change the format specication to printf:
int x = 123456; printf ("x is %l\n", x);

There are a few other things to note about the size of an int: Portable software doesnt usually rely on the size of an int. The software from the Free Software Foundation is an exception: one of the explicit design goals is a 32-bit target machine. The only 64-bit machine that is currently of any signicance is the DEC Alpha. You dont need to expect too many problems there. 16 bit machinesincluding the 8086 architecture, which is still in use under MSDOS are a different matter, and you may experience signicant pain porting, say, a GNU program to MS-DOS. If you really want to do this, you should look at the way gcc has been adapted to MS-DOS: it continues to run in 32-bit protected mode and has a library wrapper* to allow it to run under MS-DOS.

Floating point types

Floating point data types have the same problems that integer types do: they can be of different lengths, and they can be big-endian or little-endian. I dont know of any system where ints are big-endian and oats are little-endian, or vice-versa. Apart from these problems, oats have a number of different structures, which are as good as completely incompatible. Fortunately, you dont normally need to look under the covers: as long as a oat handles roughly the same range of values as the system for which the program was written, you shouldnt have any problems. If you do need to look more carefully, for example if the programmer was making assumptions, say, about the position of the sign bit of
* A library wrapper is a library that insulates the program (in this case, a UNIX-like application) from the harsh realities of the outside world (in this case, MS-DOS).

5 February 2005 02:09

Chapter 11: Hardware dependencies

155

the mantissa, then you should prepare for some serious re-writing.

Pointer size
For years, people assumed that pointers and ints were the same size. The lax syntax of early C compilers didnt even raise an eyebrow when people assigned ints to pointers or vice-versa. Nowadays, a number of machines have pointers that are not the same size as ints. If you are using such a machine, you should pay particular attention to compiler warnings that ints are assigned to pointers without a cast. For example, if you have 16-bit ints and 32-bit pointers, sloppy pointer arithmetic can result in the loss of the high-order bits of the address, with obvious consequences.

Address space
All modern UNIX variants offer virtual memory, though the exact terminology varies. If you read the documentation for System V.4, you will discover that it offers virtual memory, whereas System V.3 only offered demand paging. This is more marketspeak than technology: System V.2, System V.3, and System V.4 each have very different memory management, but we can dene virtual memory to mean any kind of addressing scheme where a process address space can be larger than real memory (the hardware memory installed in the system). With this denition, all versions of System V and all the other versions of UNIX you are likely to come across have virtual memory. Virtual memory makes you a lot less dependent on the actual size of physical memory. The software from the Free Software Foundation makes liberal use of the fact: programs from the GNU project make no attempt to economize on memory usage. Linking the gcc C++ compiler cc1plus with GNU ld uses about 23 MB of virtual address space on System V.3 on an Intel architecture. This works with just about any memory conguration, as long as Your processes are allowed as much address space as they need (if you run into trouble, you should recongure your kernel for at least 32 MB maximum process address space, more if the system allows it). You have enough swap space. You can wait for the virtual memory manager to do its thing.

From a conguration viewpoint, we have different worries: Is the address space large enough for the program to run? How long are pointers? A 16 bit pointer can address only 64 kilobytes, a 32 bit pointer can address 4 GB. How do we address memory? Machines with 16 bit pointers need some kind of additional hardware support to access more than 64 kilobytes. 32 bit pointers are adequate for a at addressing scheme, where the address contained in the pointer can address the entire virtual address space.

5 February 2005 02:09

156

Modern UNIX systems run on hardware with 32 bit pointers, even if some machines have ints with only 16 bits, so you dont need to worry much about these problems. Operating systems such MS-DOS, which runs on machines with 16 bit pointers, have signicant problems as a result, and porting 32 bit software to them can be an adventure. Well touch on these problems in Chapter 20, Compilers, page 346.

Character order
The biggest headache you are likely to encounter in the eld of hardware dependencies is the differing relationship between int and character strings from one architecture to the next. Nowadays, all machines have integers large enough to hold more than one character. In the old days, characters in memory werent directly addressable, and various tricks were employed to access individual characters. The concept of byte addressing, introduced with the IBM System/360, solved that problem, but introduced another: two different ways of looking at bytes within a word arose. One camp decided to number the bytes in a register or a machine word from left to right, the other from right to left. For hardware reasons, text was always stored from low byte address to high byte address. A couple of examples will make this more intelligible. As we saw above, text is always stored low byte to high byte, so in any architecture, the text UNIX would be stored as 0 U 1 N 2 I 3 X

Some architectures, such Sparc and Motorola 68000, number the bytes in a binary data word from left to right. This arrangement is called big-endian. On a big-endian machine, the bytes are numbered from left to right, so the number 0x12345678 would be stored like 0 12 1 34 2 56 3 78

Others, notably older Digital Equipment machines and all Intel machines, number the bytes the other way round: byte 0 in a binary data word is on the right, byte 3 is on the left. This arrangement is called little-endian.* The same example on a little-endian machine would look like: 3 12 2 34 1 56 0 78

This may look just the same as before, but the byte numbers are now numbered from right to left, so the text now reads:
* The names big-endian and little-endian are derived from Jonathan Swifts Gullivers Travels, where they were a satirical reference to the conicts between the Catholics and the Church of England in the 18th Century.

5 February 2005 02:09

Chapter 11: Hardware dependencies

157

3 X

2 I

1 N

0 U

As a result, this phenomenon is sometimes called the NUXI* syndrome. This is only one way to look at it, of course: from a memory point of view, where the bytes are numbered left to right, it looks like 0 78 and 0 U 1 N 2 I 3 X 1 56 2 34 3 12

Its rather confusing to look at the number 0x12345678 as 78563412, so the NUXI (or XINU) view predominates. Its easier to grasp the concepts if you remember that this is all a matter of the mapping between bytes and words, and that text is always stored correctly from low byte to high byte. An alternative term for big-endian and little-endian is the term byte sex. To make matters even more confusing, machines based on the MIPS chips are veritable hermaphroditesall have congurable byte sex, and the newer machines can even run different processes with different byte sex. The problem of byte sex may seem like a storm in a teacup, but it crops up in the most unlikely situation. Consider the following code, originally written on a VAX, a little-endian machine:
int c = 0; read (fd, &c, 1); if (c == q) exit (0);

On a little-endian machine, the single character is input to the low-order byte of the word, so the comparison is correct, and entering the character q causes the program to stop. On a 32-bit big-endian machine, entering the character q sets c to the value 0x71000000, not the same value as the character q. Any good or even mediocre compiler will of course warn you if you hand the address of an int to read, but only if you remember to include the correct header les: it happens anyway.
* Why not XINU? Because the term arose when words were 16 bits long. The PDP-11, for example, stored ints (16 bit quantities) in a little-endian format, so pairs of bytes were swapped. The PDP-11 also had 32 bit long quantities that were stored with their component words in a big-endian format. This arrangement has been called mixed-endian, just to add to the general confusion.

5 February 2005 02:09

158

This discussion has concentrated on how characters are ordered within words, but the same considerations also affect bit elds within a word. Most hardware platforms dont support bit elds directly: theyre an idea in the mind of the compiler. Nonetheless, all architectures dene a bit order: some number from left to right, some from right to left. Well-written programs dont rely on the order of bit elds in ints, but occasionally you see register denitions as bit elds. For example, the 4.4BSD sources for the HP300 include the following denition:
struct ac_restatdb { short ac_eaddr; u_int ac_res1:2, ac_ie:1, ac_ee:1, ac_acc:1, ac_exc:1, ac_imp:1, ac_full:1; };

/* element address */ /* /* /* /* /* /* import enabled (IEE only) export enabled (IEE only) accessible from MTE */ element in abnormal state 1 == user inserted medium element contains media */ */ */ */ (IEE only) */

This denition denes individual bits in a hardware register. If the board in question ts in machines that number the bits differently, then the code will need to be modied to suit.

Data alignment
Most architectures address memory at the byte level, but that doesnt mean that the underlying hardware treats all bytes the same. In the interests of efciency, the processor accesses memory several bytes at a time. A 32-bit machine, for example, normally accesses data 4 bytes at a time this is one of the most frequent meanings of the term 32-bit machine. Its the combined responsibility of the hardware and the software to make it look as if every byte is accessed in the same way. Conicts can arise as soon as you access more than a byte at a time: if you access 2 bytes starting in the last byte of a machine word, you are effectively asking the machine to fetch a word from memory, throw away all of it except the last byte, then fetch another word, throw away all except the rst, and make a 16 bit value out of the two remaining bytes. This is obviously a lot more work than accessing 2 bytes at an even address. The hardware can hide a lot of this overhead, but in most architectures there is no way to avoid the two memory accesses if the address spans two bus words. Hardware designers have followed various philosophies in addressing data alignment. Some machines, such as the Intel 486, allow unaligned access, but performance is reduced. Others, typically RISC machines, were designed to consider this to be a Bad Thing and dont even try: if you attempt to access unaligned data, the processor generates a trap. Its then up to the software to decide whether to signal a bus error or simulate the transferin either case its undesirable. Compilers know about alignment problems and solve them by moving data to the next address that matches the machines data access restrictions, leaving empty space, so-called padding in between. Since the C language doesnt have any provision for specifying

5 February 2005 02:09

Chapter 11: Hardware dependencies

159

alignment information, youre usually stuck with the solution supplied by the compiler writer: the compiler automatically aligns data of specic types to certain boundaries. This doesnt do much harm with scalars, but can be a real pain with structs when you transfer them to disk. Consider the following program excerpt:
struct emmental { char flag; int count; short choice; int date; short weekday; double amount; } emmental; read_disk (struct emmental *rec) { if (read (disk, rec, sizeof (rec)) < sizeof (rec)) report_bad_error (disk); }

On just about any system, emmental looks like a Swiss cheese: on an i386 architecture, shorts need to be on a 2-byte boundary and ints and doubles need to be on a 4-byte boundary. This information allows us to put in the offsets:
struct emmental { char flag; /* 3 bytes empty space */ int count; short choice; /* 2 bytes empty space */ int date; short weekday; /* 2 bytes empty space */ double amount; } emmental;

/* offset 0 */ /* offset 4 */ /* offset 8 */ /* offset 12 */ /* offset 16 */ /* offset 20 */

As if this werent bad enough, on a Sparc doubles must be on an 8-byte boundary, so on a Sparc we have 6 bytes of empty space after weekday, to bring the offset up to 24. As a result, emmental has 21 useful bytes of information and up to 13 of wasted space. This is, of course, a contrived example, and good programmers would take care to lay the struct out better. But there are still valid reasons why you encounter this kind of alignment problem: If flag, count and choice are a key in a database record, they need to be stored in this sequence. A few years ago, even most good programmers didnt expect to have to align a double on an 8-byte boundary.

5 February 2005 02:09

160

A lot of the software you get looks as if it has never seen a good programmer. Apart from the waste of space, alignment brings a host of other problems. If the rst three elds really are a database key, somebody (probably the database manager) has to ensure that the gaps are set to a known value. If this database is shared between different machines, our read_disk routine is going to be in trouble. If you write the record on an i386, it is 28 bytes long. If you try to read it in on a Sparc, read_disk expects 32 bytes and fails. Even if you x that, amount is in the wrong place. A further problem in this example is that Sparcs are big-endian and i386s are little-endian: after reading the record, you dont just need to compact it, you also need to ip the bytes in the shorts, ints and doubles. Good portable software has accounted for these problems, of course. On the other hand, if your program compiles just ne and then falls at on its face when you try to run it, this is one of the rst things to check.

Instruction alignment
The part of the processor that performs memory access usually doesnt distinguish between fetching instructions from memory and fetching data from memory: the only difference is what happens to the information after it has reached the CPU. As a result, instruction alignment is be subject to the same considerations as data alignment. Some CPUs require all instructions to be on a 32 bit boundarythis is typically the case for RISC CPUs, and it implies that all instructions should be the same lengthand other CPUs allow instructions to start at any address, which is virtually a requirement for machines with variable length instructions.* As with data access, being allowed to make this kind of access doesnt make it a good idea. For example, the Intel 486 and Pentium processors execute instructions aligned on any address, but they run signicantly faster if the target address of a jump instruction is aligned at the beginning of a processor word the alignment of other instructions is not important. Many compilers take a ag to tell them to align instructions for the i486.

* Some machines with variable length instructions do have a requirement that an instruction t in a single machine word. This was the case with the Control Data 6600 and successors, which had a 60 bit word and 15 or 30 bit instructions. If a 30 bit instruction would have started at the 45 bit position inside a word, it had to be moved to the next word, and the last 15 bits of the previous instruction word were lled with a nop, a no-operation instruction.

5 February 2005 02:09

Kernel dependencies
The biggest single problem in porting software is the operating system. The operating system services play a large part in determining how a program must be written. UNIX versions differ enough in some areas to require signicant modications to programs to adapt them to a different version. In this and the following chapters, well look at what has happened to UNIX since it was essentially a single system, round the time of the Seventh Edition. Many books have been written on the internals of the various UNIX avours, for example The Design of the UNIX System by Maurice Bach for System V.2, The Design and the Implementation of the 4.3BSD UNIX Operating System by Sam Lefer, Kirk McKusick, Mike Karels, and John Quarterman for 4.3BSD, and The Magic Garden explained: The Internals of UNIX System V Release 4 by Berny Goodheart and James Cox for System V.4. In addition, a number of books have been written about programming in these environments Advanced Programming in the UNIX environment by Richard Stevens gives an excellent introduction to System V.4 and 4.3+BSD"* for programmers. In this chapter and the ones following it, well restrict our view to brief descriptions of aspects that can cause problems when porting software from one UNIX platform to another. Well look at specic areas in Chapter 12, Kernel dependencies, Chapter 13, Signals, Chapter 14, File systems and Chapter 15, Terminal drivers. In the rest of this chapter, well look at: Interprocess communication Non-blocking I/O Miscellaneous aspects of kernel functionality

The descriptions are not enough to help you use the functionality in writing programs: they are intended to help you understand existing programs and rewrite them in terms of functions available to you. If you need more information, you may nd it in the 4.4BSD man pages (see Appendix E, Where to get sources), or in Advanced Programming in the UNIX environment, by Richard Stevens.
* 4.3BSD was released in 1987, 4.4BSD in 1994. In the time in between, releases had names like 4.3BSD Tahoe, 4.3BSD Reno, and NET/2. For want of a better term, Stevens refers to systems roughly corresponding to NET/2 as 4.3+BSD. 161

5 February 2005 02:09

162

Interprocess communication
interprocess communication (frequently written as the abbreviation IPC), the ability to transfer data between processes, was one of the important original concepts of UNIX. The original methods were what you might expect of a concept that, at the time, was revolutionary and still under development: there were more than a few limitations. Even today there is no agreement on how interprocess communication should take place. In this section well look very briey at the various kinds of interprocess communication, and what to do if the package you are porting uses a method your kernel doesnt support. To start with the bad news: if you nd your kernel doesnt support the IPC model that the package expects, you will probably need to make signicant modications to adapt it to a different model. Interprocess communication was originally limited to a single processor, but of course network communication is also a form of interprocess communication. Well touch briey on network communication in the following discussion. UNIX systems offer a bewildering number of different forms of interprocess communication: Pipes are the original form of communication and are found in all versions of UNIX. They have the disadvantages that they transfer data in one direction only, and that they can only connect two processes that have a common ancestor. Sockets are the BSD interprocess communication mechanism: they are by far the most powerful mechanism, offering unidirectional, bidirectional and network communication. In BSD systems, they are even used to implement the pipe system call. STREAMS* is a generalized I/O concept available in newer System V systems and their derivatives. It was originally intended to replace character device drivers, but its exibility makes it useful for interprocess communication as well. Like sockets, it can be used both for local and remote communication. UNIX Network Programming, by Richard Stevens, describes STREAMS in some detail, and The Magic Garden Explained describes the implementation. We wont consider them further here. Stream pipes differ from normal pipes by being able to transfer data in both directions. They have no particular connection with STREAMS. FIFOs, also called named pipes, are like pipes, but they have a name in the le system hierarchy. Named stream pipes are stream pipes with names. They bear the same relationship to stream pipes that FIFOs do to normal pipes. System V IPC is a bundle that offers message queues, yet another form of message passing, shared memory, which enables processes to pass data directly, and semaphores, which synchronize processes.

* Why the shouting? STREAMS was derived from the Eighth Edition Streams concept (see S Stream Input-Output System, by Dennis Ritchie). System V always spells it in upper case, so this is a convenient way of distinguishing between the implementations.

5 February 2005 02:09

Chapter 12: Kernel dependencies

163

In the following sections, well look at these features in a little more detail.

Pipes
The original UNIX interprocess communication facility was pipes. Pipes are created by the pipe function call:
#include <unistd.h> int pipe (int *fildes);

This call creates a pipe with two le descriptors, a read descriptor and a write descriptor. It returns the value of the read descriptor to fildes [0] and the value of the write descriptor to fildes [1]. At this point, only the creating process can use the pipe, which is not very useful. After calling fork, however, both of the resultant processes can use the pipe. Depending on their purpose, the processes may decide to close one direction of the pipe: for example, if you write output to the more program, you dont expect any reply from more, so you can close the read le descriptor. A fair amount of code is involved in opening a pipe, starting a new process with fork and exec and possibly waiting for it terminate with wait. The standard library functions popen and pclose make this job easier:
#include <stdio.h> FILE *popen(const char *command, const char *type); int pclose(FILE *stream);

popen creates a pipe, then forks and execs a shell with command as its parameter. type species whether the pipe should be open for reading (r) or writing (w). Since pipes are unidirectional, they cannot be opened both for reading and for writing. After opening the command, you can write to the process with normal write commands. On completion, pclose waits for the child process to terminate and closes the le descriptors.

Sockets
Sockets were originally developed at Berkeley as part of the TCP/IP networking implementation introduced with 4.2BSD, but they are in fact a general interprocess communication facility. In BSD systems, the other interprocess communication facilities are based on sockets. Most of the features of sockets are related to networking, which we dont discuss here. The call is:
#include <sys/types.h> #include <sys/socket.h> int socket (int domain, int type, int protocol);

domain species the communications domain. Common domains are AF_UNIX (UNIX

5 February 2005 02:09

164

domain),* used for local communication, AF_INET (Internet domain), and AF_ISO (ISO protocol domain). type species the type of socket. For local interprocess communication, you would use SOCK_STREAM, which supplies a reliable two-way communication channel. protocol species the communications protocol to use. In the UNIX domain, this parameter is not used and should be set to 0.

As we shall see in the next section, the way that pipes are implemented means that you need two sockets to simulate a pipe. You can do this with the socketpair system call, which creates a pair of le descriptors with identical properties.
#include <sys/types.h> #include <sys/socket.h> int socketpair (int domain, int type, int protocol, int *sv);

Currently, socketpair works only in the UNIX domain, so you dont have much choice in the parameters: domain must be AF_UNIX, type must be SOCK_STREAM, and protocol is meaningless in the UNIX domain. The only important parameter is sv, which is where the socket descriptors are returnedexactly the same thing as the fildes parameter to pipe. Most systems have some kind of socket support, but sometimes it is just an emulation library that omits signicant functionality, such as the UNIX domain and the socketpair call. Many older System V sockets emulation libraries also have a bad reputation regarding performance and reliability. On the other hand, many System V.3 ports included the original Berkeley socket implementation in the kernel.

Other kinds of pipe

Pipes have two main restrictions: They are unidirectional: you write to one descriptor, you read from the other. It would be a nice idea to be able to read from and write to the same descriptor. They are anonymous: you dont open an existing pipe, you create a new one, and only you and your descendents can use it. It would be nice to be able to use pipes like regular les.

In fact, you can get all combinations of these properties. Weve seen regular pipesthe others are stream pipes, FIFOs and named stream pipes. Well look at them in the following sections:

* Not all UNIX implementations support UNIX domain sockets. In particular, some System V systems support only the Internet domain. People with a System V background often place the emphasis on the word domain, and some even refer to UNIX domain sockets as domain sockets. As you can see from the above, this is incorrect.

5 February 2005 02:09

Chapter 12: Kernel dependencies

165

Stream pipes
Most systems allow you to create bidirectional pipes. For some reason, theyre generally called stream pipes, which is not a good name at all. In System V.4, regular pipes are bi-directional, so you dont need to do anything special. In 4.4BSD, the socketpair system call, which we have already seen, creates stream pipes, so youd expect regular pipes to be bidirectional in 4.4BSD as well. In fact, before returning, the library function pipe closes one descriptor in each direction, so 4.4BSD pipes really are unidirectional. If you want a stream pipe, just use the socketpair system call. In System V.3 systems with STREAMS, bidirectional pipes are possible too, but things are more difcult: you have to connect two streams back to back. See UNIX Network Programming for a discussion of how to do this.

FIFOs
FIFOs are pipes with le names, which allow unrelated processes to communicate with each other. To create a FIFO, you use the function mkfifo:
#include <sys/stat.h> int mkfifo (const char *path, mode_t mode);

This call corresponds exactly to mkdir, except that it creates a FIFO instead of a directory. BSD implements mkfifo as a system call, while System V.4 implements it as a library function that calls mknod to do the work. System V.3 systems frequently implemented it as a system call. Once you have created a FIFO, you can use it just like a le: typically, one process, the listener process, opens the FIFO for reading, and one or more open it for writing to the listener process.

Named stream pipes

Stream pipes are bidirectional, but they dont normally have names. FIFOs have names, but theyre usually not bidirectional. To get both of these properties, we need a new kind of connection, a named stream pipe. In 4.4BSD, this can be done by binding a name to a stream socket see the man pages for bind for further details. In System V.4, you can create a named stream pipe with the connld STREAMS module. See Advanced Programming in the UNIX environment for more details.

System V IPC
System V supplies an alternative form of interprocess communication consisting of three features: shared memory, message queues and semaphores. SunOS 4 also supports System V IPC, but pure BSD systems do not. In the industry there is a signicant amount of aversion to this implementation, which is sometimes called The Three Ugly Sisters.

5 February 2005 02:09

166

System V IPC is overly complicated and sensitive to programming bugs, which are two of the main reasons why it has not been implemented on other systems. Converting programs written for System V IPC to other methods of interprocess communication is non-trivial. If you have a BSD system with kernel sources, it might be easier to implement Daniel Boulets free software implementation (see Appendix E, Where to get sources).

Shared memory
An alternative form of interprocess communication involves sharing data between processes. Instead of sending a message, you just write it into a buffer that is also mapped into the address space of the other process. There are two forms of shared memory that you may encounter on UNIX systemsSystem V shared memory and mmap, which is more commonly used for mapping les to memory. Well look at mmap in Chapter 14, File systems, page 232. System V shared memory is implemented with four system calls:
#include <sys/types.h> #include <sys/ipc.h> #include <sys/shm.h> int shmget (key_t key, int size, int shmflg); int shmctl (int shmid, int cmd, ... /* struct shmid_ds *buf */); void *shmat (int shmid, void *shmaddr, int shmflg); int shmdt (void *shmaddr);

shmget allocates a shared memory segment or adds the process to the list of processes sharing the segment. The shared memory segment identier is conceptually like a le name or an identier, but for some reason they are called keys when talking about System V shared memory. It returns a segment identier, conceptually like a le number. shmctl performs control operations on shared memory segments. It can set ownerships and permissions, retrieve status information, or remove shared memory segments. Like les, shared memory segments remain on the system until explicitly removed, even if they are currently not assigned to any process. shmat attaches the shared memory segment shmid to the calling process. shmdt detaches a shared memory segment from the calling process.

With some limitations, you can use mmap to replace System V shared memory. The limitations are that mmap on non-System V platforms normally maintains separate data pages for each process, so if you write to a page in one process, other processes will not see the new data. You need to call msync in order to update the segments used by other processes. Between the time when you modify the segment and when you call msync, the data is inconsistent. msync is not a fast call, so this could also cripple performance.

5 February 2005 02:09

Chapter 12: Kernel dependencies

167

Message queues
As if there werent enough ways of passing data between processes already, System V IPC includes message queues. Message queues are rather like FIFOs, but there are two differences: A FIFO transmits a byte stream, but a message queue is record oriented. Messages can have different priorities, which determine the sequence in which they are received, if the receiving process allows them to queue up.

The system calls to handle message queues are analogous to the shared memory calls:
#include <sys/types.h> #include <sys/ipc.h> #include <sys/msg.h> int int int int msgget msgsnd msgrcv msgctl (key_t key, (int msqid, (int msqid, (int msqid, int msgflg); const void *msgp, size_t msgsz, int msgflg); void *msgp, size_t msgsz, long msgtyp, int msgflg); int cmd, .../* struct msqid_ds *buf */);

msgget opens an existing queue or creates a new queue. msgsnd sends a message. msgrcv receives a message. msgctl performs control functions on message queues.

Message queues were originally intended to offer fast interprocess communication. Nowadays they have little to offer that a FIFO couldnt handle. If you run into problems with message queues, you might prefer to replace them with FIFOs.

Semaphores
One disadvantage with shared memory implementations is that one process doesnt know when another process has done something. This can have a number of consequences: Two processes may modify the same area at the same time. One process may be waiting for the other process to do something, and needs to know when it has nished.

There are two possible solutions: send a signal, or use semaphores. A semaphore is a means of voluntary process synchronization, similar to advisory locking. To use the facility, a process requests access to the semaphore. If access is currently not possible, the process blocks until access is permitted. Unlike locking, semaphores allow more than one process access to the semaphore at any one point. They do this by maintaining a counter, a small positive integer, in the semaphore. When a process accesses the semaphore, it decrements the counter. If the value of the counter is still non-negative, the process has access, otherwise it is blocked. This could be used to gain access to a limited number of resources.

5 February 2005 02:09

168

System V semaphores look supercially similar to System V shared memory. There are three functions:
int semctl (int semid, int semnum, int cmd, ... /* union semun arg */); int semget (key_t key, int nsems, int semflg); int semop (int semid, struct sembuf *sops, size_t nsops);

The implementation is less than perfect. In particular, it is overly complex, and it almost encourages deadlocks, situations where no process can continue: Instead of a single counter, a System V semaphore declares an array of counters. The size of the array is determined by the nsems parameter of the semget system call. It takes two calls (semget and semctl) to create and initialize a semaphore. Theoretically, this creates an opportunity for another process to come and initialize the semaphore differently. Its possible for semaphores to remain locked after a process ends, which means that a reboot is necessary to unlock the semaphore again. A ag is provided to specify that a semaphore should be removed on exit, but you cant rely upon it completely. The implementation is not very fast.

Miscellaneous system functionality

The rest of this chapter describes miscellaneous system calls that can occasionally cause problems when porting.

exec
exec is one of the original system calls at the heart of the UNIX system, so it may come as a surprise to discover that exec is no longer a system call on modern systemsinstead, it is implemented as a library function in terms of new system calls such as execve. Even the Seventh Edition man pages stated
Plain exec is obsoleted by exece, but remains for historical reasons.

Nowadays, there are a large number of alternatives. Your system probably has most of the following calls:
#include <unistd.h> extern char **environ; int int int int int int int exec exece execl execle execlp execlpe exect (char (char (char (char (char (char (char *path, *path, *path, *path, *file, *file, *path, char char char char char char char *argv *argv *arg, *arg, *arg, *arg, *argv []); [], char *envp []); ..., NULL); ..., NULL, char *envp []); ..., NULL); ..., NULL, char *envp []); [], char *envp []);

5 February 2005 02:09

Chapter 12: Kernel dependencies

int int int int execv execve execvp execvpe (char (char (char (char *path, *path, *file, *file, char char char char *argv *argv *argv *argv []); [], char *envp []); []); [], char *envp []);

169

All these functions do exactly the same thing: they replace the process image with a process image from the absolute executable whose le name is specied in the rst argument (path or file). They differ only in the manner in which they supply the parameters: The parameter path species an absolute pathname. If this le does not exist, the call fails. Alternatively, the parameter file species a le to be searched via the PATH environment variable, the way the shell does when a le name is specied. The parameter argv is a pointer to a NULL terminated list of parameters. Alternatively, you can place the arguments, including the terminating NULL, in the call as a series of args. If the parameter envp is specied, it is a pointer to a NULL-terminated list of environment variables. This is typically used when the child process should be given a different environment from the parent process. If envp is not specied, the environment variables are taken from the parents environment (via the global pointer environ).

One further function deserves mention: exect, which is supplied only in newer BSD systems, takes the same parameters as execve, but enables program tracing facilities. The total storage available for the argument list and the enviroment varies from system to system. System V traditionally has only 5120 characters. POSIX.1 requires at least 20480 characters, and this is the standard value for newer BSD systems. Many Makeles take advantage of these large parameter lists, and frequently a package fails to build under System V because the parameter lists are too long: you get the message
make: execve: /bin/sh: Arg list too long

We looked at what we can do to solve these problems in Chapter 5, Building the package, page 74.

getrlimit and setrlimit

The Seventh Edition made a number of arbitrary choices about kernel limits. For example, each process was allowed to have 50 les open at any one time. In the course of time, a number of these kernel limits were made congurable, and some systems allowed the process to modify them directly, up to a hard limit. SunOS, BSD and System V.4 supply the system calls getrlimit and setrlimit in order to manipulate this conguration information:
#include <sys/time.h> #include <sys/resource.h> struct rlimit

5 February 2005 02:09

170
{ int rlim_cur; int rlim_max; };

/* current (soft) limit */ /* hard limit */

int getrlimit (int resource, struct rlimit *rlp); int setrlimit (int resource, struct rlimit *rlp);

The rlimit structure denes two values for each resource, the current value and the maximum value. getrlimit returns this information, setrlimit sets a new current value. Table 12-1 shows which limits can be set:
Table 121: getrlimit and setrlimit resources resource RLIMIT_CORE RLIMIT_CPU RLIMIT_DATA RLIMIT_FSIZE RLIMIT_MEMLOCK RLIMIT_NOFILE

System all all all all 4.4BSD all

Description The maximum size, in bytes, of a core image le. The maximum amount of CPU time that a process may consume. The maximum size, in bytes, of the process data segment. The largest size, in bytes, that any le may attain. The maximum size, in bytes, which a process may lock into memory using the mlock function. The maximum number of les that a process may open at one time. This is also one more than the highest le number that the process may use. The maximum number of simultaneous processes for the current user id. The maximum size, in bytes, that the resident set of a processes may attain. This limits the amount of physical memory that a process can occupy. The maximum size, in bytes, that the stack segment of a processes may attain. The maximum size, in bytes, that the mapped address space of a processes may attain.

RLIMIT_NPROC RLIMIT_RSS

4.4BSD 4.4BSD, SunOS 4 all System V.4

RLIMIT_STACK RLIMIT_VMEM

If your system doesnt have these functions, theres not much you can do except guess. In some cases, header les will contain similar information declared as constants, but its not a very satisfactory alternative.

5 February 2005 02:09

Chapter 12: Kernel dependencies

171

Process groups
Where other operating systems use a single program to perform an operation, UNIX frequently uses a group of cooperating processes. Its useful to be able to dene such a group, particularly when they access terminals. Advanced Programming in the UNIX environment, by Richard Stevens, describes all you will want to know about process groups. Here, well look at some minor differences in implementations.

setpgid
setpgid adds a process to a process group:
#include <unistd.h> int setpgid (pid_t pid, pid_t pgrp);

pid is the process ID of the process that is to be added to the process group, and pgrp is the process group to which it should be added. It returns 0 on success and -1 with an error code in errno on failure. Normally you will see setpgid used to add the calling process to a group; this can be done by setting pid to 0. System V versions also allow pgrp to be 0: this species that the process id should be the same as pid, and that this process will become a process group leader.

setpgrp
setpgrp is obsolescent. There are two different implementations, both of which duplicate functionality supplied by other functions: In more modern BSD systems, it is the same thing as setpgid:
int setpgrp (pid_t pid, pid_t pgrp); BSD versions

In System V, it creates a new process group with the calling process as group leader, and adds the calling process to the group. It also releases the controlling terminal of the calling process. This is the same thing as setsid:
int setpgrp (); System V versions

If you run into trouble with this function, its best to replace it with setpgid or setsid, depending on the functionality that was intended.

setsid
setsid creates a new process group with the calling process as group leader, and adds the calling process to the group. It also releases the calling process from its controlling terminal:
#include <unistd.h> int setsid ();

5 February 2005 02:09

172

Real and effective user IDs

Occasionally the UNIX security system causes unintended problems: a trusted program may require access to facilities to which the user should not have unlimited access. For example, the program ps requires access to /dev/kmem, kernel memory, which is normally accessible only to the super user. Serial communication programs such as uucp require access to the serial ports, but in order to avoid conicts, only trusted users have access to the ports. UNIX solves this problem by allowing the programs always to run as a specic user or group. If you execute a program that has the setuid bit set in the le permissions, it runs as if its owner had execed it, no matter who really started it. Similarly, the setgid bit causes the program to run as if it had been executed in the group to which the le belongs. These user and group ids are called effective user ID and effective group ID, and they are the only permissions that are relevant when a process accesses a le. Similar considerations apply to group IDs. In the following discussion, well consider user IDs, but unless mentioned otherwise, everything I say about user IDs also applies to group IDs. A number of subtle problems arise from this scheme. One of the most obvious ones is that programs frequently also need to be able to access your les. Theres no guarantee that this will always work. For example, uucp needs to be setuid to user uucp in order to access the communication ports, but it also frequently needs to transfer data to your home directory. If your permissions are set so that uucp cant access your home directory, it will not be able to perform the transfer. This is obviously not the intention: somehow, uucp needs access both to the serial ports and to your les. This means that we need to maintain at least two user IDs, the effective user ID and the real user ID. Modern systems also supply a saved set user ID. On System V.4, its a conguration option (set the conguration constant _POSIX_SAVED_IDS). BSD uses the saved set user ID in a different way from System V, as we will see below. The system manipulates user IDs in the following ways: If you execute a program that is not setuid, it sets all IDs to the effective user ID of the process that executes it. If you execute a program that has the setuid permission set, it sets the effective user ID to the owner of the program, and the real user ID to the effective ID of the process that executes it. If there is a saved set user ID, it also sets it to the owner of the program. At run time you can change between IDs with the system call setuid. There are also a number of alternative calls. Well look at them in the following sections.

setuid
setuid changes the effective user ID. If your current effective user ID is root, you can set it to any valid user ID. There, unfortunately, the similarity ends: In systems without a saved set user ID, including SunOS 4 and System V.3, setuid sets the effective user ID and the real user ID if the current effective user ID is root, otherwise

5 February 2005 02:09

Chapter 12: Kernel dependencies

173

it sets only the effective user ID. The function call succeeds if the argument to setuid is the real user ID or the effective user ID, or if the effective user ID is root. Once you have changed away from the old effective user ID and root, there is no way to change back. On System V systems with saved set user ID, setuid sets the effective user ID and the real user ID if the current effective user ID is root, otherwise it sets only the effective user ID. It does not change the saved set user ID. The function call succeeds if the argument to setuid is the real user ID, the effective user ID, or the saved set user ID, or if the effective user ID is root. This means that you can switch back and forth between the ID of the program owner and the ID of the process which started it. On BSD systems with saved set user ID, setuid sets the real, effective, and saved set user IDs. The function call succeeds if the argument to setuid is the real user ID, or if the effective user ID is root. Unlike System V.4, non-root users cannot use setuid to set the user ID to the saved set user ID. The saved set user ID is of no use to BSD setuidinstead, BSD systems use seteuid, which sets only the effective user ID to either the real user ID or the saved set user ID.

setreuid
BSD versions since 4.2BSD have the system call setreuid, which takes two parameters:
int setreuid (int ruid, int euid);

You can use it to swap the effective and real user IDs, so you dont really need a saved set user ID. For non-privileged users, ruid and euid can be either the current real user ID or the current effective user ID, or -1 to indicate no change. This function was needed in BSD up to and including 4.3BSD, since these versions did not support the concept of a saved set user ID. On non-BSD systems only, you can replace this function with setuid if your system supports saved set user IDs.

seteuid
As we noted above, BSD setuid cannot change to the saved set user ID. The BSD solution to this problem, which has been proposed for adoption in a new revision of POSIX.1, is the function seteuid. It sets the effective user ID to euid if euid corresponds either to the real user ID or the saved set user ID. Unlike setuid, it sets only the effective user ID.

setruid
In addition to seteuid, BSD systems provide the call setruid, which sets the real user ID to the effective or real user ID. setruid is considered non-portable. Future BSD releases plan to drop it.

5 February 2005 02:09

174

Comparison of user ID calls

User IDs are much more complicated than they should be. In fact, there are only two things youll want to do, and for non-root users they work only with programs which have setuid permissions: change from the initial effective user ID to the real user ID, and change back again. Changing from effective to real user ID is simple: in all systems, you can use the setuid system call, though in 4.3BSD and SunOS 4 this will mean that you cant change back. In these systems, its better to use code like
int euid = geteuid (); int ruid = getuid (); setreuid (euid, ruid); /* get current effective user ID */ /* and real user ID */ /* and swap them */

Changing back again is more complicated: On older systems, including XENIX and System V.3, and on System V.4 systems without _POSIX_SAVED_IDS, you cant do it. For the older systems, about the only workaround is not to change away from the initial effective user IDyou might be able to spawn a process which does the necessary work under the real user ID. On BSD systems up to and including 4.3BSD, and under SunOS 4, you can do it only if you changed with setreuid, as in the example above. In this case, you just need to continue with
setreuid (ruid, euid);

On System V.4 systems with _POSIX_SAVED_IDS, use setuid (ssuid), where ssuid is the saved set user ID. You can get the value of ssuid by calling geteuid before changing the initial effective user ID, since theyre the same at program start. On BSD systems which support saved set user IDs, use seteuid (ssuid). As with System V.4, you can get the value of ssuid by calling geteuid before changing the initial effective user ID.

vfork
vfork was introduced in 3BSD as a more efcient version of fork: in those days, fork copied each data area page of the parent process for the child process, which could take a considerable time. Typically, the rst thing a child does is to call exec to run a new program, which discards the data pages, so this was effectively wasted time. vfork modied this behaviour so that the pages were shared and not copied. This is inherently very dangerous: very frequently the parent waits until the child has done something before continuing. During this time, the child can modify the parents data, since it is shared. More modern techniques, such as copy on write*, have eliminated the need for this function. You should be able to replace it with fork (the semantics are identical). Unfortunately, some obscene programs rely on the fact that they can manipulate the parents data
* With copy on write, the data pages are set to be write-protected. The rst write causes an interrupt, effectively a bus error, which the system intercepts. The system makes a copy of the single page and resets write protection for both the original and the copy, allowing the write to proceed.

5 February 2005 02:09

Chapter 12: Kernel dependencies

175

before the parent continues. These programs need to be xed.

wait and friends

wait has been in UNIX as long as anybody can remember:
#include <sys/types.h> #include <sys/wait.h> pid_t wait (int *status);

Unfortunately, various avours dene the value of the status return differently. This is a cosmetic difference, not a real difference: the status information consists of a number of bit elds that depend on the kind of status: The low-order 7 bits contain the number of the signal that terminated the process, or 0 if the process called exit. The bit 0x80 is set if a core dump was taken. The next 8 bits are the return code if the process called exit.

If the process is stopped (if it can be restarted), the low-order 8 bits are set to 127 (0x7f), and the next byte contains the number of the signal that stopped the process. This information is the same on all versions of UNIX, but there is no agreement on how to represent this information. Older BSD systems dened a union to represent it:
union __wait { int w_status; struct { unsigned short unsigned short unsigned short } w_T; struct { unsigned short unsigned short } w_S; };

/* status as int */

w_Termsig:7; w_Coredump:1; w_Retcode:8;

/* termination signal */ /* core dump indicator */ /* exit code if w_termsig==0 */

w_Stopval:8; w_Stopsig:8;

/* == W_STOPPED if stopped */ /* signal that stopped us */

Modern systems dene macros: WIFEXITED (status) is true if the process terminated via a call to exit. If this is true, WEXITSTATUS (status) returns the low order 8 bits of the process exit value. WIFSIGNALED (status) is true if the process was terminated by receiving a signal. If this is true, the following macros apply:

5 February 2005 02:09

176

WTERMSIG (status) evaluates to the number of the signal that caused the termination of the process. WCOREDUMP (status) is true if a core dump was created.

WIFSTOPPED (status) is true if the process is stopped and can be restarted. This macro can be true only if the waitpid call specied the WUNTRACED option or if the child process is being traced. If this is true, WSTOPSIG (status) returns the number of the signal that caused the process to stop.

Some systems offer both of these options, sometimes incompletely. For example, SunOS 4 denes w_Coredump in the union __wait, but does not dene the corresponding WCOREDUMP macro. These varying differences cause problems out of all proportion to the importance of the information contained. In particular, the newer macros do not allow you to change the status, you can only read it. Some programs, for example BSD make, modify the status. This makes it difcult to port it to System V or another system which does not understand union wait.

waitpid
waitpid is a variant of wait that waits for a specic process to terminate. It is part of all modern UNIX implementations:
#include <sys/wait.h> pid_t waitpid (pid_t wpid, int *status, int options);

waitpid waits for process pid to terminate. Its behaviour is governed by a number of bitmapped options: Set WNOHANG to specify to return immediately, even if no status is available. If the status is not available, the functions return the process number 0. Not all systems support this behaviour. Specify WUNTRACED if you want the status of stopped processes as well as processes that have terminated. Some systems do not return complete status information for stopped processes. Under System V.4, use WCONTINUED to report the status of any process that has continued (in other words, one that is no longer stopped) since the last status report. Also under System V.4 you can set the option WNOWAIT to specify that the process should not terminate (it remains a zombie). This means that you can call waitpid again and get the same information.

The value of status is the same as with waitsee the previous section for further details. If you run into problems with waitpid, it may be a bug: some versions of System V.3, including most current versions of SCO UNIX, return a process ID if a process is waiting, and an error number such as ECHILD (10) if nothing is waiting, so if your freshly ported program keeps reporting the demise of process 10, this could be the problem. Its almost impossible to

5 February 2005 02:09

Chapter 12: Kernel dependencies

177

work around this bug about the only thing you can do is to use some other system call.

wait3 and wait4

Newer BSD implementations supply the functions wait3 and wait4 as alternatives to wait. They correspond to wait and waitpid respectively, but return an additional parameter rusage with accounting information for the terminated process:
pid_t wait3 (int *status, int options, struct rusage *rusage); pid_t wait4 (pid_t wpid, int *status, int options, struct rusage *rusage);

Not all implementations return usage information to rusage when the process is stopped (and not terminated). The denition of struct rusage is implementation-dependent and dened in sys/resource.h. See the le sys/sys/resource.h in the 4.4BSD Lite distribution for further details.

5 February 2005 02:09

Signals
Signals are another area in UNIX where the initial implementation was inadequate, and multiple implementations have developed in the course of the time. If you try to port software which assumes the presence of one implementation, and your system doesnt support this implementation, you could be in for a signicant amount of rewriting. The situation isnt improved by the fact that there are a number of subtle differences between the various implementations and even between different systems with the same implementation. In this chapter, well look at those aspects of signal implementation which are of interest to porting. There have been four different implementations in the course of UNIX history: The Seventh Edition had so-called unreliable signals. and handled them with the signal system call. System V still supplies them with the signal system call. As we will see on page 188, the use of the signal function does not automatically imply unreliable signals. 4.2BSD introduced the rst implementation of reliable signals. It uses the functions signal, sigvec, sigblock, sigsetmask and sigpause. System V introduced an alternative implementation of reliable signals. It uses the functions sigset, sighold, sigrelse, sigignore and sigpause. Finally, POSIX.1 dened a third implementation of reliable signals. These are based on the BSD signals and use the functions sigaction, sigprocmask, sigpending and sigsuspend.

Most people think of signals as the way the operating system or an outside user stops a program that is misbehaving. More generally, they are a means to cause execution of functions out of sequence, and have thus been called software interrupts. Hardware interrupts cause the system to interrupt normal processing and perform a specic sequence of instructions. Signals do the same thing in software: when a process receives a signal, the kernel simulates a call to a pre-dened routine.* The routine, called a signal handler, handles the signal and possibly returns to the caller. It would be a signicant overhead for every program to supply a
* This is not a real call: when the kernel delivers the signal, it modies the process stack and registers so that it looks as if the signal handler has just been called. When the process continues executing, it is in the signal handler. Nobody ever really calls the signal handler. 179

5 February 2005 02:09

180

signal handler for every conceivable signal, so the kernel supplies two default methods of handling the signal. The choice of a signal handler or one of the two defaults is called the disposition of the signal. Initially, each signals disposition is set either to ignore the signal or to terminate the process if the signal occurs. In some cases, the system writes a core le, a copy of the state of the process, when the process is terminated. Signals may come from a number of different sources: External events. For example, pressing CTRL-C or DEL on most systems causes the terminal driver to send a SIGINT signal to the foreground process group of the terminal. Internal events. For example, alarm causes a SIGALRM signal after the specied timeout. Hardware interrupts. For example, if a process attempts to access a page that is not part of its address space, it will receive a SIGSEGV or SIGBUS signal. As the result of another process calling kill.

In this chapter, well consider which signals are supported by which operating systems, and how signals are implemented in different operating systems.

Supported signals
The Seventh Edition had 15 signals, and current implementations allow up to 31, though not all are used. In the course of time, the meanings have also diverged somewhat. Table 13-1 gives an overview of which signals are present in which implementations.
Table 131: Signal usage Signal

V 7

S V R 3

S V R 4

B S D

SIGABRT SIGALRM SIGBUS SIGCHLD SIGCLD SIGCONT SIGEMT SIGFPE SIGHUP SIGILL SIGINFO

P O S I X

action

purpose

core kill core ignore ignore ignore core core kill core ignore

abort call2 real-time timer expired bus error child status has changed child status has changed continue after stop emulate instruction executed oating-point exception line hangup illegal instruction status request from keyboard

5 February 2005 02:09

Chapter 13: Signals Table 131: Signal usage (continued) Signal V S S B

181

V R 3

V R 4

S D

P O S I X

action

purpose

SIGINT SIGIO SIGIOT SIGKILL SIGPIPE SIGPROF SIGPWR SIGQUIT SIGSEGV SIGSTOP SIGSYS SIGTERM SIGTRAP SIGTSTP SIGTTIN SIGTTOU SIGURG SIGUSR1 SIGUSR2 SIGVTALRM SIGWINCH SIGXCPU SIGXFSZ
1

kill ignore core kill kill kill ignore core core stop core kill core stop stop stop ignore kill kill kill ignore core core

interrupt program (usually from terminal driver) I/O completion outstanding1 IOT instruction2 kill program4 write on a pipe with no reader proling timer alarm power fail/restart quit program (usually from terminal driver) segmentation violation stop4 invalid system call software termination signal trace trap stop signal generated from keyboard background read from control terminal background write to control terminal urgent condition present on socket User dened signal 1 User dened signal 2 virtual time alarm Window size change cpu time limit exceeded le size limit exceeded

Sometimes called SIGPOLL in System V. SIGIOT and SIGABRT usually have the same signal number. 3 Not available in all versions. 4 This signal cannot be caught or ignored.
2

Unreliable and reliable signals

The terms unreliable signals and reliable signals need explaining. The problem relates to what happens when a signal handler is active: if another signal occurs during this time, and it is allowed to be delivered to the process, the signal handler will be entered again. Now its not difcult to write reentrant* signal handlersin fact, its a very good idea, because it
* A reentrant function is one which can be called by functions which it has calledin other words, it

5 February 2005 02:09

182

means that you can use one signal handler to handle multiple signalsbut if the same signal reoccurs before the signal handler has nished handling the previous instance, it could happen again and again, and the result can be a stack overow with repeated signal handler calls. The original signal implementation, which we call unreliable signals, had a simplistic attitude to this problem: it reset the signal dispostion to the default, which meant that if another signal occurred while the previous one was being processed, the system would either ignore the signal (so it would lose the signal) or terminate the process (which is probably not what you want). It was up to the signal handler to reinstate the signal disposition, and this couldnt be done immediately without running the risk of stack overow. All newer signal implementations provide so-called reliable signals. The signal disposition is not changed on entry to the signal handler, but a new signal will not be delivered until the signal handler returns. This concept is called blocking the signal: the system notes that the signal is pending, but doesnt deliver it until it is unblocked. There are a number of things that the term reliable signal does not mean: It doesnt imply that the underlying kernel implementation is bug-free. Depending on the implementation, there is still a slight chance that the kernel will lose the signal. It doesnt imply that a signal cannot get lost. The method used to queue signals is to set a bit in a bit mask. If multiple signals of the same kind occur while the signal is blocked, only one will be delivered. It doesnt imply that you dont need reentrant signal handlers. The system blocks only the signal that is currently being handled. If you have a single handler for multiple signals, it will need to be reentrant. In particular, this means that you should at least be very careful with static variables and preferably use few local variables (since they take up stack space). You should also be careful with the functions you callwell take another look at this on page 187.

The semantics of each implementation differ in subtle ways, so changing to a different set of signal calls involves more than just changing the function calls and parameters. Heres a brief overview of the differences you might encounter: With unreliable signals, after a signal occurs, the signal disposition is reset to default, so the signal handler must reinstate itself before returning. If a second signal occurs before the disposition is reinstated, the process may be terminated (if the default disposition is terminate) or the signal may be completely forgotten (if the default disposition is ignore). The names and purposes of the signals differ signicantly from one implementation to the next. See Table 13-2 for an overview. In modern implementations, the function call signal varies in its meaning. In System V, it uses the old, unreliable Seventh Edition signal semantics, while in BSD it is an interface to the sigaction system call, which provides reliable signals. If youre porting BSD signal to System V, you should modify the code use sigaction instead.

can be entered again before it has returned. This places a number of restrictions on the function. In particular, it cannot rely on external values, and may not use static storage.

5 February 2005 02:09

Chapter 13: Signals

183

The rst parameter to a signal handler is always the number of the signal. Both System V.4 and BSD can supply additional parameters to the signal handlers. Well look at the additional parameters in more detail on page 183. The handling of interrupted system calls varies from one system to the next. Well look into this topic in more detail on page 186. The difference between the signals SIGBUS and SIGSEGV is purely historical: it relates to the PDP-11 hardware interrupt that detected the problem. In modern systems, it depends on the whim of the implementor when you get which signal. POSIX.1 denes only SIGSEGV, but this doesnt help much if the processor generates SIGBUS anyway. Its best to treat them as being equivalent. SIGCLD is a System V version of SIGCHLD. A number of hairy problems can arise with SIGCLD; well look at them in more detail on page 186. SIGILL was generated by the abort function in early BSD implementations. Early System V used SIGIOT instead. All modern implementations generate SIGABRT. Frequently youll nd that these two signals are in fact dened to have the same number; if you run into troubles where one or the other is undened, you could possibly do just this:
#define SIGIOT SIGABRT

Signal handlers
Modern versions of UNIX dene signal handlers to be of type
void (*signal (int signum, void (*handler))) (int hsignum)

This is probably one of the most confusing denitions you are likely to come across. To understand it, it helps to remember that we are talking about two functions: The signal handler, called handler in this declaration, takes an int parameter hsignum and returns a void pointer to the old signal handler function, which is of the same type as itself. The function signal, which takes two parameters. The rst is signum, the number of the signal to be handled, of type int, and the second is a pointer to a signal handler function handler. It also returns a void pointer to a signal handler function.

In fact, in many implementations the signal handler function takes additional parameters, and you may nd that your program takes advantage of them. Well look at these in the following sections.

System V.4 signal handlers

The System V.4 signal handler interface offers additional functionality in certain circumstances: if you use the sigaction interface and you set the ag SA_SIGINFO in sa_flags, the signal handler is invoked as if it were dened

5 February 2005 02:09

184
void handler (int signum, struct siginfo *info, struct ucontext *context);

siginfo is an enormous structure, dened in /usr/include/siginfo.h, which starts with

struct siginfo { int si_signo; int si_code; int si_errno; ... more stuff, including space for further }

/* signal from signal.h */ /* code from above */ /* error from errno.h */ growth

ucontext is dened in /usr/include/ucontext.h and contains information about the user context at the time of the signal application. It includes the following elds: uc_sigmask is the blocked signal mask. us_stack points to the top of stack at the time the signal was delivered. uc_mcontext contains the processor registers and any implementation specic context data.

For example, assume you had set the signal handler for SIGFPE with the call in Example 13-1.
Example 131:
void bombout_handler (int signum, struct siginfo *info, struct ucontext *context); sigset_t bombout_mask; struct sigaction bad_error = {&bombout_handler, handler for the signal &bombout_mask, signals to mask SA_SIGINFO}; we want additional info sigemptyset (&bombout_mask); sigaction (SIGFPE, &bad_error, NULL); no signals in mask

On receipt of a SIGFPE, signal will be set to SIGFPE. info->si_signo will also be set to SIGFPE. On an i386 machine, info->si_code might be, for example, FPE_INTDIV (indicating an integer divide by zero) or FPE_FLTUND (indicating oating point underow). The value of info->si_errno cant be relied on to have any particular value. context->uc_sigmask contains the current signal mask. context->uc_stack will point to the stack in use at the time the signal was delivered.

5 February 2005 02:09

Chapter 13: Signals

185

context->uc_mcontext will contain the contents of the processor registers at the time of the interrupt. This can be useful for debugging.

BSD signal handlers

BSD signal handlers do not use the ag SA_SIGINFO for sa_flags. Signal handlers always receive three parameters:
void handler (int signum, int code, struct sigcontext *context);

code gives additional information about certain signalsyou can nd this information in the header le /usr/include/machine/trap.h. This le also contains information about how hardware interrupts are mapped to signals. context is hardware-dependent context information that can be used to restore process state under some circumstances. For example, for a Sparc architecture it is dened as
struct sigcontext { int sc_onstack; int sc_mask; /* begin machine dependent portion */ int sc_sp; int sc_pc; int sc_npc; int sc_psr; int sc_g1; int sc_o0; };

/* sigstack state to restore */ /* signal mask to restore */ /* /* /* /* /* /* %sp to restore */ pc to restore */ npc to restore */ psr to restore */ %g1 to restore */ %o0 to restore */

The program of Example 13-1 wont compile under BSD, since BSD doesnt dene SA_SIGINFO, and the parameters for bombout_handler are different. We need to modify it a little:
void bombout_handler (int signum, int code, struct sigcontext *context); sigset_t bombout_mask; struct sigaction bad_error = {&bombout_handler, handler for the signal &bombout_mask, signals to mask 0}; ... the rest stays the same

If you enter this signal handler because of a SIGFPE, you might nd: signum will be set to SIGFPE. On an i386 machine, code might be, for example, FPE_INTOVF_TRAP (indicating an integer divide by zero) or FPE_FLTUND_TRAP (indicating oating point underow). The value of sc_onstack would be the previous sigstack state. context->sc_mask contains the current blocked signal mask, like context->uc_sigmask in the System V.4 example.

5 February 2005 02:09

186

The rest of the context structure shows the same kind of register information that System V.4 stores in context->uc_mcontext.

SIGCLD and SIGCHLD

System V treats the death of a child differently from other implementations: The System V signal SIGCLD differs from the BSD and POSIX.1 signal SIGCHLD and from all other signals by remaining active until you call wait. This can cause innite recursion in the signal handler if you reinstate the signal via signal or sigset before calling wait. If you use the POSIX.1 sigaction call, you dont have to worry about this problem. When a child dies, it becomes a zombie. As all voodoo fans know, a zombie is one of the Living Dead, neither alive nor dead. In UNIX terminology, when a child process dies it becomes a zombie: the text and data segments are freed, and the les are closed, but the process table entry and some other information remain until it is exorcized by the parent process, which is done by calling wait. By default, System V ignores SIGCLD and SIGCHLD, but the system creates zombies, so you can nd out about child status by calling wait. If, however, you change the default to explicitly ignore the signal, the system ignores SIGCHLD and SIGCLD, but it also no longer creates zombie processes. If you set the disposition of SIGCHLD and SIGCLD to ignore, but you call wait anyway, it waits until all child processes have terminated, and then returns -1 (error), with errno set to ECHILD. You can achieve the same effect with sigaction by specifying the SA_NOCLDWAIT ag in sa_flags. There is no way to achieve this behaviour in other versions of UNIX: if you nd your ported program is collecting zombies (which you will see with the ps program), it might be that the program uses this feature to avoid having to call wait. If you experience this problem, you can solve it by adding a signal handler for SIGCLD that just calls wait and returns. The signal number for SIGCLD is the same as for SIGCHLD. The semantics depend on how you enable it: if you enable it with signal, you get SIGCLD semantics (and unreliable signals), and if you enable it with sigaction you get SIGCHLD and reliable signals. Dont rely on this, however. Some versions of System V have special coding to ensure that a separate SIGCLD signal is delivered for each child that dies.

Interrupted system calls

Traditional UNIX kernels differentiate between fast and slow system calls. Fast calls are handled directly by the kernel, while slow calls require the cooperation of other processes or device drivers. While the call is being executed, the calling process is suspended. If a signal for a process occurs while the process is suspended, the behaviour depends both on whether the call is fast or slow, and on the signal implementation. On traditional systems, if the priority is numerically less than (of a higher priority than) the constant PZERO, the signal is slow and remains pending until the priority rises above PZERO. Otherwise it is fast, and the system call is interrupted. Typically, this means that disk and network operations are not interrupted, since they run at a priority below PZERO, whereas terminal and serial line operations can be interrupted. Some newer systems treat the relationship between priority and delivering signals more exibly.

5 February 2005 02:09

Chapter 13: Signals

187

In the Seventh Edition, if a system call was interrupted, it returned an error, and errno was sent to EINTR. It was up to the process to decide whether to repeat the call or not. This added a signicant coding overhead to just about every program; the result was that programs usually did not provide for interrupted system calls, and died when it happened. Later signal implementations improved on this state of affairs: In 4.2BSD, signals automatically restarted the system calls ioctl, read, readv, wait, waitpid, write and writev. In 4.3BSD, the 4.2BSD signal implementation was modied so that the user could elect not to restart specic system calls after interruption. The default remained to restart the system call. In POSIX.1, when you call sigaction you can specify that system calls interrupted by specic signals should be restarted. This is done with the SA_RESTART ag in the eld sa_flags. If this ag is not set, the calls will not be restarted. SunOS 4 does not have SA_RESTART, but it has SA_INTERRUPT instead, which is effectively the reverse of SA_RESTART: system calls will be restarted unless SA_INTERRUPT is set,

On modern systems, the action taken depends on the system calls you have used and the system you are using: With System V, you have the choice of no restart (unreliable signal or System V sigset and friends) or POSIX.1 selective restart based on the signal (SA_RESTART with sigaction). With BSD, you have the choice of no restart (reliable signal based on sigaction), default restart based on system calls (sigvec and friends) or again the POSIX.1 selective restart based on the signal (SA_RESTART with sigaction).

Calling functions from signal handlers

By denition, signals interrupt the normal ow of program execution. This can cause problems if they call a function that has already been invoked, and which has saved some local state. The function needs to be written specially to avoid such problemsit should block either all signals during execution, or, preferably, it should be written reentrantly. Either solution is difcult, and typically system libraries do not support this kind of reentrancy. On the other hand, theres not much you can do without calling some library routine. POSIX.1 denes safe routines that you can call from a signal handler. They are:
_exit cfsetispeed close execve geteuid access cfsetospeed creat fcntl getgid alarm chdir dup fork getgroups cfgetispeed chmod dup2 fstat getpgrp cfgetospeed chown execle getegid getpid

5 February 2005 02:09

188 getppid mkdir pipe setpgid sigdelset sigprocmask tcdrain tcsendbreak umask waitpid getuid mkfifo read setsid sigemptyset sigsuspend tcflow tcsetattr uname write kill open rename setuid sigfillset sleep tcflush tcsetpgrp unlink link pathconf rmdir sigaction sigismember stat tcgetattr time utime lseek pause setgid sigaddset sigpending sysconf tcgetpgrp times wait

In addition, System V.4 allows abort, exit, longjmp, and signal.

Current signal implementations

In this section, well look at the differences between individual signal implementations. Well concentrate on what you need to do to convert from one to another. If you do need to convert signal code, you should use the POSIX.1 signal implementation whenever practical.

Seventh Edition signal function

The Seventh Edition provided only one signal function, signal, which is the granddaddy of them all. All systems supply signal, though on some systems, such as newer BSD systems, it is a library function that calls sigaction. This also means that you cant rely on specic semantics if you use signalavoid it if at all possible. Older UNIX systems (specically, those that did not expect function prototypes to be used) implicitly dened the return type of signal to be an int. This does not change the meaning of the return value, but it can confuse more pedantic compilers. About the only system still on the market that returns an int from signal is XENIX.

BSD signal functions

The BSD signal functions were the rst attempt at reliable signals, and they form the basis of the POSIX.1 implementation. All modern systems offer the POSIX.1 implementation as well, and on many BSD systems the functions described in this section are just an interface to the POSIX.1 functions.

Signal sets
A central difference between the Seventh Edition and System V implementations, on the one side, and the BSD and POSIX.1 implementations, on the other side, is the way signals can be specied. The Seventh Edition functions treat individual signals, which are specied by their number. The BSD routines introduced the concept of the signal set, a bit map of type sigset_t, that species any number of signals, as illustrated in Figure 13-1:

5 February 2005 02:09

Chapter 13: Signals

189

31 1

30 1

29 0

11

10 0

9 0

...

...

SIGUSR2

SIGUSR1

SIGINFO

SIGSEGV

SIGBUS

SIGKILL

SIGHUP

(none)

Figure 131. BSD and POSIX.1 signal sets

For each signal, if the corresponding bit in the bit map is set, the signal is said to be included in the set. In this example, the signals specied are SIGUSR2, SIGUSR1 and SIGHUP. This method enables any number of signals to be specied as the parameter of one call. The kernel maintains two special signal sets for each process: the signal mask and the pending signal set. The signal mask species which signals should currently not be delivered to the process these signals are said to be blocked. This does not mean that they will be ignored: if a signal occurs while it is blocked, the kernel notes that it has occurred and sets its bit in the pending signal set. When a subesequent call to sigsetmask resets the bit for this signal in the signal mask, the kernel delivers the signal to the process and clears the bit in the pending signal set.

sigsetmask
sigsetmask sets the process signal mask:
#include <sys/signal.h> int sigsetmask (int mask);

sigsetmask can be dened in terms of the POSIX.1 function sigprocmask using the SIG_SETMASK ag see page 194 for more details.

sigblock
sigblock modies the process signal mask. Unlike sigsetmask, it performs a logical OR of the specied mask with the current signal mask, so it can only block signals and not enable them.
#include <sys/signal.h> int sigblock (int mask);

sigblock can be dened in terms of the POSIX.1 function sigprocmask using the SIG_BLOCK ag see page 194 for more details.

5 February 2005 02:09

190

sigvec
sigvec corresponds to the Seventh Edition signal: it sets the disposition of a signal. In addition, it can block other signals during the processing of a signal.
#include <signal.h> ... in signal.h is the definition struct sigvec { void (*sv_handler) (); sigset_t sv_mask; int sv_flags; }; sigvec (int signum, struct sigvec *vec, struct sigvec *ovec);

signum is the signal whose disposition is to be changed. vec species the new disposition of the signal, and the function returns the old disposition to ovec. If vec->sv_mask is non-zero, it species the signals to block while the signal handler is running. This is logically ored with the current signal mask, so it works like an implicit sigblock on entering the signal handler. On exit from the signal handler, the kernel reinstates the previous signal mask. flags can consist of: SV_ONSTACK species to take the signal on alternate signal stack, if one has been dened. SV_INTERRUPT species that system calls should not be restarted after the signal handler has completed.

sigvec is almost identical to the POSIX.1 function sigaction described on page 193 only the names of the sigvec structure and its members are different. Note, however, that the ag SV_INTERRUPT has the opposite meaning from the POSIX.1 ag SA_RESTART, which frequently has the same numeric value.

sigpause
sigpause combines the functionality of sigmask and pause: it rst sets the signal mask and then calls pause to wait for a signal to occur.
#include <sys/signal.h> int sigpause (sigset_t sigmask);

Typical use of BSD signal functions

Most signal coding consists of initialization. Typical programs set the disposition of the signals in which they are interested during program initialization, and dont change them much after that. For example, with BSD signals you might see code like that in Example 13-2.
Example 132:

5 February 2005 02:09

Chapter 13: Signals Example 132: (continued)

struct struct sigvec sigvec sigvec sigvec hupvec = {&hup_handler, 0, 0}; /* disposition of SIGHUP */ sigvec iovec = {&io_handler, 1 << SIGHUP, 0}; /* disposition of SIGIO */ (SIGHUP, &hupvec, NULL); /* instate handlers for SIGHUP, */ (SIGIO, &iovec, NULL); /* SIGIO, */ (SIGURG, &iovec, NULL); /* and SIGURG */

191

Occasionally a process will use sigpause, usually to wait for I/O. In Example 13-3, it blocks the signals SIGINT and SIGQUIT:
Example 133:
sigpause ((1 << SIGINT) | (1 << SIGQUIT)); /* wait for a signal */

System V signal functions

The following signal functions were implemented in System V and are effectively obsolete: the POSIX.1 functions have replaced them even in System V.3. The syntax of the function calls is more like the Seventh Edition than POSIX.1. In particular, they do not support the concept of a signal set. If you do nd it necessary to replace System V signals with POSIX.1 signals, there is considerable scope for simplication by merging multiple System V calls (one per signal) into a single POSIX.1 call.

sigset
sigset is the System V reliable equivalent of signal:
#include <signal.h> void (*sigset (int sig, void (*disp) (int))) (int);

Unlike signal, the signal is not disabled when the signal handler is executing instead it is blocked until the signal handler terminates.

sighold
sighold blocks the delivery of signal sig by setting the corresponding bit in the process signal mask. Semantically this corresponds to the POSIX.1 function sigprocmask with the SIG_BLOCK ag, but it can block only one signal per call.
#include <signal.h> int sighold (int sig);

sigrelse
sigrelse allows the delivery of signal sig by resetting the corresponding bit in the process signal mask. Semantically this corresponds to the POSIX.1 function sigprocmask with the SIG_UNBLOCK ag, but it can release only one signal per call.

5 February 2005 02:09

192
#include <signal.h> int sigrelse (int sig);

sigignore
sigignore sets the disposition of signal sig to SIG_IGNthe kernel ignores the signal.
#include <signal.h> int sigignore (int sig);

sigpause
#include <signal.h> int sigpause (int sig);

sigpause enables the delivery of signal sig and then waits for delivery of any signal. CAUTION This is not the same as the BSD function sigpause described on page 190. BSD sigpause takes a signal mask as an argument, System V sigpause takes a single signal number. In addition, BSD sigpause only resets the mask temporarilyuntil the function return whereas System V sigpause leaves it in this condition.

Example of System V signal functions

On page 190, we looked at what typical BSD code might look like. The System V equivalent of this program might perform the initialization in Example 13-4. System V doesnt supply the functionality associated with SIGIO and SIGURGit uses SIGPOLL instead. See Chapter 14, File systems, pages 209 and 225, for more details of SIGIO and SIGPOLL respectively.
Example 134:
sigset (SIGHUP, &hup_handler); sigset (SIGPOLL, &io_handler); /* instate handlers for SIGHUP */ /* and SIGPOLL */

System V sigpause has a different syntax, so we need to set the signal mask explicitly with calls to sighold, and also to release them explicitly with sigrelse
Example 135:
sighold (SIGINT); sighold (SIGQUIT); sigpause (0); sigrelse (SIGINT); sigrelse (SIGQUIT); /* /* /* /* /* block SIGINT */ and SIGQUIT */ wait for something to happen */ unblock SIGINT */ and SIGQUIT */

POSIX.1 signal functions

All modern UNIX implementations claim to support POSIX.1 signals. These are the functions to use if you need to rewrite signal code. They are similar enough to the BSD functions to be confusing. In particular, the BSD functions pass signal masks as longs, whereas the POSIX.1 functions pass pointers to the signal maskthis enables the number of signals to exceed the number of bits in a long.

5 February 2005 02:09

Chapter 13: Signals

193

sigaction
sigaction is the POSIX.1 equivalent of signal. It species the disposition of a signal. In addition, it can specify a mask of signals to be blocked during the processing of a signal, and a number of ags whose meaning varies signicantly from system to system.
#include <signal.h> struct sigaction { void (*sa_handler)(); sigset_t sa_mask; int sa_flags; };

/* handler */ /* signals to block during processing */

void sigaction (int sig, const struct sigaction *act, struct sigaction *oact);

signum is the signal whose disposition is to be changed. act species the new disposition of the signal, and the function returns the old disposition to oact. If act->sa_mask is non-zero, it species which signals to block while the signal handler is running. This is logically ored with the current signal mask, so it works like an implicit sigblock on entering the signal handler. Heres an overview of the ags:
Table 132: sigaction ags Parameter SA_ONSTACK

supported by BSD, System V

meaning Take the signal on the alternate signal stack, if one has been dened. POSIX.1 does not dene the concept of an alternate signal stacksee page 196 for more details. Linux plans similar functionality with the SA_STACK ag, but at the time of writing it has not been implemented. Reset the disposition of this signal to SIG_DFL when the handler is entered (simulating Seventh Edition semantics). This is the same as the Linux SA_ONESHOT ag. Reset the disposition of this signal to SIG_DFL when the handler is entered (simulating Seventh Edition semantics). This is the same as the System V SA_RESETHAND ag.

SA_RESETHAND

System V

SA_ONESHOT

Linux

5 February 2005 02:09

194 Table 132: sigaction ags (continued) Parameter supported meaning SA_RESTART

by BSD, Linux, System V System V System V

Restart system calls after the signal handler has completed (see page 186). Provide additional parameters to signal handler (see page 183). Dont block this signal while its signal handler is active. This means that the signal handler can be called from a function which it calls, and thus needs to be reentrant. Dont create zombie children on SIGCLD (see page 186). Dont generate SIGCHLD when a child stops, only when it terminates. Disable the signal mask (allow all signals) during the execution of the signal handler. Disable automatic restart of signals. This corresponds to the SunOS 4 ag SV_INTERRUPT to sigvec (see page 190). Currently not implemented.

SA_SIGINFO SA_NODEFER

SA_NOCLDWAIT SA_NOCLDSTOP SA_NOMASK SA_INTERRUPT

System V Linux, System V Linux Linux

sigprocmask
sigprocmask manipulates the process signal mask. It includes functional modes that correspond to both of the BSD functions sigblock and sigsetmask:
#include <signal.h> int sigprocmask (int how, const sigset_t *set, sigset_t *oset)

The parameter how determines how the mask is to be manipulated. It can have the following values:
Table 133: sigprocmask functional modes Parameter SIG_BLOCK SIG_UNBLOCK SIG_SETMASK

meaning Create a new signal mask by logically oring the current mask with the specied set. Reset the bits in the current signal mask specied in set. Replace the current signal mask by set.

5 February 2005 02:09

Chapter 13: Signals

195

sigpending
#include <signal.h> int sigpending (sigset_t *set);

sigpending returns the pending signal mask to set. These are the signals pending delivery but currently blocked, which will be delivered as soon as the signal mask allows. The return value is an error indication and not the signal mask. This function does not have an equivalent in any other signal implementation

sigsuspend
#include <sys/signal.h> int sigsuspend (const sigset_t *sigmask);

sigsuspend temporarily sets the process signal mask to sigmask, and then waits for a signal. When the signal is received, the previous signal mask is restored on exit from sigsuspend. It always returns -1 (error), with errno set to EINTR (interrupted system call).

Example of POSIX.1 signal functions

On page 190, we looked at a simple example of signal setup, and on page 192 we changed it for System V. Changing it from BSD to POSIX.1 is mainly a matter of changing the names. We change the calls to sigvec to sigaction, and their parameters are now also of type struct sigaction instead of struct sigvec. Unfortunately, there is a problem with this example: POSIX.1 does not dene any of the I/O signals to which this example refers. This is not as bad as it sounds, since there are no pure POSIX.1 systems, and all systems offer either SIGIO/SIGURG or SIGPOLL. In Example 13-6, well stick with the BSD signals SIGIO and SIGURG:
Example 136:
struct sigaction hupvec = {&hup_handler, 0, 0}; /* disposition of SIGHUP */ struct sigaction iovec = {&io_handler, 1 << SIGHUP, 0}; /* disposition of SIGIO */ sigaction (SIGHUP, &hupvec, NULL); /* instate handlers for SIGHUP, */ sigaction (SIGIO, &iovec, NULL); /* SIGIO, */ sigaction (SIGURG, &iovec, NULL); /* and SIGURG */ sigset_t blockmask; /* create a mask */ sigemptyset (&blockmask); /* clear signal mask */ sigaddset (&blockmask, SIGINT); /* add SIGINT to the mask */ sigaddset (&blockmask, SIGQUIT); /* add SIGQUIT to the mask */

Example 13-7 shows the corresponding call to sigsuspend:

Example 137:
sigsuspend (&blockmask); /* let the action begin */

Well look at sigemptyset and sigaddset in the next section. Its unfortunate that this part of the initialization looks so complicatedits just part of the explicit programming style that POSIX.1 desires. On most systems, you could get the same effect without the calls to sigemptyset and sigaddset by just dening

5 February 2005 02:09

196
int blockmask = (1 << SIGINT) | (1 << SIGQUIT); sigpause ((sigset_t *) &blockmask); /* let the action begin */

The only problem with this approach (and its a showstopper) is that its not portable: on a different system, sigset_t might not map to int.

Signals under Linux

Linux signals are an implementation of POSIX.1 signals, and we discussed some of the details in the previous section. In addition, its good to know that: For compatibility, SIGIOT is dened as SIGABRT. POSIX.1 does not dene SIGIOT. As we saw, POSIX.1 does not supply the signals SIOPOLL, SIGIO and SIGURG. Linux does, but they it maps all three signals to the same numerical value. If you really want to, you can simulate unreliable signals under Linux with sigaction and the SA_ONESHOT ag.

Other signal-related functions

A signicant advantage of the BSD and POSIX.1 signal functions over the Seventh Edition and System V versions is that they have signal set parameters. The down side of signal sets is that you need to calculate the values of the bits. The following functions are intended to make manipulating these structures easier. They are usually implemented as macros: sigemptyset (sigset_t *set) sets set to the empty signal setin other words, it excludes all signals. sigfillset (sigset_t *set) sets all valid signals in set. sigaddset (sigset_t *set, int signum) adds signal signum to set. sigdelset (sigset_t *set, int signum) removes signal signum from set. sigismember (sigset_t *set, int signum) returns 1 if signum is set in set and 0 otherwise.

sigstack and sigaltstack

As we have already discussed, a signal is like a forced function call. On modern processors with stack-oriented hardware, the call uses stack space. In some cases, a signal that arrives at the wrong time could cause a stack overow. To avoid this problem, both System V and BSD (but not POSIX.1) allow you to dene a specic signal stack. On receipt of a signal, the stack is switched to the alternate stack, and on return the original stack is reinstated. This can also occasionally be of interest in debugging: if a program gets a signal because of a reference beyond the top of the stack, its not much help if the signal destroys the evidence. BSD supplies the sigstack system call:

5 February 2005 02:09

Chapter 13: Signals

#include <sys/signal.h> struct sigstack { caddr_t ss_sp; int ss_onstack;

197

/* Stack address */ /* Flag, set if currently * executing on this stack */

}; int sigstack (const struct sigstack *ss, struct sigstack *oss);

ss may be NULL. If it is not, the process signal stack is set to ss->ss_sp. ss->ss_onstack tells sigstack whether the process is currently executing on the stack.

oss may also be NULL. If it is not, information about the current signal stack is returned to it. System V supplies the function sigaltstack:
#include <signal.h> typedef struct { char *ss_sp; int ss_size; int ss_flags; } stack_t; int sigaltstack (const stack_t *ss, stack_t

/* Stack address */ /* Stack size */ /* Flags, see below */

*oss);

ss may be NULL. If it is not, the process signal stack is set to ss->ss_sp, and its size is set to ss->ss_size. oss may also be NULL. If it is not, information about the current signal stack is returned to it. The structure element ss_flags may contain the following ags: SS_DISABLE species that the alternate stack is to be disabled. ss_sp and ss_size are ignored. This ag is also returned in oss when the alternate stack is disabled. SS_ONSTACK (returned) indicates that the process is currently executing on the alternate stack. If this is the case, a modication of the stack is not possible.

setjmp and longjmp

When you return from a function, C language syntax does not give you a choice of where to return to: you return to the instruction after the call. Occasionally, deep in a series of nested function calls, you will discover you need to return several levels down the stackeffectively, you want to perform multiple returns. Standard structured programming techniques do not handle this requirement well, and you cant just perform a goto to the location, because that would leave the stack in a mess. The library functions setjmp and longjmp provide this non-local return.

5 February 2005 02:09

198

What does this have to do with signals? Nothing, really, except that the receipt of a signal is one of the most common reasons to want to perform a non-local return: a signal can interrupt processing anywhere where the process signal mask allows it. In many cases, the result of the signal processing is not related to the processing that was interrupted, and it may be necessary to abort the processing and perform a non-local return. For example, if you are redisplaying data in an X window and the size of the window changes, you will get a SIGWINCH signal. This requires a complete recalculation of what needs to be displayed, so there is no point in continuing the current redisplay operation. Non-local returns are implemented with the functions setjmp, longjmp, and friends. setjmp saves the process context and longjmp restores itin other words, it returns to the point in the program where setjmp was called. Unlike a normal function return, a longjmp return may involve discarding a signicant part of the stack. There are a number of related functions:
#include <setjmp.h> int setjmp (jmp_buf env); void longjmp (jmp_buf env, int val); int _setjmp (jmp_buf env); void _longjmp (jmp_buf env, int val); void longjmperror (void); int sigsetjmp (sigjmp_buf env, int savemask); void siglongjmp (sigjmp_buf env, int val);

The denitions of jmp_buf and sigjmp_buf are less than illuminating: they are just dened as an array of ints long enough to contain the information that the system saves. In fact, they contain the contents of the registers that dene the process context stack pointer, frame pointer, program counter, and usually a number of other registers. From the user point of view, setjmp is unusual in that it can return more often than you call it. Initially, you call setjmp and it returns the value 0. If it returns again, its because the program called longjmp, and this time it returns the value parameter passed to longjmp, which normally should not be 0. The caller can then use this value to determine whether this is a direct return from setjmp, or whether it returned via longjmp:
int return_code = setjmp (env); if (return_code) { /* non-0 return code: return from longjmp */ printf ("longjmp returned %d\n", return_code); }

These functions are confusing enough in their own right, but they also have less obvious features: It doesnt make any sense for longjmp to return 0, and System V.4 longjmp will never return 0, even if you tell it toit will return 1 instead. BSD longjmp will return whatever you tell it to. The setjmp functions save information about the state of the function that called them. Once this function returns, this information is no longer valid. For example, the

5 February 2005 02:09

Chapter 13: Signals

199

following code will not work:

jmp_buf env; int mysetjmp () { int a = 0; if (a = setjmp (env)) printf ("Bombed out\n"); return a; } foo () { ... mysetjmp (); ... } /* save area for setjmp */

/* catch bad errors */

The return instruction from mysetjmp to foo frees its local environment. The memory which it occupies, and which the call to setjump saved, will be overwritten by the next function call, so a longjmp cannot restore it. BSD attempts to determine whether the parameter env to the longjmp functions is invalid (such as in the example above). If it detects such an error, it will call longjmperror, which is intended to inform that the longjmp has failed. If longjmperror returns, the process is aborted. If longjmp does not recognize the error, or if the system is not BSD, the resulting process state is indeterminate. To quote the System V.4 man page: If longjmp is called even though env was never primed by a call to setjmp, or when the last such call was in a function that has since returned, absolute chaos is guaranteed. In fact, the system will probably generate a SIGSEGV or a SIGBUS, but the core dump will probably show nothing recognizable. When longjmp returns to the calling function, automatic variables reect the last modications made to them in the function. For example:
int foo () { int a = 3; if (setjmp (env)) { printf ("a: %d\n", a); return a; } a = 2; longjmp (env, 4); }

At the point where longjmp is called, the variable a has the value 2, so this function will print a:2.

5 February 2005 02:09

200

When longjmp returns to the calling function, register variables will normally have the values they had at the time of the call to setjmp, since they have been saved in the jump buffer. Since optimizers may reassign automatic variables to registers, this can have confusing results. If you compile the example above with gcc and optimize it, it will print a: 3. This is clearly an unsuitable situation: the solution is to declare a to be volatile (see Chapter 20, Compilers, page 340 for more information). If we do this, a will always have the value 2 after the longjmp. BSD setjmp includes the signal mask in the state information it saves, but System V.4 setjmp does not save the signal mask. If you want to simulate System V.4 semantics under BSD, you need to use _setjmp and _longjmp, which do not save the signal mask. In either system, you can use sigsetjmp, which saves the signal mask only if save is non-zero. Except for the type of its rst parameter, the corresponding siglongjmp is used in exactly the same manner as longjmp. The functions must be paired correctly: if you _setjmp, you must _longjmp, and if you setjmp you must longjmp.

kill
kill is one of the most badly named system calls in the UNIX system. Its function is to send a signal:
#include <signal.h> int kill (pid_t pid, int sig);

Normally, pid is the process ID of the process that should receive the signal sig. There are a couple of additional tricks, however: If pid is 0, the kernel sends sig to all processes whose process group ID is the same as the group ID of the calling process. If pid is -1, most implementations broadcast the signal to all user processes if the signal is sent by root. Otherwise the signal is sent to all processes with the same effective user ID. BSD does not broadcast the signal to the calling process, System V does. POSIX.1 does not dene this case. If pid is < -1, System V and BSD broadcast the signal to all processes whose process group ID is abs (pid) (abs is the absolute value function). Again, non-root processes are limited to sending signals to processes with the same effective user ID. BSD can also perform this function with the call killpg.

Another frequent use of kill is to check whether a process exists: kill (pid, 0) will not actually send a signal, but it will return success if the process exists and an error indication otherwise.

5 February 2005 02:09

Chapter 13: Signals

201

killpg
killpg broadcasts a signal to all processes whose process group ID is abs (pid). It is supplied with BSD systems:
#include <sys/signal.h> int killpg (pid_t pgrp, int sig);

This function sends the signal to the process group of the specied process, assuming that you have the same effective user ID as the recipient process, or you are super-user. You can use pid 0 to indicate your own process group. If you dont have this function, you can possibly replace it with kill(-pgid)see the section on kill above.

raise
raise is an ANSI C function that enables a process to send a signal to itself. It is dened as
int raise (int signum);

Older systems dont have raise. You can fake it in terms of kill and getpid:
kill (getpid (), signum);

sys_siglist and psignal

At the name implies, sys_siglist is a list and not a function. More exactly, it is an array of signal names, indexed by signal number, and is typically supplied with BSD-derived systems. For example,
printf ("Signal %d (%s)\n", SIGSEGV, sys_siglist [SIGSEGV]);

returns
Signal 11 (Segmentation fault)

Some systems supply the function psignal instead of sys_siglist. It prints the text corresponding to a signal. You can get almost the same effect as the printf above by writing
char msg [80]; sprintf (msg, "Signal %d", SIGSEGV); psignal (SIGSEGV, msg);

This gives the output:

Signal 11: Segmentation fault

5 February 2005 02:09

File systems
UNIX owes much of its success to the simplicity and exibility of the facilities it offers for le handling, generally called the le system. This term can have two different meanings: 1. It can be a part of a disk or oppy which can be accessed as a collection of les. It includes regular les and directories. A oppy is usually a single le system, whereas a hard disk can be partitioned into several le systems and possibly also non-le system parts, such as swap space and bad track areas. It can be the software in the kernel which accesses the le systems above.

2.

UNIX has a single le hierarchy, unlike MS-DOS, which uses a separate letter for each le system (A: and B: for oppies, C: to Z: for local and network accessible disks). MS-DOS determines the drive letter for the le systems at boot time, whereas UNIX only determines the location of the root le system / at boot time. You add the other le systems to the directory tree by mounting them:
$ mount /dev/usr /usr

This mounts the le system on the disk partition /dev/usr onto the directory /usr, so if the root directory of /dev/usr contains a le called foo, after mounting you can access it as /usr/foo. Anything useful is bound to attract people who want to make it more useful, so it should come as no surprise that a large number of improvements have been made to the le system in the course of time. In the rest of this chapter, well look at the following aspects in more detail: File systems introduced since the Seventh Edition. Differences in function calls, starting on page 206. Non-blocking I/O, starting on page 220. File locking, starting on page 226. Memory-mapped les, starting on page 232.

203

5 February 2005 02:09

204

File system structures

The original Seventh Edition le system isat least in spiritthe basis for all current le system implementations. All UNIX le systems differ in one important point from almost all non-UNIX le systems: At the lowest level, the le system refers to les by numbers, so-called inodes. These are in fact indices in the inode table, a part of the le system reserved for describing les. At a higher level, the directory system enables a le to be referred to by a name. This relationship between a name and an inode is called a link, and it enables a single le to have multiple names.

One consequence of this scheme is that it is normally not possible to determine the le name of an open le. The Seventh Edition le system is no longer in use in modern systems, though the System V le system is quite similar. Since the Seventh Edition, a number of new le systems have addressed weaknesses of the old le system: New le types were introduced, such as symbolic links, fos and sockets. The performance was improved. The reliability was increased signicantly. The length of the le names was increased.

Well look briey at some of the differences in the next few sections.

The Berkeley Fast File System

The rst alternative le system to appear was the Berkeley Fast File System, (FFS), now called the Unix File System (ufs).* It is described in detail in A Fast File System for UNIX, by Kirk McKusick, Bill Joy, Sam Lefer and Robert Fabry, and The Design and the Implementation of the 4.3BSD UNIX Operating System by Sam Lefer, Kirk McKusick, Mike Karels, and John Quarterman. Its main purpose was to increase speed and storage efciency. Compared to the Seventh Edition le system, the following differences are relevant to porting software: The maximum le name size was increased from 14 to 255 characters. The size of the inode number was increased from 16 to 32 bits, thus allowing an effectively unlimited number of les. Symbolic links were introduced.

A symbolic link differs from a normal link in that it points to another le name, and not an inode number.
* Dont confuse the Berkeley FFS with SCOs afs, which is sometimes referred to as a Fast File System. In fact, afs is very similar to s5fs, though later versions have symbolic links and longer le names.

5 February 2005 02:09

Chapter 14: File systems

205

Symbolic links
A symbolic link is a le whose complete contents are the name of another le. To access via a symbolic link, you rst need to nd the directory entry to which it is pointing, then resolve the link to the inode. By contrast, a traditional link (sometimes called hard link) links a le name to an inode. Several names can point to the same inode, but it only takes one step to nd the le. This seemingly minor difference has a number of consequences: A denite relationship exists between the original le and the symbolic link. In a normal link, each of the le names have the same relationship to the inode; in a symbolic link, the symbolic link name refers to the main le name. This difference is particularly obvious if you remove the original le: with a normal link, the other name still works perfectly. With a symbolic link, you lose the le. Theres nothing to stop a symbolic link from pointing to another symbolic linkin fact, its quite common, and is moderately useful. It also opens the possibility of looping: if the second symbolic link points back to the rst, the system will give up after a few iterations with the error code ELOOP. Symbolic links have two le permissions. In practice, the permission of the link itself is of little consequencenormally it is set to allow reading, writing and execution for all users (on an ls -l listing you see lrwxrwxrwx). The permission that counts is still the permission of the original le. Symbolic links allow links to different le systems, even (via NFS) to a le system on a different machine. This is particularly useful when using read-only media, such as CDROMs. See Chapter 3, Care and feeding of source trees, page 39, for some examples. Symbolic links open up a whole new area of possible errors. Its possible for a symbolic link to point to a le that doesnt exist, so you cant access the le, even if you have a name and the correct permissions.

Other file systems

Other le systems have emerged since ufs, including: The System V le system, s5fs, a minor evolution of the Seventh Edition File system with some performance and stability modications, and without multiplexed les. Even in System V, ufs has replaced it. For all practical purposes, you can consider it to be obsolete. The Veritas File System, vxfs and the Veritas Journalling File system, vjfs. From the point of view of porting, they are effectively compatible with ufs. The Network File System, NFS,* a method of sharing le systems across networks. It allows a system to mount le systems connected to a different machine. NFS runs on

* People just dont seem to be able to agree whether to write le system names in upper case (as bets an abbreviation), or in lower case (the way most mount commands want to see them). It appears that NFS is written in upper case more frequently than the other names.

5 February 2005 02:09

206

just about any system, including System V.3 and DOS, but unfortunately not XENIX. It can offer a partial escape from the 14 character le limit, no symlinks syndrome. It is reasonably transparent, but unfortunately does not support device les. Remote File Sharing, rfs. This is AT&Ts answer to NFS. Although it has a number of advantages over NFS, it is not widely used.

Along with new le systems, new le types have evolved. We have already looked at symbolic links, which we can think of as a new le type. Others include FIFOs (First In First Out) and sockets, means of interprocess communications that we looked at in Chapter 12, Kernel dependencies. In practice, you run into problems only when you port software developed under ufs, vjfs or vxfs to a s5fs system. If you can, you should change your le system. If you cant do that, here are some of the things that could give you headaches: File name length. Theres very little you can do about this: if the le names are longer than your kernel can understand, you have to change them. There are some subtle problems here: some 14-character le systems accept longer names and just silently truncate them, others, notably SCO, signal an error. It should be fairly evident what your le system does when you try to do it. If your system has the pathconf system call, you can also interrogate this programmatically (see page 212). Lack of symbolic links is another big problem. You may need far-reaching source changes to get around this problem, which could bite you early on in the port: you may have an archive containing symbolic links, or the conguration routines might try to create them.

Another, more subtle difference is that BSD and System V do not agree on the question of group ownership. In particular, when creating a le, the group ownership may be that of the directory, or it may be that of the process that creates the le. BSD always gives the le the group of the directory; in System V.4, it is the group of the process, unless the set group ID bit is set in the directory permissions, in which case the le will belong to the same group as the directory.

Function calls
The Seventh Edition left a surprising amount of functionality up to the system library. For example, the kernel supplied no method to create a directory or rename a le. The methods that were used to make up for these deciencies were not always reliable, and in the course of the time these functions have been implemented as system calls. Current systems offer the following functions, some of them system calls:

chsize
chsize changes the end of le of an open le.

5 February 2005 02:09

Chapter 14: File systems

int chsize (int fd, long size);

207

It originated in XENIX and has been inherited by System V.3.2 and System V.4. It corresponds both in function and in parameters to the System V version of ftruncate: if the new end-of-le pointer is larger than the current end-of-le pointer, it will extend the le to the new size.

dup2
All systems offer the system call dup, which creates a copy of a le descriptor:
int dup (int oldd);

oldd is an open le descriptor; dup returns another le descriptor pointing to the same le. The problem with dup is that you dont have any control over the number you get back: its the numerically smallest le descriptor currently not in use. In many cases, you want a specic number. This is what dup2 does:
int dup2 (int oldd, int newd);

With newd you specify the number of the new descriptor. If its currently allocated, dup2 closes it rst. You can fake this with dup, but its painful. The F_DUPFD subfunction of fcntl does the same thing as dup2, so you can use it if it is available (see page 208). dup2 is available on nearly every UNIX system, including the Seventh Edition. Somehow some earlier versions of System V dont have it, howeverrecall that System V derived from the Sixth Edition, not the Seventh Edition. See Chapter 1, Introduction, page 4.

fchdir and friends

Various systems offer functions with names like fchdir, fchmod, fchown, and fchroot. These are effectively the same as the corresponding functions chdir, chmod, chown, and chroot, except they take the number of an open le instead of its name. For example:
#include <sys/stat.h> int chmod (const char *path, mode_t mode); int fchmod (int fd, mode_t mode);

You can replace them with a corresponding call to ch* if you know the name of the le associated with the le descriptor; otherwise you could be in trouble.

fcntl
All modern versions of UNIX supply a function called fcntl, which is rather like an ioctl for disk les:
#include <sys/fcntl.h> int fcntl (int fd, int cmd, union anything arg);

5 February 2005 02:09

208

Table 14-1 shows common command values.

Table 141: fcntl commands

Command F_DUPFD F_GETFD F_SETFD F_FREESP

System all all all SVR4, Solaris 2.X all all BSD SVR4 BSD

Meaning Duplicate a le descriptor, like dup. Return the lowest numbered descriptor that is higher than the int value arg. Get the close-on-exec ag associated with fd. Set the close-on-exec ag associated with fd. Free storage space associated with a section of the le fd. See the section on le locking on page 230 for more details. Get descriptor status ags (see below). Set descriptor status ags to arg. Get the process ID or the complement of the process group currently receiving SIGIO and SIGURG signals. Get the user ID of the owner of the le. This function is not documented for Solaris 2.X. Set the process or process group to receive SIGIO and SIGURG signals. If arg is negative, it is the complement of the process group. If it is positive, it is a process ID. Set the user ID of the owner of the le. This function is not documented for Solaris 2.X. Get le record lock information. See the section on locking on page 226, for more details. Set or clear a le record lock. Set or clear a le record lock, waiting if necessary until it becomes available. Check legality of le ag changes. Used by lockd to handle NFS locks. Used by lockd to handle NFS locks. Used by lockd to handle NFS locks.

F_GETFL F_SETFL F_GETOWN F_GETOWN F_SETOWN

F_SETOWN F_GETLK F_SETLK F_SETLKW F_CHKFL F_RSETLK F_RSETLKW F_RGETLK

SVR4 all all all SVR3 SVR4 SVR4 SVR4

As you can see from the table, arg is not always supplied, and when it is, its meaning and type vary depending on the call. A couple of these functions deserve closer examination:

5 February 2005 02:09

Chapter 14: File systems

209

F_SETFD and F_GETFD manipulate the close on exec ag. This is normally dened in sys/fcntl.h as 1. Many programs use the explicit constant 1, which is theoretically nonportable, but which works with current systems. By default, exec inherits open les to the new program. If the close on exec ag is set, exec automatically closes the le. F_GETOWN and F_SETOWN have very different meanings for BSD and System V.4. In BSD, they get and set the process ID that receives SIGIO and SIGURG signals; in System V.4, they get and set the le owner, which can also be done with stat or fstat. There is no direct equivalent to the BSD F_SETOWN and F_GETOWN in System V, since the underlying implementation of non-blocking I/O is different. Instead, you call ioctl with the I_SETSIG request see page 225 for more details. The request F_CHKFL is dened in the System V.3 header les, but it is not documented. F_GETFL and F_SETFL get and set the le status ags that were initally set by open. Table 14-2 shows the ags.

Table 142: fcntl le status ags

Flag O_NONBLOCK

System all

Meaning Do not block if the operation cannot be performed immediately. Instead, the read or write call returns -1 with errno set to EWOULDBLOCK. Append each write to the end of le. Send a SIGIO signal to the process group when I/O is possible. write waits for writes to complete before returning. Open for reading only. Open for reading and writing. Open for writing only.

O_APPEND O_ASYNC O_SYNC O_RDONLY O_RDWR O_WRONLY

all BSD System V System V System V System V

getdents and getdirentries

getdents (System V.4) and getdirentries (BSD) are marginally compatible system calls that read a directory entry in a le-system independent format. Both systems provide a header le /usr/include/sys/dirent.h, which denes a struct dirent, but unfortunately the structures are different. In System V, the structure and the call are:
struct dirent { ino_t d_ino; off_t d_off;

5 February 2005 02:09

210
unsigned short d_reclen; char d_name[1]; }; int getdents(int fd, struct dirent *buf, size_t nbyte);

getdirentries is the corresponding BSD system call:

struct dirent { unsigned long d_fileno; unsigned short d_reclen; unsigned short d_namlen; char d_name[MAXNAMLEN + 1]; };

/* /* /* /*

"file number" (inode number) of entry */ length of this record */ length of string in d_name */ name must be no longer than this */

int getdirentries(int fd, char *buf, int nbytes, long *basep);

Because of these compatibility problems, you dont normally use these system calls directly you use the library call readdir instead. See the description of readdir on page 213 for more information.

getdtablesize
Sometimes its important to know how many les a process is allowed to open. This depends heavily on the kernel implementation: some systems have a xed maximum number of les that can be opened, and may allow you to specify it as a conguration parameter when you build a kernel. Others allow an effectively unlimited number of les, but the kernel allocates space for les in groups of about 20. Evidently, the way you nd out about these limits depends greatly on the system you are running: On systems with a xed maximum, the constant NOFILE, usually dened in /usr/include/sys/param.h, species the number of les you can open. On systems with a congurable maximum, you will probably also nd the constant NOFILE, only you cant rely on it to be correct. On some systems that allocate resources for les in groups, the size of these groups may be dened in /usr/include/sys/ledesc.h as the value of the constant NDFILE. BSD systems offer the function getdtablesize (no parameters) that returns the maximum number of les you can open. Modern systems offer the getrlimit system call, which allows you to query a number of kernel limits. See Chapter 12, Kernel dependencies, page 169, for details of getrlimit.

5 February 2005 02:09

Chapter 14: File systems

211

ioctl
ioctl is a catchall function that performs functions that werent thought of ahead of time. Every system has its own warts on ioctl, and the most common problem with ioctl is a call with a request that the kernel doesnt understand. We cant go into detail about every ioctl function, but we do examine terminal driver ioctl calls in some depth in Chapter 15, Terminal drivers, starting on page 252.

lstat
lstat is a version of stat. It is identical to stat unless the pathname species a symbolic link. In this case, lstat returns information about the link itself, whereas stat returns information about the le to which the link points. BSD and System V.4 support it, and it should be available on any system that supports symbolic links.

ltrunc
ltrunc truncates an open le in the same way that ftruncate does, but the parameters are more reminiscent of lseek:
int ltrunc (int fd, off_t offset, int whence);

fd is the le descriptor. offset and whence specify the new end-of-le value: If whence is SEEK_SET, ltrunc sets the le size to offset. If whence is SEEK_CUR, ltrunc sets the le size to offset bytes beyond the current seek position. If whence is SEEK_END, ltrunc increases the le size by offset.

No modern mainstream system supports ltrunc. You can replace a call ltrunc (fd, offset, SEEK_SET) with ftruncate (fd, offset). If you have calls with SEEK_CUR and SEEK_END, you need to rst establish the corresponding offset with a call to lseek:
ftruncate (fd, lseek (fd, offset, SEEK_CUR)); or SEEK_END

mkdir and rmdir

Older versions of UNIX did not supply a separate system call to create a directory; they used mknod instead. Unfortunately, this meant that only the superuser could create directories. Newer versions supply mkdir and rmdir. The syntax is:
#include <sys/stat.h> int mkdir (const char *path, mode_t mode) #include <unistd.h> int rmdir (const char *path)

If your system does not have the mkdir system call, you can simulate it by invoking the

5 February 2005 02:09

212

mkdir utility with the library function system.

open
Since the Seventh Edition, open has acquired a few new ags. All modern versions of UNIX support most of them, but the following differ between versions: O_NDELAY is available only in earlier versions of System V. It applies to devices and FIFOs (see Chapter 12, Kernel dependencies, page 165, for more information on FIFOs) and species that both the call to open and subsequent I/O calls should return immediately without waiting for the operation to complete. A call to read returns 0 if no data is available, which is unfortunately also the value returned at end-of-le. If you dont have O_NDELAY, or if this ambiguity bugs you, use O_NONBLOCK. O_NONBLOCK species that both the call to open and subsequent I/O calls should return immediately without waiting for completion. Unlike O_NDELAY, a subsequent call to read returns -1 (error) if no data is available, and errno is set to EAGAIN. System V.4 and 4.4BSD have a ag, called O_SYNC in System V.4 and O_FSYNC in 4.4BSD, which species that each call to write write should write any buffered data to disk and update the inode. Control does not return to the program until these operations complete. If your system does not support this feature, you can probably just remove it, though you lose a little bit of security. To really do the Right Thing, you can include a call to fsync after every I/O.

pathconf and fpathconf

pathconf and fpathconf are POSIX.1 functions that get conguration information for a le or directory:
#include <unistd.h> long fpathconf (int fd, int name); long pathconf (const char *path, int name);

The parameter name is an int, not a name. Despite what it is called, it species the action to perform:
Table 143: pathconf actions name _PC_LINK_MAX _PC_MAX_CANON _PC_MAX_INPUT _PC_NAME_MAX

Function Return the maximum number of links that can be made to an inode. For terminals, return the maximum length of a formatted input line. For terminals, return the maximum length of an input line. For directories, return the maximum length of a le name.

5 February 2005 02:09

Chapter 14: File systems Table 143: pathconf actions (continued) name Function _PC_PATH_MAX _PC_PIPE_BUF _PC_CHOWN_RESTRICTED

213

_PC_NO_TRUNC

_PC_VDISABLE

Return the maximum length of a relative path name starting with this directory. For FIFOs, return the size of the pipe buffer. return TRUE if the chown system call may not be used on this le. If fd or path refer to a directory, then this information applies to all les in the directory. return TRUE if an attempt to create a le with a name longer than the maximum in this directory would fail with ENAMETOOLONG. For terminals, return TRUE if special character processing can be disabled.

read
The function read is substantially unchanged since the Seventh Edition, but note the comments about O_NDELAY and O_NONBLOCK in the section about open on page 212.

rename
Older versions of UNIX dont have a system call to rename a le: instead, they make a link and then delete the old le. This can cause problems if the process is stopped in the middle of the operation, and so the atomic rename function was introduced. If your system doesnt have it, you can still do it the old-fashioned way.

revoke
revoke is used in later BSD versions to close all le descriptors associated with a special le, even those opened by a different process. It is not available with System V.4. Typically, this call is used to disconnect serial lines. After a process has called revoke, a call to read on the device from any process returns an end-of-le indication, a call to close succeeds, and all other calls fail. Only the le owner and the super user may use this call.

readdir and friends

In the Seventh Edition, reading a directory was simple: directory entries were 16 bytes long and consisted of a 2-byte inode number and a 14 byte le name. This was dened in a struct direct:
struct direct { ino_t d_ino; char d_name[DIRSIZ];

5 February 2005 02:09

214
};

With the introduction of ufs, which supports names of up to 256 characters, it was no longer practical to reserve a xed-length eld for the le name, and it became more difcult to access directories. A family of directory access routines was introduced with 4.2BSD:
#include <sys/types.h> #include <dirent.h> DIR *opendir (const char *filename); struct dirent *readdir (DIR *dirp); long telldir (const DIR *dirp); void seekdir (DIR *dirp, long loc); void rewinddir (DIR *dirp); int closedir (DIR *dirp); int dirfd (DIR *dirp);

Along with the DIR type, there is a struct dirent that corresponds to the Seventh Edition struct direct. Unfortunately, System V denes struct dirent and DIR differently from the original BSD implementation. In BSD, it is
struct dirent { unsigned long d_fileno; unsigned short d_reclen; unsigned short d_namlen; char d_name [255 + 1]; }; /* directory entry */ /* /* /* /* file number of entry */ length of this record */ length of string in d_name */ maximum name length */

/* structure describing an open directory. */ typedef struct _dirdesc { int dd_fd; /* directory file descriptor */ long dd_loc; /* offset in current buffer */ long dd_size; /* amount of data from getdirentries */ char *dd_buf; /* data buffer */ int dd_len; /* size of data buffer */ long dd_seek; /* magic cookie from getdirentries */ } DIR;

System V denes
struct dirent { ino_t d_ino; off_t d_off; unsigned short d_reclen; char d_name [1]; }; typedef struct { int dd_fd; int dd_loc; int dd_size;

/* inode number of entry */ /* offset of directory entry */ /* length of this record */ /* name of file */

/* file descriptor */ /* offset in block */ /* amount of valid data */

5 February 2005 02:09

Chapter 14: File systems

char *dd_buf; } DIR; /* directory block */ /* stream data from opendir() */

215

There are a number of ugly incompatibilities here: The eld d_fileno in the BSD dirent struct is not a le descriptor, but an inode number. The System V name d_ino makes this fact clearer, but it introduces a name incompatiblity. A number of the BSD elds are missing in the System V structures. You can calculate dirent.d_namlen by subtracting the length of the other elds from dirent.d_reclen. For example, based on the System V dirent structure above:
d_namlen = dirent.d_reclen - sizeof (ino_t) /* length of the d_ino field */ - sizeof (d_off) /* length of the d_off field */ - sizeof (unsigned short); /* length of the d_reclen field */

System V.4 has two versions of these routines: a System V version and a BSD version. Many reports have claimed that the BSD version is broken, though its possible that the programmers were using the wrong header les. If you do run into trouble, you should make sure the header les match the avour of dirent and DIR that you have.

readv and writev

readv and writev perform a so-called scatter read and gather write. These functions are intended to write to a le a number of pieces of data spread in memory, or to read from a le to a number of places.
#include <unistd.h> #include <sys/types.h> #include <sys/uio.h> in sys/uio.h is the definition: struct iovec { caddr_t iov_base; int iov_len; }; int readv(int d, struct iovec *iov, int iovcnt); int writev (int d, struct iovec *iov, int iovcnt);

Each iovec element species an address and the number of bytes to transfer to or from it. The total number of bytes transferred would be the sum of the iov_len elds of all iovcnt elements. readv and writev are available only for BSD and System V.4 systemsif you dont have them, its relatively easy to fake them in terms of read or write. The reasons why these calls exist at all are: Some devices, such as tape drives, write a physical record for each call to write. This can result in a signicant drop in performance and tape capacity.

5 February 2005 02:09

216

For tape drives, the only alternative is to copy the data into one block before writing. This, too, impacts performance, though not nearly as much as writing smaller blocks. Even for devices that dont write a physical block per write, its faster to do it in the kernel with just a single function call: you dont have as many context switches.

statfs and statvfs

statfs or statvfs return information about a le system in a format referred to as a generic superblock. All current UNIX versions supply one or the other of these functions, but the information they return varies greatly. XENIX, System V.3, BSD, and BSD-derived SunOS operating systems supply statfs. System V.4 supplies statvfs. BSD systems dene statfs like this:
typedef quad fsid_t; #define MNAMELEN 32 /* length of buffer for returned name */

struct statfs { short f_type; /* type of filesystem (see below) */ short f_flags; /* copy of mount flags */ long f_fsize; /* fundamental file system block size */ long f_bsize; /* optimal transfer block size */ long f_blocks; /* total data blocks in file system */ long f_bfree; /* free blocks in fs */ long f_bavail; /* free blocks avail to non-superuser */ long f_files; /* total file nodes in file system */ long f_ffree; /* free file nodes in fs */ fsid_t f_fsid; /* file system id */ long f_spare[6]; /* spare for later */ char f_mntonname[MNAMELEN]; /* mount point */ char f_mntfromname[MNAMELEN]; /* mounted filesystem */ };

SunOS 4.1.3 denes them as:

#include <sys/vfs.h> typedef struct { long val[2]; } fsid_t; struct statfs { long f_type; long f_bsize; long f_blocks; long f_bfree; long f_bavail; long f_files; long f_ffree;

/* type of info, zero for now */ /* fundamental file system block size */ /* total blocks in file system */ /* free blocks */ /* free blocks available to non-super-user */ /* total file nodes in file system */ /* free file nodes in fs */

5 February 2005 02:09

Chapter 14: File systems

fsid_t f_fsid; long f_spare[7]; }; /* file system id */ /* spare for later */

217

System V.3 and XENIX dene:

struct { short long long long long long long char char }; statfs f_fstyp; f_bsize; f_frsize; f_blocks; f_bfree; f_files; f_ffree; f_fname[6]; f_fpack[6]; /* /* /* /* /* /* /* File system type */ Block size */ Fragment size (if supported) */ Total number of blocks on file system */ Total number of free blocks */ Total number of file nodes (inodes) */ Total number of free file nodes */ /* Volume name */ /* Pack name */

int statfs (const char *path, struct statfs *buf); int fstatfs (int fd, struct statfs *buf);

System V.4 and Solaris 2.X use statvfs, which is dened as

#include <sys/types.h> #include <sys/statvfs.h> struct statvfs { u_long f_bsize; /* preferred file system block size */ u_long f_frsize; /* fundamental filesystem block size */ u_long f_blocks; /* total # of blocks on file system */ u_long f_bfree; /* total # of free blocks */ u_long f_bavail; /* # of free blocks available */ u_long f_files; /* total # of file nodes (inodes) */ u_long f_ffree; /* total # of free file nodes */ u_long f_favail; /* # of inodes available */ u_long f_fsid; /* file system id (dev for now) */ char f_basetype [FSTYPSZ]; /* target fs type name */ u_long f_flag; /* bit mask of flags */ u_long f_namemax; /* maximum file name length */ char f_fstr [32]; /* file system specific string */ u_long f_filler [16]; /* reserved for future expansion */ }; #define ST_RDONLY 0x01 #define ST_NOSUID 0x02 #define ST_NOTRUNC 0x04 /* read-only file system */ /* does not support setuid/setgid */ /* does not truncate long file names */

int statvfs (const char *path, struct statvfs *buf); int fstatvfs (int fd, struct statvfs *buf);

Theres not much to say about these functions: if you have problems, hopefully this information will help you gure out what the author intended.

5 February 2005 02:09

218

symlink
symlink creates a symbolic link in le systems that support symbolic links:
#include <unistd.h> int symlink (const char *real_name, const char *symbolic_name);

A symbolic link symbolic_name is created to the name real_name.

sysfs
sysfs is a System V function that returns information about the kinds of le systems congured in the system. This function has the rather strange property of not being compatible with ANSI Cthe parameters it accepts depend on the function supplied:
#include <sys/fstyp.h> #include <sys/fsid.h> int sysfs ((int) GETFSIND, const char *fsname);

This call translates fsname, a null-terminated le-system type identier, into a le-system type index.
int sysfs ((int) GETFSTYP, int fs_index, char *buf);

This call translates fs_index, a le-system type index, into a NUL-terminated le-system type identier in the buffer pointed to by buf.
int sysfs((int) GETNFSTYP);

This call returns the total number of le system types congured in the system.

truncate and ftruncate

These functions set the EOF pointer of a le. truncate nds the le via its le name, and ftruncate requires the le number of an open le.
#include <unistd.h> int truncate (const char *path, off_t length); int ftruncate (int fd, off_t length);

These functions are available with BSD and System V.4. There is a subtle difference between the way the BSD and System V.4 versions work: if the le is smaller than the requested length, System V.4 extends the le to the specied length, while BSD leaves it as it is. Both versions discard any data beyond the end if the current EOF is longer. If your system doesnt have these functions, you may be able to perform the same function with chsize (page 206) or the fcntl function F_FREESP (page 208).

5 February 2005 02:09

Chapter 14: File systems

219

ustat
ustat returns information about a mounted le system, and is supported by System V and SunOS 4, but not by BSD. The call is:
struct ustat { daddr_t f_tfree; ino_t f_tinode; char f_fname [6]; char f_fpack [6];

/* /* /* /*

Total blocks available */ Number of free inodes */ File system name */ File system pack name */

int ustat (dev_t dev, struct ustat *buf);

On BSD systems, you can get this information with the statfs system call, which requires a path name instead of a device number.

utime and utimes

utime is available in all versions of UNIX.
#include <sys/types.h> #include <utime.h> int utime (const char *path, const struct utimbuf *times);

utime sets the modication timestamp of the le dened by path to the time specied in times. In the Seventh Edition, times was required to be a valid pointer, and only the le owner or root could use the call. All newer versions of UNIX allow times to be a NULL pointer, in which case the modication timestamp is set to the current time. Any process that has write access to the le can use utime in this manner. BSD implements this function in the C library in terms of the function utimes:
#include <sys/time.h> sys/time.h defines: struct timeval { long tv_sec; /* seconds */ long tv_usec; /* and microseconds */ }; int utimes (const char *file, const struct timeval *times); #include <sys/types.h> #include <utime.h> utime.h defines: struct utimbuf { time_t actime; /* access time */ time_t modtime; /* modification time */ }; int utime (char *path, struct utimbuf *times);

5 February 2005 02:09

220

The difference between utime and utimes is simply in the format of the access time: utime supplies the time in time_t format, which is accurate to a second, whereas utimes uses the timeval struct which is (theoretically) accurate to one microsecond. BSD systems supply the utime function as a library call (which, not surprisingly, calls utimes). On XENIX and early System V systems you can fake utimes using utime.

Non-blocking I/O
In early versions of UNIX, all device I/O was blocking: if you made a call to read and no data was available, or if you made a call to write and the device wasnt ready to accept the data, the process would sleep until the situation changed. This is still the default behaviour. Blocking I/O can be restrictive in many situations, and many schemes have been devised to allow a process to continue execution before the I/O operation completes. On current systems, you select non-blocking I/O either by supplying the ag O_NONBLOCK to open, or by calling the fcntl function F_SETFL with the O_NONBLOCK ag (see page 209). One problem with non-blocking I/O is that you dont automatically know when a request is complete. In addition, if you have multiple requests outstanding, you may not really care which nishes rst, you just want to know when one nishes. Two approaches have been used to inform a process when a request completes. One is to call a function that returns information about current request status, and that may optionally block until something completes. Traditionally, BSD uses select to perform this function, whereas System V uses poll. The other solution is to send a signal (SIGPOLL in System V, SIGIO or SIGURG in BSD) when the request nishes. In both systems, this has the disadvantage of not supplying any information about the request that completed, so if you have more than one request outstanding, you still need to call select or poll to handle the situation.

select
select is called with the following parameters:
#define FD_SETSIZE 512 my maximum FD count, see below #include <unistd.h> #include <sys/types.h> #include <sys/time.h>

These header les dene the structs:

typedef struct fd_set { fd_mask fds_bits [howmany (FD_SETSIZE, NFDBITS)]; } fd_set; struct timeval { long tv_sec; long tv_usec;

/* seconds */ /* and microseconds */

5 February 2005 02:09

Chapter 14: File systems

}; int select (int nfds, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, struct timeval *timeout);

221

The parameters readfds, writefds, and exceptfds are bit maps, one bit per possible le descriptor. Recall that le descriptors are small non-negative integers. select uses the le descriptor as an index in the bit map. This gives us a problem when porting: we dont know how many les our implementation supports. In modern systems, there is usually no xed limit. The solution chosen is a kludge: choose a sufciently high number. The expression howmany (FD_SETSIZE, NFDBITS) evaluates to the number of words of NFDBITS required to store FD_SETSIZE bits:
#define howmany(bits, wordsize) ((bits + wordsize - 1) / wordsize)

In 4.4BSD FD_SETSIZE defaults to 256 (in sys/types.h). Nowadays, a server with many requestors could quite easily exceed that value. Because of this, you can set it yourself: just dene FD_SETSIZE before including /usr/include/sys/types.h, as indicated in the syntax overview above. Setting variables of type fd_mask is tricky, so a number of macros are supplied:
FD_SET (fd, &fdset) FD_CLR (fd, &fdset) FD_ISSET (fd, &fdset) FD_ZERO (&fdset) /* set bit fd in fdset*/ /* clear bit fd in fdset */ /* return value of bit fd in fdset */ /* clear all bits in fdset */

select examines the les specied in readfds for read completion, the les specied in writefds for write completion and the les specied in exceptfds for exceptional conditions. You can set any of these pointers to NULL if youre not interested in this kind of event. The action that select takes depends on the value of timeout: If timeout is a NULL pointer, select blocks until a completion occurs on one of the specied les. If both timeout->tv_sec and timeout->tv_usec are set to 0, select checks for completions and returns immediately. Otherwise select waits for completion up to the specied timeout.

select returns -1 on error conditions, and the number of ready descriptors (possibly 0) otherwise. It replaces the contents of readfds, writefds, and exceptfds with bit maps indicating which les had a corresponding completion. So far, we havent even mentioned nfds. Strictly speaking, its not needed: you use it to indicate the number of le descriptors that are worth examining. By default, open and dup allocate the lowest possible le descriptors, so select can save a lot of work if you tell it the highest le number that is worth examining in the bit maps. Since le descriptors start at 0, the number of le descriptors is 1 higher than the highest le descriptor number. This baroque function has a couple of other gotchas waiting for you:

5 February 2005 02:09

222

The state of readfds, writefds, and exceptfds is undened if select returns 0 or -1. System V clears the descriptors, whereas BSD leaves them unchanged. Some System V programs check the descriptors even if 0 is returned: this can cause problems if you port such a program to BSD. The return value is interpreted differently in BSD and System V. In BSD, each completion event is counted, so you can have up to 3 completions for a single le. In System V, the number of les with completions is returned. On completion without timeout, Linux decrements the value of timeout by the time elapsed since the call: if timeout was initially set to 30 seconds, and I/O completes after 5 seconds, the value of timeout on return from select will be 25 seconds. This can be of use if you have a number of outstanding requests, all of which must complete in a certain time: you can call select again for the remaining le descriptors without rst calculating how much time remains. In Linux, this feature can be disabled by setting the STICKY_TIMEOUTS ag in the COFF/ELF personality used by the process. Other versions of UNIX do not currently suppport this feature, although both System V and BSD suggest that it will be implemented. For example, the man pages for 4.4BSD state:
Select() should probably return the time remaining from the original timeout, if any, by modifying the time value in place. This may be implemented in future versions of the system. Thus, it is unwise to assume that the timeout value will be unmodied by the select() call.

If you nd a system without select that does support poll, you can probably replace select with pollits just a SMOP.*

Typical use of select

Programs which use select generally start a number of I/O transfers and then go to some central place to wait for something to happen. The code could look like:
if (select (maxfnum, /* number of files to check */ &readfds, /* mask of read completions */ &writefds, /* mask of write completions */ &exceptfds, /* mask of exception completions */ NULL) > 0) /* no timeout */ { /* we have completions, */ int fd; for (fd = 0; fd < maxfnum; fd++) { if (FD_ISSET (fd, readfds)) /* this file has a read completion */ read_completion (fd); /* process it */ if (FD_ISSET (fd, writefds)) /* this file has a write completion */ write_completion (fd); /* process it */ if (FD_ISSET (fd, exceptfds)) /* this file has a exception completion */

* To quote the New Hackers Dictionary: SMOP: /S-M-O-P/ [Simple (or Small) Matter of Programming] n. 2. Often used ironically . . . when a suggestion for a program is made which seems easy to the suggester, but is obviously (to the victim) a lot of work.

5 February 2005 02:09

Chapter 14: File systems

exception_completion (fd); /* process it */ }

223

As we saw above, FD_ISSET is a macro which checks if bit fd is set in the bit mask. The foo_completion functions do whatever is needed on completion of I/O for this le descriptor. See Advanced Programming in the UNIX environment, by Richard Stevens, for further information.

poll
poll takes a different approach from select:
#include <stropts.h> #include <poll.h> ... in poll.h is the definition struct pollfd { int fd; /* file descriptor */ short events; /* requested events */ short revents; /* returned events */ }; int poll (struct pollfd *fds, unsigned long nfds, int timeout);

For each le of interest, you set up a pollfd element with the le number and the events of interest. events and revents are again bit maps. events can be made up of the following values:
Table 144: poll event codes Event POLLIN POLLRDNORM POLLRDBAND POLLPRI POLLOUT POLLWRNORM POLLWRBAND

Meaning Data other than high priority data is available for reading. Normal data* (priority band = 0) is available for reading. Data from a non-zero priority band is available for reading. High priority data is available for reading. Normal data may be written without blocking. The same as POLLOUT: normal data may be written without blocking. Priority data (priority band > 0) may be written without blocking.

When it succeeds, poll sets the corresponding bits in revents to indicate which events

* STREAMS recognizes 256 different data priority bands. Normal data is sent with priority band 0, but urgent data with a higher priority can "leapfrog" normal data. See UNIX Network Programming, by W. Richard Stevens, for further information.

5 February 2005 02:09

224

occurred. In addition, it may set the following event bits:

Table 145: poll result codes Event POLLERR POLLHUP POLLNVAL

Meaning An error has occurred on the device or stream. A hangup has occurred. The specied fd is not open.

Timeout processing is nearly the same as for select, but the parameter timeout is specied in milliseconds. Since it is an int, not a pointer, you cant supply a NULL pointer; instead, you set the value to INFTIM (dened in stropts.h) if you want the call to block.. To summarize: If timeout is set to INFTIM, poll blocks until a completion occurs on one of the specied les. If timeout is set to 0, a check is made for completions and poll returns immediately. If timeout is non-zero, poll waits for completion up to the specied timeout.

Typical use of poll

Like select, programs which use poll generally start a number of I/O transfers and then go to some central place to wait for something to happen. In this case, the code could look like:
if (poll (pollfds, maxfnum, NULL) > 0) /* wait for something to complete */ { int fd; for (fd = 0; fd < maxfnum; fd++) { if (pollfds [fd].revents) /* something completed */ ... check the result bits which interest you and perform the appropriate actions } }

The code for starting the request and enabling SIGIO and SIGURG for the line assumes that the le has been opened and the number stored in an array of le numbers.

rdchk
rdchk is a XENIX function that checks if data is available for reading on a specic le descriptor:
int rdchk (int fd);

It returns 1 if data is available, 0 if no data is currently available, and -1 on error (and errno is set). If you dont have it, you can implement it in terms of select or poll.

5 February 2005 02:09

Chapter 14: File systems

225

SIGPOLL
System V systems can arrange to have the signal SIGPOLL delivered when a request completes. It is not completely general: the le in question must be a STREAMS device, since only STREAMS drivers generate the SIGPOLL signal. The ioctl call I_SETSIG enables SIGPOLL. The third parameter species a bit mask of events to wait for:
Table 146: I_SETSIG event mask bits Mask bit S_INPUT S_HIPRI S_OUTPUT S_WRNORM S_MSG S_ERROR S_HANGUP S_RDNORM S_RDBAND S_WRBAND S_BANDURG

Event A normal priority message is on the read queue. A high priority message is on the read queue. The write queue is no longer full. The same thing as S_OUTPUT: The write queue is no longer full. A signal message is at the front of the read queue. An error message has arrived at the stream head. A hangup message has arrived at the stream head. A normal message is on the read queue. An out of band message is on the read queue. We can write out of band data. In conjunction with S_RDBAND, generate SIGURG instead of SIGPOLL.

In addition to the call to ioctl, the process needs to set up a signal handler for SIGPOLLthe default disposition is to terminate the process, which is probably not what you want.

SIGIO
BSD systems have a similar mechanism to SIGPOLL, called SIGIO. Like SIGPOLL, it also has its restrictions: it can be applied only to terminal or network devices. In addition, when outof-band data* arrives, a second signal, SIGURG, is generated. SIGIO and SIGURG are enabled by the O_ASYNC ag to open and a couple of calls to fcntlsee page 209 for more details: First, specify the process or process group that should receive the signals, using the fcntl subfunction F_SETOWN in order to enable reception of SIGURG. If you want to use SIGIO, set the O_ASYNC le status ag with the fcntl subfunction F_SETFL. As with System V, you need to dene a signal handler for SIGIO and SIGURG.

* Sockets use the term out-of-band to refer to data which comes in at a higher priority, such as TCP urgent mode. Like STREAMS priority data, this data will be presented ahead of normal data.

5 February 2005 02:09

226

File locking
The Seventh Edition did not originally allow programs to coordinate concurrent access to a le. If two users both had a le open for modication at the same time, it was almost impossible to prevent disaster. This is an obvious disadvantage, and all modern versions of UNIX supply some form of le locking. Before we look at the functions that are available, its a good idea to consider the various kinds of lock. There seem to be two of everything. First, the granularity is of interest: le locking applies to the whole le. range locking applies only to a range of byte offsets. This is sometimes misleadingly called record locking. With le locking, no other process can access the le when a lock is applied. With range locking, multiple locks can coexist as long as their ranges dont overlap. Secondly, there are two types of lock: Advisory locks do not actually prevent access to the le. They work only if every participating process ensures that it locks the le before accessing it. If the le is already locked, the process blocks until it gains the lock. mandatory locks prevent (block) read and write access to the le, but do not stop it from being removed or renamed. Many editors do just this, so even mandatory locking has its limitations. Finally, there are also two ways in which locks cooperate with each other: exclusive locks allow no other locks that overlap the range. This is the only was to perform le locking, and it implies that only a single process can access the le at a time. These locks are also called also called write locks. shared locks allow other shared locks to coexist with them. Their main purpose is to prevent an exclusive lock from being applied. In combination with mandatory range locking, a write is not permitted to a range covered by a shared lock. These locks are also called read locks. There are ve different kinds of le or record locking in common use: Lock les, also called dot locking, is a primitive workaround used by communication programs such as uucp and getty. It is independent of the system platform, but since it is frequently used well look at it briey. It implements advisory le locking. After the initial release of the Seventh Edition, a le locking package using the system call locking was introduced. It is still in use today on XENIX systems. It implements mandatory range locking. BSD systems have the system call flock. It implements advisory le locking. System V, POSIX.1, and more recent versions of BSD support range locking via the fcntl system call. BSD and POSIX.1 systems provide only advisory locking. System V supplies a choice of advisory or mandatory locking, depending on the le permissions. If you need to rewrite locking code, this is the method you should use.

5 February 2005 02:09

Chapter 14: File systems

227

System V also supplies range locking via the lockf library call. Again, it supplies a choice of advisory or mandatory locking, depending on the le permissions.

The decision between advisory and mandatory locking in System V depends on the le permissions and not on the call to fcntl or lockf. The setgid bit is used for this purpose. Normally, in executables, the setgid bit species that the executable should assume the effective group ID of its owner group when execed. On les that do not have group execute permission, it species mandatory locking if it is set, and advisory locking if it is not set. For example, A le with permissions 0764 (rwxrw-r--) will be locked with advisory locking, since its permissions include neither group execute nor setgid. A le with permissions 0774 (rwxrwxr--) will be locked with advisory locking, since its permissions dont include setgid. A le with permissions 02774 (rwxrwsr--) will be locked with advisory locking, since its permissions include both group execute and setgid. A le with permissions 02764 will be locked with mandatory locking, since it has the setgid bit set, but group execute is not set. If you list the permissions of this le with ls -l, you get rwxrwlr-- on a System V system, but many versions of ls, including BSD and GNU versions, will list rwxrwSr--.

Lock files
Lock les are the traditional method that uucp uses for locking serial lines. Serial lines are typically used either for dialing out, for example with uucp, or dialing in, which is handled by a program of the getty family. Some kind of synchronization is needed to ensure that both of these programs dont try to access the line at the same time. The other forms of locking we describe only apply to disk les, so we cant use them. Instead, uucp and getty create lock les. A typical lock le will have a name like /var/spool/uucp/LCK..ttyb, and for some reason these double periods in the name have led to the term dot locking. The locking algorithm is straightforward: if a process wants to access a serial line /dev/ttyb, it looks for a le /var/spool/uucp/LCK..ttyb. If it nds it, it checks the contents, which specify the process ID of the owner, and checks if the owner still exists. If it does, the le is locked, and the process cant access the serial line. If the le doesnt exist, or if the owner no longer exists, the process creates the le if necessary and puts its own process ID in the le. Although the algorithm is straightforward, the naming conventions are anything but standardized. When porting software from other platforms, it is absolutely essential that all programs using dot locking should be agreed on the lock le name and its format. Lets look at the lock le names for the device /dev/ttyb, which is major device number 29, minor device number 1. The ls -l listing looks like:
$ ls -l /dev/ttyb crw-rw-rw1 root sys 29, 1 Feb 25 1995 /dev/ttyb

5 February 2005 02:09

228

Table 14-7 describes common conventions:

Table 147: uucp lock le names and formats System 4.3BSD 4.4BSD System V.3 System V.4

Name /usr/spool/uucp/LCK..ttyb /var/spool/uucp/LCK..ttyb /usr/spool/uucp/LCK..ttyb /var/spool/uucp/LK.032.029.001

PID format binary, 4 bytes binary, 4 bytes ASCII, 10 bytes ASCII, 10 bytes

A couple of points to note are: The digits in the lock le name for System V.4 are the major device number of the disk on which /dev is located (32), the major device number of the serial device (29), and the minor device number of the serial device (1). Some systems, such as SCO, have multiple names for terminal lines, depending on the characteristics which it should exhibit. For example, /dev/tty1a refers to a line when running without modem control signals, and /dev/tty1A refers to the same line when running with modem control signals. Clearly only one of these lines can be used at the same time: by convention, the lock le name for both devices is /usr/spool/uucp/LCK..tty1a. The locations of the lock les vary considerably. Apart from those in the table, other possibilities are /etc/locks/LCK..ttyb, /usr/spool/locks/LCK..ttyb, and /usr/spool/uucp/LCK/LCK..ttyb.

Still other methods exist. See the le policy.h in the Taylor uucp distribution for further discussion. Lock les are unreliable. It is quite possible for two processes to go through this algorithm at the same time, both nd that the lock le doesnt exist, both create it, and both put their process ID in it. The result is not what you want. Lock les should only be used when there is really no alternative.

locking system call

locking comes from the original implementation introduced during the Seventh Edition. It is still available in XENIX. It implements mandatory range locking.
int locking (int fd, int mode, long size);

locking locks a block of data of length size bytes, starting at the current position in the le.

5 February 2005 02:09

Chapter 14: File systems

229

mode can have one of the following values:

Table 148: locking operation codes Parameter LK_LOCK LK_NBLCK LK_NBRLCK LK_RLCK LK_UNLCK

Meaning Obtain an exclusive lock for the specied block. If any part is not available, sleep until it becomes available. Obtain an exclusive lock for the specied block. If any part is not available, the request fails, and errno is set to EACCES. Obtains a shared lock for the specied block. If any part is not available, the request fails, and errno is set to EACCES. Obtain a shared lock for the specied block. If any part is not available, sleep until it becomes available. Unlock a previously locked block of data.

flock
flock is the weakest of all the lock functions. It provides only advisory le locking.
#include (defined #define #define #define #define <sys/file.h> in sys/file.h) LOCK_SH 1 LOCK_EX 2 LOCK_NB 4 LOCK_UN 8

/* /* /* /*

shared lock */ exclusive lock */ dont block when locking */ unlock */

int flock (int fd, int operation);

flock applies or removes a lock on fd. By default, if a lock cannot be granted, the process blocks until the lock is available. If you set the ag LOCK_NB, flock returns immediately with errno set to EWOULDBLOCK if the lock cannot be granted.

fcntl locking
On page 207 we discussed fcntl, a function that can perform various functions on open les. A number of these functions perform advisory record locking, and System V also offers the option of mandatory locking. All locking functions operate on a struct flock:
struct flock { short l_type; short l_whence; off_t l_start; off_t l_len; long l_sysid; pid_t l_pid;

/* /* /* /* /* /*

lock type: read/write, etc. */ type of l_start */ starting offset */ len = 0 means until end of file */ Only SVR4 */ lock owner */

5 February 2005 02:09

230
};

In this structure, l_type species the type of the lock, listed in Table 14-9.
Table 149: flock.l_type values

value F_RDLCK F_WRLCK F_UNLCK

Function Acquire a read or shared lock. Acquire a write or exclusive lock. Clear the lock.

The offset is specied in the same way as a le offset is specied to lseek: flock->l_whence may be set to SEEK_SET (offset is from the beginning of the le), SEEK_CUR (offset is relative to the current position) or SEEK_EOF (offset is relative to the current end of le position).

All fcntl lock operations use this struct, which is passed to fcntl as the arg parameter. For example, to perform the operation F_FOOLK, you would write:
struct flock flock; error = fcntl (myfile, F_FOOLK, &flock);

The following fcntl operations relate to locking: F_GETLK gets information on any current lock on the le. when calling, you set the elds flock->l_type, flock->l_whence, flock->l_start, and flock->l_len to the value of a lock that we want to set. If a lock that would cause a lock request to block already exists, flock is overwritten with information about the lock. The eld flock->l_whence is set to SEEK_SET, and flock->l_start is set to the offset in the le. flock->l_pid is set to the pid of the process that owns the lock. If the lock can be granted, flock->l_type is set to F_UNLK and the rest of the structure is left unchanged, F_SETLK tries to set a lock (flock->l_type set to F_RDLCK or F_WRLCK) or to reset a lock (flock->l_type set to F_UNLCK). If a lock cannot be obtained, fcntl returns with errno set to EACCES (System V) or EAGAIN (BSD and POSIX). F_SETLKW works like F_SETLK, except that if the lock cannot be obtained, the process blocks until it can be obtained. System V.4 has a further function, F_FREESP, which uses the struct flock, but in fact has nothing to do with le locking: it frees the space dened by flock->l_whence, flock->l_start, and flock->l_len. The data in this part of the le is physically removed, a read access returns EOF, and a write access writes new data. The only reason this operation uses the struct flock (and the reason we discuss it here) is because struct flock has suitable members to describe the area that needs to be freed. Many le systems allow data to be freed only if the end of the region corresponds with the end of le, in which case the call can be replaced with ftruncate.

5 February 2005 02:09

Chapter 14: File systems

231

lockf
lockf is a library function supplied only with System V. Like fcntl, it implements advisory or mandatory range locking based on the le permissions. In some systems, it is implemented in terms of fcntl. It supports only exclusive locks:
#include <unistd.h> int lockf (int fd, int function, long size);

The functions are similar to those supplied by fcntl. l_type species the type of the lock, as shown in Table 14-10.
Table 1410: lockf functions

value F_ULOCK F_LOCK F_TLOCK F_TEST

Function Unlock the range. Acquire exclusive lock. Lock if possible, otherwise return status. Check range for other locks.

lockf does not specify a start offset for the range to be locked. This is always the current position in the leyou need to use lseek to get there if you are not there already. The following code fragments are roughly equivalent:
flock->ltype = F_WRLK; /* lockf only supports write locks */ flock->whence = SEEK_SET; flock->l_start = filepos; /* this was set elsewhere */ flock->l_len = reclen; /* the length to set */ error = fcntl (myfile, F_GETLK, &flock); ...and lseek (myfile, SEEK_SET, filepos); /* Seek the correct place in the file */ error = lockf (myfile, F_LOCK, reclen);

Which locking scheme?

As weve seen, le locking is a can of worms. Many portable software packages offer you a choice of locking mechanisms, and your system may supply a number of them. Which do you take? Here are some rules of thumb: fcntl locking is the best choice, as long as your system and the package agree on what it means. On System V.3 and V.4, fcntl locking offers the choice of mandatory or advisory locking, whereas on other systems it only offers advisory locking. If your package expects to be able to set mandatory locking, and youre running, say, 4.4BSD, the package may not work correctly. If this happens, you may have to choose flock locking instead.

5 February 2005 02:09

232

If your system doesnt have fcntl locking, you will almost certainly have either flock or lockf locking instead. If the package supports it, use it. Pure BSD systems dont support lockf, but some versions simulate it. Since lockf can also be used to require mandatory locking, its better to use flock on BSD systems and lockf on System V systems. Youll probably not come across any packages which support locking. If you do, and your system supports it, its not a bad choice. If all else fails, use lock les. This is a very poor option, thoughits probably a better idea to consider a more modern kernel.

Memory-mapped files
Some systems offer a feature called memory mapped les: the data of a le is mapped to a particular area of memory, so you can access it directly rather than by calling read and write. This increases performance, since the virtual memory system is more efcient than the le system. The following function calls are used to implement memory mapping: You need to open the le with the le system calls open or creat. mmap maps the le into memory. msync ensures that updates to the le map are ushed back to the le. munmap frees the mapped le data.

In the following sections, well look at these functions more closely.

mmap
mmap maps a portion of a le to memory.
#include <sys/types.h> #include <sys/mman.h> caddr_t mmap (caddr_t addr, int len, int prot, int flags, int fd, off_t offset);

addr species the address at which the le should be mapped. Unless you have good reasons to do otherwise, you should specify it as NULL and let mmap choose a suitable address itself. If mmap cant place the memory where it is requested, the subsequent behaviour depends on the ag MAP_FIXEDsee the discussion of ags below. len species the length to map. prot species the accessibility of the resultant memory region, and may be any combination of PROT_EXEC (pages may be executed), PROT_READ (pages may be read) or PROT_WRITE (pages may be written). In addition, System V.4 allows the specication PROT_NONE (pages may not be accessed at all).

5 February 2005 02:09

Chapter 14: File systems

233

flags is a bit map that species properties of the mapped region. It consists of a combination of the following bit-mapped ags: MAP_ANON species that the memory is not associated with any specic le. In many ways, this is much the same thing as a call to malloc: you get an area of memory with nothing in it. This ag is available only in BSD. MAP_FILE species that the region is mapped from a regular le or character-special device. This ag, supplied only in BSD, is really a dummy and is used to indicate the opposite of MAP_ANON: if you dont have it, ignore it. MAP_FIXED species that mmap may use only the specied addr as the address of the region. The 4.4BSD man page discourages the use of this option. MAP_INHERIT permits regions to be inherited across exec system calls. Only supported in 4.4BSD. MAP_PRIVATE species that modications to the region are private: if the region is modied, a copy of the modied pages is created and the modications are copied to them. This ag is used in debuggers and to perform page-aligned memory allocations: malloc doesnt allow you to specify the address you want. In some systems, such as System V.4, MAP_PRIVATE is dened as 0, so this is the default behaviour. In others, such as SunOS 4, you must specify either MAP_PRIVATE or MAP_SHAREDotherwise the call fails with an EINVAL error code. MAP_SHARED species that modications to the region are shared: the virtual memory manager writes any modications back to the le.

On success, mmap returns the address of the area that has been mapped. On failure, it returns -1 and sets errno.

msync
Writes to the memory mapped region are treated like any other virtual memory access: the page is marked dirty, and thats all that happens immediately. At some later time the memory manager writes the contents of memory to disk. If this le is shared with some other process, you may need to explicitly ush it to disk, depending on the underlying cooperation between the le system and the virtual memory manager. System V.4 maps the pages at a low level, and the processes share the same physical page, so this problem does not arise. BSD and older versions of System V keep separate copies of memory mapped pages for each process that accesses them. This makes sharing them difcult. On these systems, the msync system call is used to ush memory areas to disk. This solution is not perfect: the possibility still exists that a concurrent read of the area may get a garbled copy of the data. To quote the 4.4BSD man pages:
Any required synchronization of memory caches also takes place at this time. Filesystem operations on a le that is mapped for shared modications are unpredictable except after an msync.

5 February 2005 02:09

234

Still, its better than nothing. The call is straightforward:

void msync (caddr_t addr, int len);

addr must be specied and must point to a memory mapped page; len may be 0, in which case all modied pages are ushed. If len is not 0, only modied pages in the area dened by addr and len are ushed.

munmap
munmap unmaps a memory mapped le region:
void munmap (caddr_t addr, int len);

It unmaps the memory region specied by addr and len. This is not necessary before terminating a programthe region is unmapped like any other on terminationand it carries the danger that modications may be lost, since it doesnt ush the region before deallocating. About the only use is to free the area for some other operation.

5 February 2005 02:09

Terminal Drivers
Terminal I/O is a real can of worms. In the Seventh Edition, it wasnt exactly simple. To quote the terminal driver man page,
The terminal handler has clearly entered the race for ever-greater complexity and generality. Its still not complex and general enough for TENEX fans.

Since then, things have gone steadily downhill. The most important terminal driver versions are: The old terminal driver, derived from the Seventh Edition terminal driver. This driver is still in use in XENIX and older BSD versions. The System III/System V terminal driver, also called termio. The POSIX.1 termios routines, derived from termio.

Most modern systems support more than one kind of serial line driver. This is known as the line discipline. Apart from terminal drivers, the most important line disciplines for asynchronous lines are SLIP (Serial Line Internet Protocol) and PPP (Point to Point Protocol). These are very implementation dependent, and we wont discuss them further. The line discipline is set with the TIOCSETD ioctl, described on page 259. Its beyond the scope of this book to explain all the intricacies and kludges that have been added to terminal handlers over the decades. Advanced Programming in the UNIX environment, by Richard Stevens, gives you a good overview of current practice, and you shouldnt really want to know about older versions unless you have trouble with them. In the following discussion, well concentrate on the four areas that cause the most headaches when porting programs: The externally visible data structures used for passing information to and from the driver. A brief overview of the different operational modes (raw, cooked, cbreak, canonical and non-canonical).

235

5 February 2005 02:09

236

The ioctl request interface to the terminal driver, one of the favourite problem areas in porting terminal-related software. The POSIX.1 termios request interface.

The documentation of every driver describes at least two different modes of treating terminal input. The Seventh Edition and BSD drivers dene three: In raw mode, the read system call passes input characters to the caller exactly as they are entered. No processing takes place in the driver. This mode is useful for programs which want to interpret characters themselves, such as full-screen editors. cooked mode interprets a number of special characters, including the new line character \n. A read call will terminate on a \n. This is the normal mode used by programs that dont want to be bothered by the intricacies of terminal programming. cbreak mode performs partial interpretation of the special characters, this time not including \n. cbreak mode is easier to use than raw mode, and is adequate for many purposes. Its a matter of taste whether you prefer this to raw mode or not. canonical * mode performs signicant processing on input before passing it to the calling function. Up to 21 input special characters may be used to tell the driver to do things as varied as start and stop output, to clear the input buffer, to send signals to the process and to terminate a line in a number of different ways. Non-canonical input mode, in which the driver does not interpret input characters specially (this corresponds roughly to BSD cbreak mode).

By contrast, termio and termios specify two different processing modes for terminal input:

In fact, subdividing the terminal operation into modes is an oversimplication: a large number of ags modify the operational modes. Later in the chapter well look at how to set these modes with termios.

Typical terminal code

This is all rather abstract: lets look at a simple example: a program wants to read a single character from the terminal. To do this, it needs to set raw or non-canonical mode, read the character, and then reinstate the previous mode. For the old terminal driver, the code looks like Example 15-1:
Example 151:
struct sgttyb initial_status; struct sgttyb raw_status; ioctl (stdin, TIOCGETA, &initial_status); raw_status = initial_status; raw_status.sg_flags |= RAW; /* initial termios flags */ /* and the same with icanon reset */ /* get attributes */ /* make a copy */ /* and set raw mode */

* The word canon refers to (religious) law: the intent is that this should be the correct or standard way to handle input characters. See the New Hackers Dictionary for a long discussion of the term.

5 February 2005 02:09

Chapter 15: Terminal Drivers Example 151: (continued)

ioctl (stdin, TIOCSETN, &raw_status); /* set the new terminal flags */ puts ("? "); if ((reply = getchar ()) != \n) /* get a reply */ puts ("\n"); /* and finish the line */ ioctl (stdin, TIOCSETN, &initial_status); /* set the old terminal flags */

237

With the System V termio driver, it would look like Example 15-2:
Example 152:
struct termio initial_status; struct termio noicanon_status; /* initial termio flags */ /* and the same with icanon reset */

ioctl (stdin, TCGETA, &initial_status); /* get attributes */ noicanon_status = initial_status; /* make a copy */ noicanon_status.c_lflag &= ICANON; /* and turn icanon off */ ioctl (stdin, TCSETA, &noicanon_status); /* set non-canonical mode */ puts ("? "); if ((reply = getchar ()) != \n) /* get a reply */ puts ("\n"); /* and finish the line */ ioctl (stdin, TCSETA, &initial_status)) /* reset old terminal mode */

Dont rely on code like this to be termio code: termios code can look almost identical. Correct termios code uses the termios functions which we will look at on page 265, and looks like Example 15-3:
Example 153:
struct termios initial_status; struct termios noicanon_status; tcgetattr (stdin, &initial_status)l noicanon_status = initial_status; noicanon_status.c_lflag &= ICANON; /* initial termios flags */ /* and the same with icanon reset */ /* get current attributes */ /* make a copy */ /* and turn icanon off */

tcsetattr (stdin, TCSANOW, &noicanon_status); /* set non-canonical mode */ puts ("? "); if ((reply = getchar ()) != \n) /* get a reply */ puts ("\n"); /* and finish the line */ tcsetattr (stdin, TCSANOW, &initial_status); /* reset old terminal mode */

Terminology
Before we start, its a good idea to be clear about a few terms that are frequently confused: All terminal drivers buffer I/O in two queues, an input queue and an output queue. The input queue contains characters that the user has entered and the process has not yet read. The output queue contains characters that the process has written but that have not yet been output to the terminal. These queues are maintained inside the terminal driver. Dont confuse them with buffers maintained in the process data space by the stdio routines.

5 February 2005 02:09

238

The term ush can mean to discard the contents of a queue, or to wait until they have all been output to the terminal. Most of the time it means to discard the contents, and thats how well use it in this chapter. The term drain means to wait until the contents of the output queue have been written to the terminal. This is also one of the meanings of ush. Special characters, frequently called control characters, are input characters that cause the terminal driver to do something out of the ordinary. For example, CTRL-D usually causes the terminal driver to return an end-of-le indication. The term special characters is the better term, since you can set them to characters that are not ASCII control characters. For example, even today, the default erase character in System V is #: its a special character, but not an ASCII control character. The baud rate of a modem is the number of units of information it can transmit per second. Modems are analogue devices that can represent multiple bits in a single unit of information modern modems encode up to 6 bits per unit. For example, a modern V.32bis modem will transfer 14400 bits per second, but runs at only 2400 baud. Baud rates are of interest only to modem designers. As the name indicates, the bit rate of a serial line indicates how many bits it can transfer per second. Bit rates are often erroneously called baud rates, even in ofcial documentation. The number of bytes transferred per second depends on the conguration: normally, an asynchronous serial line will transmit one start bit and one stop bit in addition to the data, so it transmits 10 bits per byte. break is an obsolescent method to signal an unusual condition over an asynchronous line. Normally, a continuous voltage or current is present on a line except when data is being transferred. Break effectively breaks (disconnects) the line for a period between .25 and .5 second. The serial hardware detects this and reports it separately. One of the problems with break is that it is intimately related to the serial line hardware. DCE and DTE mean data communication equipment and data terminal equipment respectively. In a modem connection, the modem is the DCE and both terminal and computer are DTEs. In a direct connect, the terminal is the DTE and the computer is the DCE. Different cabling is required for these two situations. RS-232, also known as EIA-232, is a standard for terminal wiring. In Europe, it is sometimes referred to as CCITT V.24, though V.24 does not in fact correspond exactly to RS-232. It denes a number of signals, listed in Table 15-1.
Table 151: RS-232 signals

5 February 2005 02:09

Chapter 15: Terminal Drivers Table 151: RS-232 signals (continued)

239

RS-232 name
PG TxD RxD RTS CTS DSR SG DCD DTR RI

pin 1 2 3 4 5 6 7 8 20 22

purpose Protective ground. Used for electrical grounding only. Transmitted data. Received data. Request to send. Indicates that the device has data to output. Clear to send. Indicates that the device can receive input. Can be used with RTS to implement ow control. Data set ready. Indicates that the modem (data set in older parlance) is powered on. Signal ground. Return for the other signals. Carrier detect. Indicates that the modem has connection with another modem. Data terminal ready. Indicates that the terminal or computer is ready to talk to the modem. Ring indicator. Raised by a modem to indicate that an incoming call is ringing.

For more details about RS-232, see RS-232 made easy, second edition by Martin Seyer.

Terminal data structures

In this section, well take a detailed look at the data structures youre likely to encounter when porting software from a different platform. I have included typical literal values for the macros. Dont ever use these values! Theyre not guaranteed to be correct for every implementation, and theyre included only to help you if you nd that the program includes literals rather than macro names. When writing code, always use the names.

Old terminal driver definitions

In the Seventh Edition, most ioctl calls that took a parameter referred to a struct sgttyb, which was dened in /usr/include/sgtty.h:
struct { char char char char int char char char char char sgttyb sg_ispeed; sg_ospeed; sg_erase; sg_kill; sg_flags; sg_nldly; sg_crdly; sg_htdly; sg_vtdly; sg_width; /* /* /* /* /* /* /* /* /* /* input bit rate code */ output bit rate code */ erase character */ kill character */ Terminal flags (see Table 15-3) */ delay after \n character */ delay after \r character */ delay after tab character */ delay after vt character */ terminal line width */

5 February 2005 02:09

240
char sg_length; }; /* terminal page length */

The bit rates in sg_ispeed and sg_ospeed are encoded, and allow only a certain number of speeds:
Table 152: Seventh Edition bit rate codes Parameter B0 B50 B75 B110 B134 B150 B200 B300 B600 B1200 B1800 B2400 B4800 B9600 EXTA EXTB value 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

meaning hang up phone 50 bits/second 75 bits/second 110 bits/second 134.5 bits/second 150 bits/second 200 bits/second 300 bits/second 600 bits/second 1200 bits/second 1800 bits/second 2400 bits/second 4800 bits/second 9600 bits/second External A External B

The eld sg_flags contains a bit map specifying the following actions:
Table 153: Seventh Edition tty ags Parameter XTABS INDCTL SCOPE EVENP ODDP RAW CRMOD ECHO LCASE CBREAK

value (octal) 02000 01000 0400 0200 0100 040 020 010 04 02

value (hex) 0x400 0x200 0x100 0x80 0x40 0x20 0x10 0x8 0x4 0x2

meaning Replace output tabs by spaces. Echo control characters as a, b etc. Enable neat erasing functions on display terminals ("scopes"). Even parity allowed on input (most terminals). Odd parity allowed on input. Raw mode: wake up on all characters, 8-bit interface. Map CR into LF; echo LF or CR as CR-LF. Echo (full duplex). Map upper case to lower on input. Return each character as soon as typed.

5 February 2005 02:09

Chapter 15: Terminal Drivers Table 153: Seventh Edition tty ags (continued) Parameter value value meaning TANDEM

241

(octal) 01

(hex) 0x1

Automatic ow control.

A second structure denes additional special characters that the driver interprets in cooked mode. They are stored in a struct tchars, which is also dened in /usr/include/sgtty.h:
struct { char char char char char char }; tchars t_intrc; t_quitc; t_startc; t_stopc; t_eofc; t_brkc; /* /* /* /* /* /* interrupt (default DEL) */ quit (default \) */ start output (default Q)*/ stop output (default S) */ end-of-file (default D) */ input delimiter (like nl, default -1) */

Each of these characters can be disabled by setting it to -1 (octal 0377), as is done with the default t_brkc. This means that no key can invoke its effect.

termio and termios structures

The System V terminal driver denes a struct termio to represent the data that the Seventh Edition driver stored in sgttyb and tchars. In POSIX.1 termios, it is called struct termios. Both are very similar: compared to the Seventh Edition, they appear to have been shorter by moving the special characters, which in sgttyb were stored as individual elements, into the array c_cc:
struct termio { unsigned short c_iflag; unsigned short c_oflag; unsigned short c_cflag; unsigned short c_lflag; char c_line; unsigned char c_cc [NCC]; long c_ispeed; long c_ospeed; };

/* /* /* /* /* /* /* /*

input modes */ output modes */ control modes */ local modes */ line discipline */ special chars */ input speed, some termios */ output speed, some termios */

The variable c_line species the line discipline. It is dened in termio, and not in the POSIX.1 termios standard, but some System V versions of termios have it anyway. NCC is the number of special characters. Well look at them after the ags. Not all versions of System V dene the members c_ispeed and c_ospeed. Instead, they encode the line speed in c_cflag. The correct way to access them is via the termios utility functions cfgetispeed, cfsetispeed, cfgetospeed, cfsetospeed and cfsetspeed, which we will discuss on page 265. To make matters worse, some older System V termios implementations supplied c_ispeed and c_ospeed, but the implementation didnt use them. In addition, many systems cannot handle different input and output speeds, so setting one

5 February 2005 02:09

242

speed automatically sets the other as well. c_iflag, c_oflag, c_cflag and c_lflag (a total of 128 possible bits) take the place of the Seventh Edition sg_flags.

c_iflag
c_iflag species how the driver treats terminal input:
Table 154: termios c_iag bits Parameter IGNBRK BRKINT IGNPAR PARMRK

value (SysV) 0x1 0x2 0x4 0x8

value (BSD) 0x1 0x2 0x4 0x8

meaning Ignore break condition. Generate a SIGINT signal on break. Ignore characters with parity errors. If a parity or framing error occurs on input, accept it and insert into the input stream the three-character sequence 0xff, 0, and the character received. Enable input parity check. Strip bit 7 from character. Map NL to CR on input. Ignore CR. Map CR to NL on input. Map uppercase to lowercase on input. Enable output ow control with XON/XOFF (CTRLS/CTRL-Q). Allow any character to restart output after being stopped by CTRL-S. Enable input ow control with XON/XOFF. Enable CTS protocol for a modem line. Enable RTS signaling for a modem line. Ring the terminal bell when the input queue is full.

INPCK ISTRIP INLCR IGNCR ICRNL IUCLC1 IXON IXANY IXOFF CTSFLOW1 RTSFLOW1 IMAXBEL2
1 2

0x10 0x20 0x40 0x80 0x100 0x200 0x400 0x800 0x1000 0x2000 0x4000 0x2000

0x10 0x20 0x40 0x80 0x100 0x200 0x800 0x400

0x2000

not in POSIX.1 or BSD. not in POSIX.1 and some versions of System V. A couple of these ags are not portable: IUCLC maps lower case to upper case: if you enter a lower case character, it is converted to an upper case character and echos that way. Many people consider this a bug, not a feature. Theres no good way to implement this on a non-System V system. If you really want to have this behaviour, youll have to turn off echo and provide an echo from the program. CTSFLOW and RTSFLOW specify ow control via the RS-232 signals CTS and RTS. These are control ags, of course, not input ags, but some versions of System V put them here

5 February 2005 02:09

Chapter 15: Terminal Drivers

243

for backward compatibility with XENIX. Some other versions of System V dont dene them at all, and BSD systems and yet other System V systems supply them in c_cflags, where they belong. c_oflag species the behaviour on output.
Table 155: termios c_oag bits Parameter OPOST OLCUC1 ONLCR OCRNL ONOCR ONLRET OFILL OFDEL NLDLY1 NL0 NL1 CRDLY1 CR0 CR1 CR2 CR3 TABDLY1 TAB0 TAB1 TAB2 TAB3 BSDLY1 BS0 BS1 VTDLY1 VT0 VT1 FFDLY1 FF0 FF1

value (SysV) 0x1 0x2 0x4 0x8 0x10 0x20 0x40 0x80 0x100 0x0 0x100 0x600 0x0 0x200 0x400 0x600 0x18000 0x0 0x800 0x1000 0x1800 0x2000 0x0 0x2000 0x4000 0x0 0x4000 0x8000 0x0 0x8000

value (BSD) 0x1 0x2 0x8 0x10 0x20 0x40 0x80

meaning Postprocess output. Map lower case to upper on output. Map NL to CR-NL on output. Map CR to NL on output. Suppress CR output at column 0. NL performs CR function. Use ll characters for delay. Fill is DEL if set, otherwise NUL.* Mask bit for new-line delays: No delay after NL. One character delay after NL. Mask bits for carriage-return delays: No delay after CR. One character delay after CR. Two characters delay after CR. Three characters delay after CR. Mask bits for horizontal-tab delays: No delay after HT. One character delay after HT. Two characters delay after HT. Expand tabs to spaces. Mask bit for backspace delays: No delay after BS. One character delay after BS. Mask bit for vertical-tab delays: No delay after VT. One character delay after VT. Mask bit for form-feed delays: No delay after FF. One character delay after FF.

* The ASCII character represented by binary 0 (the C character constant \0). Not to be confused with the null pointer, which in C is usually called NULL.

5 February 2005 02:09

244 Table 155: termios c_oag bits (continued)

1

not in POSIX.1 or BSD. A number of these ags are not portable: System V supplies a large number of ags designed to compensate for mechanical delays in old hardcopy terminal equipment. Its doubtful that any of this is needed nowadays. If you do have an unbuffered hardcopy terminal connected to your BSD machine, and it loses characters at the beginning of a line or a page, you should check whether CTS/RTS ow control might not help. Or you could buy a more modern terminal. OLCUC is obsolete, of course, but if that old hardcopy terminal also doesnt support lower-case, and it doesnt upshift lower-case characters automatically, youll have to do it programatically.

c_cflag species hardware control aspects of the terminal interface:

Table 156: termios c_cag bits Parameter CBAUD1 B0 B50 B75 B110 B134 B150 B200 B300 B600 B1200 B1800 B2400 B4800 B9600 B19200 EXTA B38400 EXTB CSIZE CS5 CS6 CS7 CS8

value (SysV) 0xf 0 0x1 0x2 0x3 0x4 0x5 0x6 0x7 0x8 0x9 0xa 0xb 0xc 0xd 0xe 0xe 0xf 0xf 0x30 0x0 0x10 0x20 0x30

value (BSD)

meaning Bit rate Hang up 50 bps 75 bps 110 bps 134 bps 150 bps 200 bps 300 bps 600 bps 1200 bps 1800 bps 2400 bps 4800 bps 9600 bps 19200 bps External A 38400 bps External B Mask bits for character size: 5 bits 6 bits 7 bits 8 bits

0x300 0x0 0x100 0x200 0x300

5 February 2005 02:09

Chapter 15: Terminal Drivers Table 156: termios c_cag bits (continued) Parameter value value meaning

245

(SysV)
CSTOPB CREAD PARENB PARODD HUPCL CLOCAL RCV1EN3 XMT1EN3 LOBLK3 CTSFLOW1 CCTS_OFLOW2 CRTSCTS2 RTSFLOW1 CRTS_IFLOW2 MDMBUF2
1

(BSD) 0x400 0x800 0x1000 0x2000 0x4000 0x8000 Send two stop bits (if not set, send 1 stop bit). Enable receiver. Enable parity. Set odd parity if set, otherwise even. Hang up on last close. Disable modem control lines. see below see below Block layer output. CTS ow control of output. CTS ow control of output. CTS ow control of output (alternative symbol). RTS ow control of input. RTS ow control of input. Flow control output via Carrier.

0x40 0x80 0x100 0x200 0x400 0x800 0x1000 0x2000 0x4000 0x10000

0x10000 0x10000 0x20000 0x20000 0x100000

speeds are encoded differently in BSDsee below. not in POSIX.1 or System V. 3 not in POSIX.1 or BSD. Again, some of these ags are only available on specic platforms:
2

RCV1EN and XMT1EN are dened in some System V header les, but they are not documented. BSD systems supply CRTS_IFLOW and CCTS_OFLOW for RS-232 ow control. Some System V systems supply RTSFLOW and CTSFLOW to mean the same thing, but other System V systems dont support it, and other systems again put these ags in c_iflag.

c_lflag species the behaviour specic to the line discipline. This ag varies so much between System V and BSD that its easier to put them in separate tables. Table 15-7 describes the standard System V line discipline, and Table 15-8 describes the standard BSD line discipline,
Table 157: System V termios c_lag bits Parameter ISIG ICANON

value 0x1 0x2

meaning Allow the characters INTR, QUIT, SUSP and DSUSP to generate signals. Enable canonical input (erase and kill processing).

5 February 2005 02:09

246 Table 157: System V termios c_lag bits (continued) Paramvalue meaning eter XCASE

0x4

ECHO ECHOE ECHOK ECHONL NOFLSH

0x8 0x10 0x20 0x40 0x80

In conjunction with ICANON, map upper/lower case to an upper-case only terminal. Lower case letters are displayed in upper case, and upper case letters are displayed with a preceding backslash (\). Enable echo. Erase character removes character from screen. Echo NL after line kill character. Echo NL even if echo is off. Disable ush after interrupt or quit.

Heres the BSD version:

Table 158: BSD termios c_lag bits Parameter ECHOKE1 ECHOE ECHOK ECHO ECHONL ECHOPRT1 ECHOCTL1 ISIG ICANON ALTWERASE1

value 0x1 0x2 0x4 0x8 0x10 0x20 0x40 0x80 0x100 0x200

IEXTEN EXTPROC1 TOSTOP FLUSHO1 NOKERNINFO1 PENDIN1 NOFLSH

0x400 0x800 0x400000 0x800000 0x2000000 0x20000000 0x80000000

meaning Line kill erases line from screen. Erase character removes character from screen. Echo NL after line kill character. Enable echo. Echo NL even if echo is off. Visual erase mode for hardcopy. Echo control chars as (Char). Enable signals INTR, QUIT, SUSP and DSUSP. Enable canonical input (erase and kill processing). Use alternate WERASE algorithm. Instead of erasing back to the rst blank space, erase back to the rst non-alphanumeric character. Enable DISCARD and LNEXT. This ag carries the comment "External processing". Apart from that, it appears to be undocumented. If a background process attempts output, send a SIGTTOU to it. By default this stops the process. Status return only: output being ushed. Prevent the STATUS character from displaying information on the foreground process group. Pending input is currently being redisplayed. Dont ush input and output queues after receiving SIGINT or SIGQUIT.

not in POSIX.1.

5 February 2005 02:09

Chapter 15: Terminal Drivers

247

Converting the c_lflag bits is even more of a problem: XCASE is part of the System V upper case syndrome that we saw with c_iflag and c_oflag. BSD offers a number of echo ags that are not available in System V. In practice, this is a cosmetic difference in the way input works. Consider a BSD program with a line like:
term.c_lflag = ECHOKE | ECHOE | ECHOK | ECHOCTL;

This will fail to compile under System V because ECHOKE and ECHOCTL are undened. You can probably ignore these ags, so the way to x it would be something like:
term.c_lflag = ECHOE | ECHOK #ifdef ECHOKE | ECHOKE #endif #ifdef ECHOCTL | ECHOCTL #endif ;

Note the lonesome semicolon on the last line. The ags FLUSHO and PENDIN are status ags that cannot be set. Theres no way to get this information in System V. NOKERNINFO refers to the STATUS character, which we will see below. This is not supported in System V.

special characters
The number of special characters has increased from 6 in the Seventh Edition (struct tchars) to 8 in termio and a total of 20 in termios (though 4 of the termios special characters are reserved in other words, not dened). Despite this number, there is no provision for redening CR and NL.
Table 159: termio and termios special characters

Name
CR

Index in c_cc (SysV) (none)

Default (SysV)
\r

Index in c_cc (BSD) (none)

Default
(BSD) \r

NL

(none)

\n

(none)

\n

VINTR

DEL

CTRL-C

Function Go to beginning of line. In canonical and cooked modes, complete a read request. End line. In canonical and cooked modes, complete a read request. Generate an SIGINT signal.

5 February 2005 02:09

248 Table 159: termio and termios special characters (continued)

Name
VQUIT VERASE VKILL VEOF VEOL VEOL21 VSWTCH1, VSTART VSTOP VSUSP VDSUSP1 VREPRINT1

Index in c_cc (SysV) 1 2 3 4 5 6 7 8 9 10 11 12

Default (SysV)
CTRL-| #4 @4 CTRL-D NUL NUL NUL CTRL-Q CTRL-S CTRL-Z CTRL-Y CTRL-R

Index in c_cc (BSD) 9 3 5 0 1 2 12 13 10 11 6

Default
(BSD) CTRL-| DEL CTRL-U CTRL-D \377 \377 CTRL-Q CTRL-S CTRL-Z CTRL-Y CTRL-R

VDISCARD1

13

CTRL-O

15

CTRL-O

VWERASE1 VLNEXT1 VSTATUS1,3

14 15

CTRL-W CTRL-V

4 14 18

CTRL-W CTRL-V \377

Function Generate a SIGQUIT signal. Erase last character. Erase current input line. Return end-of-le indication. Alternate end-of-line character. Alternate end-of-line character. shl layers: switch shell. Resume output after stop. Stop output. Generate a SIGTSTP signal when typed. Generate a SIGTSTP signal when the character is read. Redisplay all characters in the input queue (in other words, characters that have been input but not yet read by any process). The term "print" recalls the days of harcopy terminals. Discard all terminal output until another DISCARD character arrives, more input is typed or the program clears the condition. Erase the preceding word. Interpret next character literally. Send a SIGINFO signal to the foreground process group. If NOKERNINFO is not set, the kernel also prints a status message on the terminal.

not in POSIX.1. shl layers are a System V method of multiplexing several shells on one terminal. They are not supported on BSD systems. 3 not supported on System V. 4 These archaic, teletype-related values are still the default for System V. The le /usr/include/sys/termio.h contains alternative denitions (VERASE set to CTRL-H and VKILL set to CTRL-X), but these need to be specically enabled by dening the preprocessor variable _NEW_TTY_CTRL.
2

5 February 2005 02:09

Chapter 15: Terminal Drivers

249

You will frequently see these names without the leading V. For example, the stty program refers to VQUIT as QUIT.

Terminal driver modes

Depending on the driver, it looks as if you have a choice of two or three operational modes on input: With the termio and termios drivers, you have the choice of canonical and non-canonical mode. With the old terminal driver, you have the choice of raw, cooked and cbreak modes.

This distinction is not as clear-cut as it appears: in fact, you can set up both drivers to do most things you want.

Canonical mode
To quote Richard Stevens Advanced Programming in the UNIX environment: Canonical mode is simpleit takes only about 30 pages for a brief description. For an even simpler description: everything in the rest of this chapter applies to canonical mode unless otherwise stated.

Non-canonical mode
Non-canonical mode ignores all special characters except INTR, QUIT, SUSP, STRT, STOP, DISCARD and LNEXT. If you dont want these to be interpreted, you can disable them by setting the corresponding entry in tchars to _POSIX_VDISABLE. The terminal mode has a strong inuence on how a read from a terminal completes. In canonical mode, a read request will complete when the number of characters requested has been input, or when the user enters one of the characters CR, NL, VEOL or (where supported) VEOL2. In non-canonical mode, no special character causes a normal read completion. The way a read request completes depends on two variables, MIN and TIME. MIN represents a minimum number of characters to be read, and TIME represents a time in units of 0.1 second. There are four possible cases: 1. Both MIN and TIME are non-zero. In this case, a read will complete when either MIN characters have been entered or TIME/10 seconds have passed since a character was entered. The timer starts when a character is entered, so at least one character must be entered for the read to complete. MIN is non-zero, TIME is zero. In this case, the read will not complete until MIN characters have been entered. MIN is zero and TIME is non-zero. The read will complete after entering one character or after TIME/10 seconds. In the latter case, 0 characters are returned. This is not the same as setting MIN to 1 and leaving TIME as it is: in this case, the read would not

2. 3.

5 February 2005 02:09

250

complete until at least one character is entered. 4. Both MIN and TIME are set to 0. In this case, read returns immediately with any characters that may be waiting.

If MIN is non-zero, it overrides the read count specied to read, even if read requests less than MIN characters: the remaining characters are kept in the input queue for the next read request. This can have the unpleasant and confusing effect that at rst nothing happens when you type something in, and then suddenly multiple reads complete. Non-canonical mode does not interpret all the special characters, but it needs space to store MIN and TIME. In 4.4BSD, two of the reserved characters are used for this purpose. Most other implementations, including XENIX, System V and some older BSDs do it differently, and this can cause problems: The value of VEOF is used for VMIN. This value is normally CTRL-D, which is decimal 4: if you switch from canonical to non-canonical mode and do not change MIN, you may nd that a read of a single character will not complete until you enter a total of four characters. The value of VEOL is used for TIME. This is normally 0.

Raw mode
Raw mode does almost no interpretation of the input stream. In particular, no special characters are recognized, and there is no timeout. The non-canonical mode variables MIN and TIME do not exist. The result is the same as setting MIN to 1 and TIME to 0 in non-canonical mode.

Cooked mode
The cooked mode of the old terminal driver is essentially the same as canonical mode, within the limitations of the driver data structurestermios offers some features that are not available with the old terminal driver, such as alternate end-of-line characters.

Cbreak mode
To quote the Seventh Edition manual:
CBREAK is a sort of half-cooked (rare?) mode.

In terms of termios, it is quite close to non-canonical mode: the only difference is that cbreak mode turns off echo. Non-canonical mode does not specify whether echo is on or off.

Emulating old terminal driver modes

Table 15-10 illustrates how you can dene old driver terminal modes with termios. Youll see that a large number of entries are not dened: raw and cbreak modes do not specify how these

5 February 2005 02:09

Chapter 15: Terminal Drivers

251

parameters are set. You can set them to whatever you feel appropriate.
Table 1510: Dening terminal modes with termios

Flag
BRKINT INPCK ISTRIP ICRNL IXON OPOST CSIZE PARENB ECHO ISIG ICANON IEXTEN VMIN VTIME

raw mode
off off off off off off CS8 off off off off off 1 0

cbreak mode
on on not defined not defined not defined not defined not defined not defined off not defined off not defined 1 0

gtty and stty

You may still occasionally run into the system calls stty and gtty, which are leftovers from the Seventh Edition. You can replace stty with the ioctl function TIOCSETP, and you can replace gtty with the ioctl request TIOCGETP. Read more on both these requests on page 257.

The Linux terminal driver

Linux has the great advantage of being a recent development, so it doesnt have a number of the warts of older terminal drivers. It goes to some trouble to be compatible, however: In addition to POSIX.1 termios, the kernel also directly supports System V termio. The library libbsd.a includes ioctl calls for the old terminal driver, which Linux users call the BSD driver. The only line discipline you can expect to work under Linux is the standard tty line discipline N_TTY.

5 February 2005 02:09

252

ioctl
ioctl is the le system catchall: if there isnt any other function to do the job, then somebody will bend ioctl to do it. Nowhere is this more evident than in terminal I/O handling. As a result of this catchall nature, its not easy to represent ioctl parameters in C. Well look at the semantics rst. The ioctl function call takes three parameters: 1. 2. 3. A le number. A request, which well look at in more detail in the next section. When present, the meaining is dened by the request. It could be an integer, another request code or a pointer to some structure dened by the request.

ioctl request codes

The key to understanding ioctl is the request code. Request codes are usually subdivided into a number of elds. For example, 4.4BSD denes four elds:
Bit 31 29 28 length 16 15 ioctl type 8 7 function subcode 0

type

The rst three bits specify the type of parameter. IOC_VOID (0x20 in the rst byte) species that the request takes no parameters, IOC_OUT (0x40 in the rst byte) species that the parameters are to be copied out of the kernel (in other words, that the parameters are to be returned to the user), and IOC_IN (0x80 in the rst byte) species that the parameters are to be copied in to the kernel (they are to be passed to ioctl). The next 13 bits specify the length of the parameter in bytes. The next byte species the type of request. This is frequently a mnemonic letter. In 4.4BSD, this eld is set to the lower-case letter t for terminal ioctls. Finally, the last byte is a number used to identify the request uniquely.

This encoding depends heavily on the operating system. Other systems (especially, of course, 16 bit systems) encode things differently, but the general principle remains the same. Both the request code and the third parameter, where present, do not map easily to C language data structures. As a result, the denition of the function varies signicantly. For example, XENIX and BSD declare it as:
#include <sys/ioctl.h> int ioctl (int fd, unsigned long request, char *argp)

and System V.4 has

5 February 2005 02:09

Chapter 15: Terminal Drivers

#include <unistd.h> int ioctl (int fs, int request,

253

/* arg */ ...);

Strictly speaking, since the request code is not a number, both int and unsigned long are incorrect, but they both do the job. When debugging a program, its not always easy to determine which request has been passed to ioctl. If you have the source code, you will see something like
ioctl (stdin, TIOCGETA, &termstat);

Unfortunately, a number of ioctl calls are embedded in libraries to which you probably dont have source, but you can gure out whats going on by setting a breakpoint on ioctl. In this example, when you hit the breakpoint, you will see something like:
(gdb) bt #0 ioctl (file=0, request=1076655123, parameter=0xefbfd58c "") at ioctl.c:6 #1 0x10af in main () at foo.c:12

The value of request looks completely random. In hexadecimal it starts to make a little more sense:
(gdb) p/x request $1 = 0x402c7413

If we compare this with the request code layout in the example above, we can recognize a fair amount of information: The rst byte starts with 0x40, IOC_OUT: the parameter exists and denes a return value. The next 13 bits are 0x2c, the length to be returned (this is the length of struct termios). The next byte is 0x74, the ASCII character t, indicating that this is a terminal ioctl request. The last byte is 0x13 (decimal 19).

Its easy enough to understand this when its deciphered like this, but doing it yourself is a lot different. The rst problem is that there is no agreed place where the ioctl requests are dened. The best place to start is in the header le sys/ioctl.h, which in the case of 4.4BSD will lead you to the le sys/ioccom.h (sys/sys/ioccom.h in the 4.4BSD distribution). Here you will nd code like:
#define #define #define #define #define #define #define IOCPARM_MASK IOCPARM_LEN(x) IOCBASECMD(x) IOCGROUP(x) IOC_VOID IOC_OUT IOC_IN 0x1fff /* parameter length, at most 13 bits */ (((x) >> 16) & IOCPARM_MASK) ((x) & (IOCPARM_MASK << 16)) (((x) >> 8) & 0xff) 0x20000000 /* no parameters */ 0x40000000 /* copy out parameters */ 0x80000000 /* copy in parameters */

These dene the basic parts of the request. Next come the individual types of request:

5 February 2005 02:09

254
#define _IOC(inout,group,num,len) \ pass a structure of length len as parameter (inout | ((len & IOCPARM_MASK) << 16) | ((group) << 8) | (num)) #define _IO(g,n) _IOC(IOC_VOID, (g), (n), 0) No parameter #define _IOR(g,n,t) _IOC(IOC_OUT, (g), (n), sizeof(t)) Return parameter from kernel #define _IOW(g,n,t) _IOC(IOC_IN, (g), (n), sizeof(t)) Pass parameter to kernel /* this should be _IORW, but stdio got there first */ #define _IOWR(g,n,t) _IOC(IOC_INOUT, (g), (n), sizeof(t)) Pass and return parameter

With these building blocks, we can now understand the real denitions:
#define #define #define #define #define #define TIOCSBRK TIOCCBRK TIOCSDTR TIOCCDTR TIOCGPGRP TIOCSPGRP _IO(t, 123) _IO(t, 122) _IO(t, 121) _IO(t, 120) _IOR(t, 119, int) _IOW(t, 118, int) /* /* /* /* /* /* set break bit */ clear break bit */ set data terminal ready */ clear data terminal ready */ get pgrp of tty */ set pgrp of tty */

These dene four requests without parameters (_IO), a request that returns an int parameter from the kernel (_IOR), and a request that passes an int parameter to the kernel (_IOW).

Terminal ioctls
For a number of reasons, its difcult to categorize terminal driver ioctl calls: As the terminal driver has changed over the course of time, some implementors have chosen to keep the old ioctl codes and give them new parameters. For example, the Seventh Edition call TIOCGETA returned the terminal parameters to a struct sgttyb. The same call in System V returns the values to a struct termio, and in 4.4BSD it returns the values to a struct termios. The documentation for many ioctl calls is extremely hazy: although System V supports the old terminal driver discipline, the documentation is very scant. Just because an ioctl function is not documented in the man pages doesnt mean that it isnt supported: its better to check in the header les (usually something like sys/termio.h or sys/termios.h). Many ioctl calls seem to duplicate functionality. There are minor differences, but even they are treacherous. For example, in the Seventh Edition the TIOCSETA function drains the output queue and discards the content of the input queue before setting the terminal state. The same function in 4.4BSD performs the function immediately. To get the Seventh Edition behaviour, you need to use TIOCSETAF. The behaviour in System V is not documented, which means that you may be at the mercy of the implementor of the device driver: on one system, it may behave like the Seventh Edition, on another like 4.4BSD.

In the following sections, well attempt to categorize the most frequent ioctl functions in the

5 February 2005 02:09

Chapter 15: Terminal Drivers

255

kind of framework that POSIX.1 uses for termios. Heres an index to the mess:
Table 1511: ioctl parameters

Name
TCFLSH TCGETA TCGETS TCSBRK TCSETA TCSETAF TCSETAW TCSETS TCSETSF TCSETSW TCXONC TIOCCBRK TIOCCDTR TIOCCONS TIOCDRAIN TIOCFLUSH TIOCGETA TIOCGETC TIOCGETD TIOCGETP TIOCGPGRP TIOCGSID TIOCGSOFTCAR TIOCGWINSZ TIOCHPCL TIOCMBIC TIOCMBIS TIOCMGET TIOCMSET TIOCNXCL TIOCNOTTY TIOCOUTQ TIOCSBRK TIOCSCTTY TIOCSDTR TIOCSETA TIOCSETAF TIOCSETAW

Function Flush I/O Get terminal state Get terminal state Drain output, send break Set terminal state Drain I/O and set state Drain output and set state Set terminal state Drain I/O and set state Drain output and set state Set ow control Clear break Clear DTR Set console Drain output queue Flush I/O Get current state Get special chars Set line discipline Get current state Get process group ID Get session ID Get DCD indication Get window size Hang up on clear Clear modem state bits Set modem state bits Get modem state Set modem state Clear exclusive use Drop controlling terminal Get output queue length Send break Set controlling tty Set DTR Set terminal state Drain I/O and set state Drain output and set state

Parameter 3 int * struct termio * struct termios * int * struct termio * struct termio * struct termio * struct termios * struct termios * struct termios * int * (none) (none) int * (none) int * struct termio * struct tchars * int *ldisc struct sgttyb * pid_t * pid_t * int * struct winsize * (none) int * int * int * int * (none) (none) int * (none) (none) (none) struct sgttyb * struct termios * struct termios *

Page 263 258 258 261 259 259 259 258 258 258 262 260 260 264 262 263 256 258 259 257 263 264 265 259 258 261 261 261 261 264 264 262 260 263 260 257 257 257

5 February 2005 02:09

256 Table 1511: ioctl parameters (continued)

Name
TIOCSETC TIOCSETD TIOCSETN TIOCSETP TIOCSPGRP TIOCSSOFTCAR TIOCSTART TIOCSTI TIOCSTOP TIOCSWINSZ

Function Set special chars Set line discipline Set state immediately Get current state Set process group ID Set DCD indication Start output Simulate input Stop output Set window size

Parameter 3 struct tchars * int *ldisc struct sgttyb * struct sgttyb * pid_t * int * (none) char * (none) struct winsize *

Page 258 259 257 257 263 265 262 262 262 259

Terminal attributes
One of the most fundamental groups of ioctl requests get and set the terminal state. This area is the biggest mess of all. Each terminal driver has its own group of requests, the request names are similar enough to be confusing, different systems use the same request names to mean different things, and even in termios, there is no agreement between BSD and System V about the names of the requests. Table 15-12 gives an overview.
Table 1512: Comparison of sgttyb, termio and termios ioctls

Function

sgtty request TIOCGETA TIOCGETC TIOCSETN TIOCSETA TIOCSETC

termio request TCGETA TCGETA TCSETA TCSETAW TCSETAF TCSETAF

termios request (BSD) TIOCGETA TIOCGETA TIOCSETA TIOCSETAW TIOCSETAF TIOCSETAF

termios request (System V) TCGETS TCGETS TCSETS TCSETSW TCSETSF TCSETSF

Get current state Get special chars Set terminal state immediately Drain output and set state Drain I/O and set state Set special chars

TIOCGETA
The call ioctl (fd, TIOCGETA, term) places the current terminal parameters in the structure term. The usage differs depending on the system: In the Seventh Edition, term was of type struct sgttyb *. In System V, term is of type struct termio *.

5 February 2005 02:09

Chapter 15: Terminal Drivers

257

In 4.4BSD, term is of type struct termios *. The Seventh Edition request TIOCSETN only sets the terminal state described in the rst 6 bytes of struct sgettyb.

TIOCSETA
The call ioctl (fd, TIOCSETA, term) sets the current terminal state from term. The usage differs depending on the system: In the Seventh Edition, term was of type struct sgttyb *. The system drained the output queue and ushed the input queue before setting the parameters. In System V.3, term is of type struct termio *. The drain and ush behaviour is not documented. In 4.4BSD, term is of type struct termios *. The action is performed immediately with no drain or ush. This is used to implement the tcsetattr function with the TCSANOW option.

TIOCGETP and TIOCSETP

TIOCGETP and TIOCSETP are obsolete versions of TIOCGETA and TIOCSETA respectively. They affect only the rst 6 bytes of the sgttyb structure (sg_ispeed to sg_flags). These requests correspond in function to the obsolete Seventh Edition system calls stty and gtty.

TIOCSETAW
The call ioctl (fd, TIOCSETAW, void *term) waits for any output to complete, then sets the terminal state associated with the device. 4.4BSD uses this call to implement the tcsetattr function with the TCSADRAIN option. In XENIX, the parameter term is of type struct termio; in other systems is it of type struct termios.

TIOCSETAF
The call ioctl (fd, TIOCSETAF, void *term) waits for any output to complete, ushes any pending input and then sets the terminal state. 4.4BSD uses this call to implement the tcsetattr function with the TCSAFLUSH option. In XENIX, the parameter term is of type struct termio, in other systems is it of type struct termios.

TIOCSETN
The call ioctl (fd, TIOCSETN, struct sgttyb *term) sets the parameters but does not delay or ush input. This call is supported by System V.3. and the Seventh Edition. In the Seventh Edition, this function works only on the rst 6 bytes of the sgttyb structure.

5 February 2005 02:09

258

TIOCHPCL
The call ioctl (fd, TIOCHPCL, NULL) species that the terminal line is to be disconnected (hung up) when the le is closed for the last time.

TIOCGETC
The call ioctl (fd, TIOCGETC, struct tchars *chars) returns the terminal special characters to chars.

TIOCSETC
The call ioctl (fd, TIOCSETC, struct tchars *chars) sets the terminal special characters from chars.

TCGETS
The call ioctl (fd, TCGETS, struct termios *term) returns the current terminal parameters to term. This function is supported by System V.4.

TCSETS
The call ioctl (fd, TCSETS, struct termios *term) immediately sets the current terminal parameters from term. This function is supported by System V.4 and corresponds to the 4.4BSD call TIOCSETA.

TCSETSW
The call ioctl (fd, TCSETSW, struct termios *term) sets the current terminal parameters from term after all output characters have been output. This function is supported by System V.4 and corresponds to the 4.4BSD call TIOCSETAW.

TCSETSF
The call ioctl (fd, TCSETSF, struct termios *term) ushes the input queue and sets the current terminal parameters from term after all output characters have been output. This function is supported by System V.4 and corresponds to the 4.4BSD call TIOCSETAF.

TCGETA
The call ioctl (fd, TCGETA, struct termio *term) stores current terminal parameters in term. Not all termios parameters can be stored in a struct termio; you may nd it advantageous to use TCGETS instead (see above).

5 February 2005 02:09

Chapter 15: Terminal Drivers

259

TCSETA
The call ioctl (fd, TCSETA, struct termio *term) sets the current terminal status from term. Parameters that cannot be stored in struct termio are not affected. This corresponds to TCSETA, except that it uses a struct termio * instead of a struct termios *.

TCSETAW
The call ioctl (fd, TCSETAW, struct termio *term) sets the current terminal parameters from term after draining the output queue. This corresponds to TCSETW, except that it uses a struct termio * instead of a struct termios *.

TCSETAF
The call ioctl (fd, TCSETAF, struct termio *term) input queue ushes the input queue and sets the current terminal parameters from term after all output characters have been output. This corresponds to TCSETF, except that it uses a struct termio * instead of a struct termios *.

TIOCGWINSZ
The call ioctl (fd, TIOCGWINSZ, struct winsize *ws) puts the window size information associated with the terminal in ws. The window size structure contains the number of rows and columns (and pixels if appropiate) of the devices attached to the terminal. It is set by user software and is the means by which most full screen oriented programs determine the screen size. The winsize structure is dened as:
struct winsize { unsigned short unsigned short unsigned short unsigned short };

ws_row; ws_col; ws_xpixel; ws_ypixel;

/* /* /* /*

rows, in characters */ columns, in characters */ horizontal size, pixels */ vertical size, pixels */

Many implementations ignore the members ws_xpixel and ws_ypixel and set them to 0.

TIOCSWINSZ
The call ioctl (fd, TIOCSWINSZ, struct winsize *ws) sets the window size associated with the terminal to the value at ws. If the new size is different from the old size, a SIGWINCH (window changed) signal is sent to the process group of the terminal. See TIOCGWINSZ for more details.

TIOCSETD
The call ioctl (fd, TIOCSETD, int *ldisc); changes the line discipline to ldisc. Not all systems support multiple line disciplines, and both the available line disciplines and their names depend on the system. Here are some typical ones:

5 February 2005 02:09

260

OTTYDISC: In System V, the old (Seventh Edition) tty discipline. NETLDISC: The Berknet line discipline. NTTYDISC: In System V, the new (termio) tty discipline. TABLDISC: The Hitachi tablet discipline. NTABLDISC: The GTCO tablet discipline. MOUSELDISC: The mouse discipline. KBDLDISC: The keyboard line discipline. TTYDISC: The termios interactive line discipline. TABLDISC: The tablet line discipline. SLIPDISC: The Serial IP (SLIP) line discipline.

TIOCGETD
The call ioctl (fd, TIOCGETD, int *ldisc) returns the current line discipline at ldisc. See the discussion in the section on TIOCSETD above.

Hardware control
TIOCSBRK
The call ioctl (fd, TIOCSBRK, NULL) sets the terminal hardware into break condition. This function is supported by 4.4BSD.

TIOCCBRK
The call ioctl (fd, TIOCCBRK, NULL) clears a terminal hardware BREAK condition. This function is supported by 4.4BSD.

TIOCSDTR
The call ioctl (fd, TIOCSDTR, NULL) asserts Data Terminal Ready (DTR). This function is supported by 4.4BSD. See page 239 for details of the DTR signal.

TIOCCDTR
The call ioctl (fd, TIOCCDTR, NULL) resets Data Terminal Ready (DTR). This function is supported by 4.4BSD. See page 239 for details of the DTR signal.

5 February 2005 02:09

Chapter 15: Terminal Drivers

261

TIOCMSET
The call ioctl (fd, TIOCMSET, int *state) sets modem state. It is supported by 4.4BSD, SunOS and System V.4, but not all terminals support this call. *state is a bit map representing the parameters listed in table Table 15-13:
Table 1513: TIOCMSET and TIOCMGET state bits Parameter TIOCM_LE TIOCM_DTR TIOCM_RTS TIOCM_ST TIOCM_SR TIOCM_CTS TIOCM_CAR TIOCM_CD TIOCM_RNG TIOCM_RI TIOCM_DSR

meaning Line Enable Data Terminal Ready Request To Send Secondary Transmit Secondary Receive Clear To Send Carrier Detect Carrier Detect (synonym) Ring Indication Ring Indication (synonym) Data Set Ready

TIOCMGET
The call ioctl (fd, TIOCMGET, int *state) returns the current state of the terminal modem lines. See the description of TIOCMSET for the use of the bit mapped variable state.

TIOCMBIS
The call ioctl (fd, TIOCMBIS, int *state) sets the modem state in the same manner as TIOMSET, but instead of setting the state bits unconditionally, each bit is logically ored with the current state.

TIOCMBIC
The call ioctl (fd, TIOCMBIC, int *state) clears the modem state: each bit set in the bitmap state is reset in the modem state. The other state bits are not affected.

TCSBRK
The call ioctl (fd, TCSBRK, int nobreak) drains the output queue and then sends a break if nobreak is not set. This function is supported in System V and SunOS. In contrast to the 4.4BSD function TIOCSBRK, TCSBRK resets the break condition automatically.

5 February 2005 02:09

262

TCXONC
The call ioctl (fd, TCXONC, int type) species ow control. It is supported in System V and SunOS. Table 15-14 shows the possible values of type.
Table 1514: TCXONC and tcow type bits Parameter TCOOFF TCOON TCIOFF TCION value 0 1 2 3

meaning suspend output restart suspended output suspend input restart suspended input

Not all drivers support input ow control via TCXONC.

Queue control
TIOCOUTQ
The call ioctl (fd, TIOCOUTQ, int *num) sets the current number of characters in the output queue to *num. This function is supported by BSD and SunOS.

TIOCSTI
The call ioctl (fd, TIOCSTI, char *cp) simulates typed input. It inserts the character at *cp into the input queue. This function is supported by BSD and SunOS.

TIOCSTOP
The call ioctl (fd, TIOCSTOP, NULL) stops output on the terminal. Its like typing CTRL-S at the keyboard. This function is supported by 4.4BSD.

TIOCSTART
The call ioctl (fd, TIOCSTART, NULL) restarts output on the terminal, like typing CTRLQ at the keyboard. This function is supported by 4.4BSD.

TIOCDRAIN
The call ioctl (fd, TIOCDRAIN, NULL) suspends process execution until all output is drained. This function is supported by 4.4BSD.

5 February 2005 02:09

Chapter 15: Terminal Drivers

263

TIOCFLUSH
The call ioctl (fd, TIOCFLUSH, int *what) ushes the input and output queues. This function is supported by 4.4BSD, System V.3 and the Seventh Edition. The System V.3 and Seventh Edition implementations ignore the parameter what and ush both queues. 4.4BSD ushes the queues if the corresponding bits FREAD and FWRITE are set in *what. If no bits are set, it clears both queues.

TCFLSH
The call ioctl (fd, TCFLSH, int type) ushes the input or output queues, depending on the ags dened in Table 15-15.
Table 1515: TCFLSH type bits Parameter TCIFLUSH TCOFLUSH TCIOFLUSH value 0 1 2

meaning ush the input queue ush the output queue ush both queues

This function is supported by System V. It does the same thing as TIOCFLUSH, but the semantics are different.

Session control
TIOCGPGRP
The call ioctl (fd, TIOCGPGRP, pid_t *tpgrp) sets *tpgrp to the ID of the current process group with which the terminal is associated. 4.4BSD uses this call to implement the function tcgetpgrp.

TIOCSPGRP
The call ioctl (fd, TIOCSPGRP, pid_t *tpgrp) associates the terminal with the process group tpgrp. 4.4BSD uses this call to implement the function tcsetpgrp.

TIOCSCTTY
TIOCSCTTY makes the terminal the controlling terminal for the process. This function is supported by BSD and SunOS systems. On BSD systems, the call is ioctl (fd, TIOCSCTTY, NULL) and on SunOS systems it is ioctl (fd, TIOCSCTTY, int type). Normally the controlling terminal will be set only if no other process already owns it. In those implementations that support type the superuser can set type to 1 in order to force the takeover of the terminal, even if another process owns it. In 4.4BSD, you would rst use the revoke system call (see Chapter 14, File systems, page 213) to force a close of all le descriptors associated with the le.

5 February 2005 02:09

264

System V and older versions of BSD have no equivalent of this function. In these systems, when a process group leader without a controlling terminal opens a terminal, it automatically becomes the controlling terminal. There are methods to ovverride this behaviour: in System V, you set the ag O_NOCTTY when you open ther terminal. In old BSD versions, you subsequently release the control of the terminal with the TIOCNOTTY request, which well look at in the next section.

TIOCNOTTY
Traditionally, the rst time a process without a controlling terminal opened a terminal, it acquired that terminal as its controlling terminal. We saw in the section on TIOCSCTTY above that this is no longer the default behaviour in BSD, and that you can override it in System V. Older BSD versions, including SunOS, did not offer either of these choices. Instead, you had to accept that you acquired a controlling terminal, and then release the controlling terminal again with ioctl TIOCNOTTY. If you nd this code in a package, and your system doesnt support it, you can eliminate it. If your system is based on System V, you should check the call to open for the terminal and ensure that the ag O_NOCTTY is set. A second use for TIOCNOTTY was after a fork, when the child might want to relinquish the controlling terminal. This can also be done with setsid (see Chapter 12, Kernel dependencies, page 171).

TIOCGSID
The call ioctl (fd, TIOCGSID, pid_t *pid) stores the terminals session ID at pid. This function is supported by System V.4.

Miscellaneous functions
TIOCEXCL
The call ioctl (fd, TIOCEXCL, NULL) sets exclusive use on the terminal. No further opens are permitted except by root.

TIOCNXCL
The call ioctl (fd, TIOCNXCL, NULL) clears exclusive use of the terminal (see TIOCEXCL). Further opens are permitted.

TIOCCONS
The call ioctl (fd, TIOCCONS, int *on) sets the console le. If on points to a non-zero integer, kernel console output is redirected to the terminal specied in the call. If on points to zero, kernel console output is redirected to the standard console. This is usually used on work stations to redirect kernel messages to a particular window.

5 February 2005 02:09

Chapter 15: Terminal Drivers

265

TIOCGSOFTCAR
The call ioctl (fd, TIOCGSOFTCAR, int *set) sets *set to 1 if the terminal Data carrier detect (DCD) signal or the software carrier ag is asserted, and to 0 otherwise. This function is supported only in SunOS 4.X, and is no longer present in Solaris 2. See page 239 for a description of the DSR line.

TIOCSSOFTCAR
The call ioctl (fd, TIOCSSOFTCAR, int *set) is a method to fake a modem carrier detect signal. It resets software carrier mode if *set is zero and sets it otherwise. In software carrier mode, the TIOCGSOFTCAR call always returns 1; otherwise it returns the real value of the DCD interface signal. This function is supported only in SunOS 4.X, and is no longer present in Solaris 2.

termios functions
It should come as no surprise that people have long wanted a less bewildering interface to terminals than the ioctl calls that we looked at in the previous section. In POSIX.1, a number of new functions were introduced with the intent of bringing some sort of order into the chaos. A total of 8 new functions were introduced, split into three groups. In addition, a further 6 auxiliary functions were added: tcgetattr and tcsetattr get and set terminal attributes using struct termios. tcgetpgrp and tcsetpgrp get and set the program group ID. tcdrain, tcflow, tcflush and tcsendbreak manipulate the terminal hardware. cfgetispeed, cfsetispeed, cfgetospeed, cfsetospeed, cfsetspeed and cfmakeraw are auxiliary functions to manipulate termios entries.

These functions do not add new functionality, but attempt to provide a more uniform interface. In some systems, they are system calls, whereas in others they are library functions that build on the ioctl interface. If you are porting a package that uses termios, and your system doesnt supply it, you have the choice of rewriting the code to use ioctl calls, or you can use the 4.4BSD library calls supplied in the 4.4BSD Lite distribution (usr/src/lib/libc/gen/termios.c). In the following sections well look briey at each function.

Direct termios functions

tcgetattr
tcgetattr corresponds to TIOCGETA described on page 256. It returns the current termios state to term.
#include <termios.h> int tcgetattr (int fd, struct termios *term)

5 February 2005 02:09

266

tcsetattr
tcgetattr sets the current termios state from term.
#include <termios.h> int tcsetattr (int fd, int action, struct termios *t)

action can have one of the values listed in Table 15-16.

Table 1516: tcsetattr action ags Parameter TCSANOW TCSADRAIN TCSAFLUSH

meaning Change terminal parameters immediately. Corresponds to the ioctl request TIOCSETA. First drain output, then change the parameters. Used when changing parameters that affect output. Corresponds to the ioctl call TIOCSETAW. Discard any pending input, drain output, then change the parameters. Corresponds to ioctl call TIOCSETAF.

See page 257 for details of the corresponding ioctl interfaces. In addition, some implementations dene the parameter TCSASOFT: if this is specied in addition to one of the above ags, the values of the elds c_cflag, c_ispeed and c_ospeed are ignored. This is typically used when the device in question is not a serial line terminal.

tcgetpgrp
tcgetpgrp returns the ID of the current process group with which the terminal is associated. It corresponds to the ioctl call TIOCGPGRP described on page 263.
#include <sys/types.h> #include <unistd.h> pid_t tcgetpgrp (int fd);

tcsetpgrp
tcsetpgrp associates the terminal with the process group tpgrp. It corresponds to the ioctl call TIOCSPGRP described on page 263.
#include <sys/types.h> #include <unistd.h> int tcsetpgrp (int fd, pid_t pgrp_id);

tcdrain
tcdrain suspends the process until all output is drained. It corresponds to the ioctl call TIOCDRAIN described on page 262.

5 February 2005 02:09

Chapter 15: Terminal Drivers

#include <termios.h> int tcdrain (int fd);

267

tcflow
tcflow species ow control. It corresponds to the ioctl call TCXONC. See the description of TCXONC on page 262 for the meaning of the parameter action.
#include <termios.h> int tcflow (int fd, int action);

tcflush
tcflush ushes input or output queues for fd.
#include <termios.h> int tcflush (int fd, int action);

action may take the values shown in Table 15-17.

Table 1517: tcflush action bits Parameter TCIFLUSH TCOFLUSH TCIOFLUSH

meaning Flush data received but not read Flush data written but not transmitted Flush both data received but not read and data written but not transmitted

This function corresponds to the ioctl request TCFLSH described on page 263.

tcsendbreak
tcsendbreak sends a break indication on the line. This is equivalent to the ioctl request TCSBRK described on page 261.
#include <termios.h> int tcsendbreak (int fd, int len);

termios auxiliary functions

In addition to the termios functions in the previous section, a number of library functions manipulate termios struct entries. With one exception, they handle line speeds. They dont have any direct effect on the lineyou need a tcsetattr for thatbut they provide a link between the viewpoint of the application and the underlying implementation. There is still no agreement on how to represent line speeds. BSD systems use the bit rate as an integer and store it in the elds c_ispeed and c_ospeed. They leave it to the driver to explain it to the hardware, so you can effectively specify any speed the hardware is capable of handling. By contrast, System V still uses the small numeric indices that were used in the

5 February 2005 02:09

268

Seventh Edition* (see page 240), which allows the eld to be stored in 4 bits. They are located in the eld c_cflag. This is not a good idea, because these speeds are the only ones System V knows about. If you have a V.32bis, V.42bis modem that claims to be able to transfer data at up to 57,600 bps, you will not be able to take full advantage of its capabilities with System V. In addition, there is only one speed constant, which sets both the input and output speeds. The functions for setting input and output speed are effectively the same thing. In addition to these problems, SCO UNIX System V.3 further complicates the issue by providing the elds s_ospeed and s_ispeed in the struct termios. The functions cfsetispeed and cfsetospeed set these elds in addition to the four bits in c_cflag, but the functions cfgetispeed and cfgetospeed retrieve the values from c_cflags, so its not clear what use the elds c_ispeed and c_ospeed are intended to be. Setting the bit rates is thus not quite as simple as it might appear: the preprocessor variables B9600 and friends might not equate to the kind of constant that the termios implementation needs, and there is no designated place in the termios structure to store the bit rates. This problem is solved by the following functions, which are normally macros: speed_t cfgetispeed (struct termios *t) returns ts input speed in speed_t format. It is undened if the speed is not representable as speed_t. int cfsetispeed (struct termios *t, speed_t speed)sets ts input speed to the internal representation of speed. speed_t cfgetospeed (struct termios *t) returns ts output speed in speed_t format. The result is undened if the speed is not representable as speed_t. int cfsetospeed (struct termios *t, speed_t speed) sets ts output speed to the internal representation of speed. void cfsetspeed (struct termios *t, speed_t speed) sets both input and output speed to the internal representation of speed. void cfmakeraw (struct termios *t) sets the whole structure t to default values.

* These constants were originally the values that were written to the interface hardware to set the speed.

5 February 2005 02:09

Timekeeping
UNIX timekeeping is an untidy area, made more confusing by national and international laws and customs. Broadly, there are two kinds of functions: one group is concerned with getting and setting system times, and the other group is concerned with converting time representations between a bewildering number of formats. Before we start, well dene some terms: A time zone is a denition of the time at a particular location relative to the time at other locations. Most time zones are bound to political borders, and vary from one another in steps of one hour, although there are still a number of time zones that are offset from adjacent time zones by 30 minutes. Time zones tend to have three-letter abbreviations (TLAs) such as PST (Pacic Standard Time), EDT (Eastern Daylight Time), BST (British Summer Time), AET (Australian Eastern Time), MEZ (Mitteleuropische Zeit). As the example shows, you should not rely on the combination ST to represent Standard Time. UTC is the international base time zone, and has the distinction of being one of those abbreviations which nobody can expand. It means Universal Coordinated Time, despite the initials. It obviously doesnt stand for the French Temps Universel Coordonn either. It corresponds very closely, but not exactly, to Greenwich Mean Time (GMT), the local time in England in the winter, and is the basis of all UNIX timestamps. The result is that for most of us, UTC is not the current local time, though it might be close enough to be confusing or far enough away to be annoying. From the standpoint of UNIX, you can consider the Epoch to be the beginning of recorded history: its 00:00:00 UTC, 1 January 1970. All system internal dates are relative to the Epoch. Daylight Savings Time is a method of making the days appear longer in summer by setting the clocks forward, usually by one hour. Thus in summer, the sun appears to set one hour later than would otherwise be the case.

Even after clarifying these denitions, timekeeping remains a pain. Well look at the main problems in the following sections:

269

5 February 2005 02:09

270

Difficult to use
The time functions are not easy to use: to get them to do anything useful requires a lot of work. Youd think that UNIX would supply a primitive call upon which you could easily build, but unfortunately there isnt any such call, and the ones that are available do not operate in an intuitively obvious way. For example, there is no standard function for returning the current time in a useful format.

Implementations differ
There is no single system call that is supported across all platforms. Functions are implemented as system calls in some systems and as library functions in others. As a result, it doesnt make sense to maintain our distinction between system calls and library functions when it comes to timekeeping. In our discussion of the individual functions, well note which systems implement them as system calls and which as library calls.

Differing time formats

There are at least four different time formats: The system uses the time_t format, which represents the number of seconds since the Epoch. This format is not subject to time zones or daylight savings time, but it is accurate only to one second, which is not accurate enough for many applications. The struct timeval format is something like an extended time_t with a resolution of 1 microsecond:
#include <sys/time.h> struct timeval { long tv_sec; long tv_usec; };

/* seconds since Jan. 1, 1970 */ /* and microseconds */

It is used for a number of newer functions, such as gettimeofday and setitimer. Many library routines represent the calendar time as a struct tm. It is usually dened in /usr/include/time.h:
struct { int int int int int int int int int tm tm_sec; tm_min; tm_hour; tm_mday; tm_mon; tm_year; tm_wday; tm_yday; tm_isdst; /* /* /* /* /* /* /* /* /* seconds after the minute [0-60] */ minutes after the hour [0-59] */ hours since midnight [0-23] */ day of the month [1-31] */ months since January [0-11] */ years since 1900 */ days since Sunday [0-6] */ days since January 1 [0-365] */ Daylight Savings Time flag */

5 February 2005 02:09

Chapter 16: Timekeeping

long tm_gmtoff; char *tm_zone; }; /* offset from UTC in seconds */ /* timezone abbreviation */

271

Unlike time_t, a struct tm does not uniquely dene the time: it may be a UTC time, or it may be local time, depending on the time zone information for the system. Dates as a text string are frequently represented in a strange manner, for example Sat Sep 17 14:28:03 1994\n. This format includes a \n character, which is seldom needed often you will have to chop it off again.

Daylight Savings Time

The support for Daylight Savings Time was rudimentary in the Seventh Edition, and the solutions that have arisen since then are not completely compatible. In particular, System V handles Daylight Savings Time via environment variables, so one users view of time could be different from the next. Recent versions of BSD handle this via a database that keeps track of local regulations.

National time formats

Printable representations of dates and times are very much a matter of local customs. For example, the date 9/4/94 (in the USA) would be written as 4/9/94 in Great Britain and 04.09.94 in Germany. The time written as 4:23 pm in the USA would be written 16.23 in France. Things get even worse if you want to have the names of the days and months. As a result, many timekeeping functions refer to the locale kept by ANSI C. The locale describes country-specic information. Since it does not vary from one system to the next, we wont look at it in more detailsee POSIX Programmers Guide, by Donald Lewine, for more information.

Global timekeeping variables

A number of global variables dene various aspects of timekeeping: The variable timezone, which is used in System V and XENIX, species the number of minutes that the standard time zone is west of Greenwich. It is set from the environment variable TZ, which has a rather bizarre syntax. For example, in Germany daylight savings time starts on the last Sunday of March and ends on the last Sunday of September (not October as in some other countries, including the USA). To tell the system about this, you would use the TZ string
MEZ-1MSZ-2;M3.5,M9.5

This states that the standard time zone is called MEZ, and that it is one hour ahead of UTC, that the summer time zone is called MSZ, and that it is two hours ahead of UTC. Summer time begins on the (implied Sunday of the) fth week in March and ends in the fth week of September.

5 February 2005 02:09

272

The punctuation varies: this example comes from System V.3, which requires a semicolon in the indicated position. Other systems allow a comma here, which works until you try to move the information to System V.3. The variable altzone, used in SVR4 and XENIX, species the number of minutes that the Daylight Savings Time zone is west of Greenwich. The variable daylight, used in SVR4 and XENIX, indicates that Daylight Savings Time is currently in effect. The variable tzname, used in BSD, SVR4 and XENIX, is a pointer to two strings, specifying the name of the standard time zone and the Daylight Savings Time zone respectively.

In the following sections well look at how to get the current time, how to set the current time, how to convert time values, and how to suspend process execution for a period of time.

Getting the current time

The system supplies the current time via the system calls time or gettimeofdayonly one of these is a system call, but the system determines which one it is.

time
#include <sys/types.h> #include <time.h> time_t time (time_t *tloc);

time returns the current time in time_t form, both as a return value and at tloc if this is not NULL. time is implemented as a system call in System V and as a library function (which calls gettimeofday) in BSD. Since it returns a scalar value, a call to time can be used as a parameter to functions like localtime or ctime.

ftime
ftime is a variant of time that returns time information with a resolution of one millisecond. It originally came from 4.2BSD, but is now considered obsolete.
#include <sys/types.h> #include <sys/timeb.h> typedef long time_t; struct timeb { time_t time; unsigned short millitm; short timezone; short dstflag; /* (typically) */

/* /* /* /*

the same time returned by time */ Milliseconds */ System default time zone */ set during daylight savings time */

5 February 2005 02:09

Chapter 16: Timekeeping

}; struct timeb *ftime (struct timeb *tp);

273

The timezone returned is the system default, possibly not what you want. System V.4 deprecates* the use of this variable as a result. Depending on which parameters are actually used, there are a number of alternatives to ftime. In many cases, time supplies all you need. However, time is accurate only to one second. On some systems, you may be able to dene ftime in terms of gettimeofday, which returns the time of the day with a 1 microsecond resolutionsee the next section. On other systems, unfortunately, the system clock does not have a ner resolution than one second, and you are stuck with time.

gettimeofday
#include <sys/time.h> struct timeval { long tv_sec; long tv_usec; };

/* seconds since Jan. 1, 1970 */ /* and microseconds */

int gettimeofday (struct timeval *tp, struct timezone *tzp); /* (BSD) */ int gettimeofday (struct timeval *tp); /* (System V.4) */

gettimeofday returns the current system time, with a resolution of 1 microsecond, to tp. The name is misleading, since the struct timeval representation does not relate to the time of day. Many implementations ignore tzp, but others, such as SunOS 4, return time zone information there. In BSD, gettimeofday is a system call. In some versions of System V.4 it is emulated as a library function dened in terms of time, which limits its resolution to 1 second. Other versions of System V appear to have implemented it as a system call, though this is not documented.

* The term deprecate is a religious term meaning to seek to avert by prayer. Nowadays used to indicate functionality that the implementors or maintainers wish would go away. This term seems to have come from Berkeley. To quote the New Hackers Dictionary: :deprecated: adj. Said of a program or feature that is considered obsolescent and in the process of being phased out, usually in favor of a specied replacement. Deprecated features can, unfortunately, linger on for many years. This term appears with distressing frequency in standards documents when the committees writing the documents realize that large amounts of extant (and presumably happily working) code depend on the feature(s) that have passed out of favor. See also {dusty deck}.

5 February 2005 02:09

274

Setting the current time

Setting the system time is similar to getting it, except that for security reasons only the superuser (root) is allowed to perform the function. It is normally executed by the date program.

adjtime
#include <sys/time.h> int adjtime (struct timeval *delta, struct timeval *olddelta);

adjtime makes small adjustments to the system time, and is intended to help synchronize time in a network. The adjustment is made graduallythe system slows down or speeds up the passage of time by a fraction of a percent until it has made the correction, in order not to confuse programs like cron which are watching the time. As a result, if you call adjtime again, the previous adjustment might still not be complete; in this case, the remaining adjustment is returned in olddelta. adjtime was introduced in 4.3BSD and is also supported by System V. It is implemented as a system call in all systems.

settimeofday
#include <sys/time.h> int gettimeofday (struct timeval *tp, struct timezone *tzp); int settimeofday (struct timeval *tp, struct timezone *tzp);

settimeofday is a BSD system call that is emulated as a library function in System V.4. It sets the current system time to the value of tp. The value of tzp is no longer used. In System V, this call is implemented in terms of the stime system call, which sets the time only to the nearest second. If you really need to set the time more accurately in System V.4, you can use adjtime.

stime
#include <unistd.h> int stime (const time_t *tp);

stime sets the system time and date. This is the original Seventh Edition function that is still available in System V. It is not supported in BSDuse settimeofday instead on BSD systems.

Converting time values

As advertised, there are a large number of time conversion functions, made all the more complicated because many are supported only on specic platforms. All are library functions. Many return pointers to static data areas that are overwritten by the next call. Solaris attempts to solve this problem with versions of the functions with the characters _r (for reentrant)

5 February 2005 02:09

Chapter 16: Timekeeping

275

appended to their names. These functions use a user-supplied buffer to store the data they return.

strftime
#include <sys/types.h> #include <time.h> #include <string.h> size_t strftime (char *s, size_t maxsize, char *format, struct tm *tm);

strftime converts the time at tm into a formatted string at s. format species the format of the resultant string, which can be no longer than maxsize characters. format is similar to the format strings used by printf, but contains strings related to dates. strftime has a rather strange return value: if the complete date string, including the terminating NUL character, ts into the space provided, it returns the length of the stringotherwise it returns 0, which implies that the date string has been truncated. strftime is available on all platforms and is implemented as a library function. System V.4 considers ascftime and cftime to be obsolete. The man pages state that strftime should be used instead.

strptime
#include <time.h> char *strptime (char *buf, char *fmt, struct tm *tm);

strptime is a library function supplied with SunOS 4. It converts the date and time string buf into a struct tm value at tm. This call bears the same relationship to scanf that strftime bears to printf.

ascftime
#include <sys/types.h> #include <time.h> int ascftime (char *buf, char *fmt, tm *tm);

ascftime converts the time at tm into a formatted string at buf. format species the format of the resultant string. This is effectively the same function as strftime, except that there is no provision to supply the maximum length of buf. ascftime is available on all platforms and is implemented as a library function.

asctime and asctime_r

#include <sys/types.h> #include <time.h> char *asctime (const struct tm *tm);

5 February 2005 02:09

276
char *asctime_r (const struct tm *tm, char *buf, int buflen);

asctime converts a time in struct tm* format into the same kind of string that is returned by ctime. asctime is available on all platforms and is always a library function. asctime_r is a version of asctime that returns the string to the user-provided buffer res, which must be at least buflen characters long. It returns the address of res. It is supplied as a library function on Solaris systems.

cftime
#include <sys/types.h> #include <time.h> int cftime (char *buf, char *fmt, time_t *clock);

cftime converts the time at clock into a formatted string at buf. format species the format of the resultant string. This is effectively the same function as strftime, except that there is no provision to supply the maximum length of buf, and the time is supplied in time_t format. cftime is available on all platforms and is implemented as a library function.

ctime and ctime_r

#include <sys/types.h> #include <time.h> extern char *tzname[2]; char *ctime (const time_t *clock); char *ctime_r (const time_t *clock, char *buf, int buflen);

ctime converts the time clock into a string in the form Sat Sep 17 14:28:03 1994\n, which has the advantage of consistency: it is not a normal representation anywhere in the world, and immediately brands any printed output with the word UNIX. It uses the environment variable TZ to determine the current time zone. You can rely on the string to be exactly 26 characters long, including the nal \0, and to contain that irritating \n at the end. ctime is available on all platforms and is always a library function. ctime_r is a version of ctime that returns its result in the buffer pointed to by buf. The length is limited to buflen bytes. ctime_r is available on Solaris platforms as a library function.

dysize
#include <time.h> int dysize (int year);

dysize return the number of days in year. It is supplied as a library function in SunOS 4.

5 February 2005 02:09

Chapter 16: Timekeeping

277

gmtime and gmtime_r

#include <time.h> struct tm *gmtime (const time_t *clock); struct tm *gmtime_r (const time_t *clock, struct tm *res);

gmtime converts a time in time_t format into struct tm* format, like localtime. As the name suggests, however, it does not account for local timezonesit returns a UTC time (this was formerly called Greenwich Mean Time, thus the name of the function). gmtime is available on all platforms and is always a library function. gmtime_r is a version of gmtime that returns the string to the user-provided buffer res. It returns the address of res. It is supplied as a library function on Solaris systems.

localtime and localtime_r

#include <time.h> struct tm *localtime (const time_t *clock); struct tm *localtime_r (const time_t *clock, struct tm *res);

localtime converts a time in time_t format into struct tm* format. Like ctime, it uses the time zone information in tzname to convert to local time. localtime is available on all platforms and is always a library function. localtime_r is a version of localtime that returns the string to the user-provided buffer res. It returns the address of res. It is supplied as a library function on Solaris systems.

mktime
#include <sys/types.h> #include <time.h> time_t mktime (struct tm *tm);

mktime converts a local time in struct tm format into a time in time_t format. It does not use tzname in the conversion it uses the information at tm->tm_zone instead. In addition to converting the time, mktime also sets the members wday (day of week) and yday (day of year) of the input struct tm to agree with day, month and year. tm->tm_isdst determines whether Daylight Savings Time is applicable: if it is > 0, mktime assumes Daylight Savings Time is in effect. If it is 0, it assumes that no Daylight Savings Time is in effect. If it is < 0, mktime tries to determine whether Daylight Savings Time is in effect or not. It is often wrong.

mktime is available on all platforms and is always a library function.

5 February 2005 02:09

278

timegm
#include <time.h> time_t timegm (struct tm *tm);

timegm converts a struct tm time, assumed to be UTC, to the corresponding time_t value. This is effectively the same thing as mktime with the time zone set to UTC, and is the converse of gmtime. timegm is a library function supplied with SunOS 4.

timelocal
#include <time.h> time_t timelocal (struct tm *tm);

timelocal converts a struct tm time, assumed to be local time, to the corresponding time_t value. This is similar to mktime, but it uses the local time zone information instead of the information in tm. It is also the converse of localtime. timelocal is a library function supplied with SunOS 4.

difftime
#include <sys/types.h> #include <time.h> double difftime (time_t time1, time_t time0);

difftime returns the difference in seconds between two time_t values. This is effectively the same thing as (int) time1 - (int) time0. difftime is a library function available on all platforms.

timezone
#include <time.h> char *timezone (int zone, int dst);

timezone returns the name of the timezone that is zone minutes west of Greenwich. If dst is non-0, the name of the Daylight Savings Time zone is returned instead. This call is obsolete it was used at a time when time zone information was stored as the number of minutes west of Greenwich. Nowadays the information is stored with a time zone name, so there should be no need for this function.

tzset
#include <time.h> void tzset ();

5 February 2005 02:09

Chapter 16: Timekeeping

279

tzset sets the value of the internal variables used by localtime to the values specied in the environment variable TZ. It is called by asctime. In System V, it sets the value of the global variable daylight. tzset is a library function supplied with BSD and System V.4.

tzsetwall
#include <time.h> void tzsetwall ();

tzsetwall sets the value of the internal variables used by localtime to the default values for the site. tzsetwall is a library function supplied with BSD and System V.4.

Suspending process execution

Occasionally you want to suspend process execution for a short period of time. For example, the tail program with the -f ag waits until a le has grown longer, so it needs to relinquish the processor for a second or two between checks on the le status. Typically, this is done with sleep. However, some applications need to specify the length of time more accurately than sleep allows, so a couple of alternatives have arisen: nap suspends execution for a number of milliseconds, and usleep suspends it for a number of microseconds.

nap
nap is a XENIX variant of sleep with ner resolution:
#include <time.h> long nap (long millisecs);

nap suspends process execution for at least millisecs milliseconds. In practice, the XENIX clock counts in intervals of 20 ms, so this is the maximum accuracy with which you can specify millisecs. You can simulate this function with usleep (see page 281 for more details).

setitimer
BSD systems and derivatives maintain three (possibly four) interval timers: A real time timer, ITIMER_REAL, which keeps track of real elapsed time. A virtual timer, ITIMER_VIRTUAL, which keeps track of process execution time, in other words the amount of CPU time that the process has used. A proler timer, ITIMER_PROF, which keeps track of both process execution time and time spent in the kernel on behalf of the process. As the name suggests, it is used to implement proling tools.

5 February 2005 02:09

280

A real time proler timer, ITIMER_REALPROF, used for proling Solaris 2.X multithreaded processes.

These timers are manipulated with the system calls getitimer and setitimer:
#include <sys/time.h> struct timeval { long tv_sec; long tv_usec; }; struct itimerval { struct timeval it_interval; struct timeval it_value; };

/* seconds */ /* and microseconds */

/* timer interval */ /* current value */

int getitimer (int which, struct itimerval *value); int setitimer (int which, struct itimerval *value, struct itimerval *ovalue);

setitimer sets the value of a specic timer which to value, and optionally returns the previous value in ovalue if this is not a NULL pointer. getitimer just returns the current value of the timer to value. The resolution is specied to an accuracy of 1 microsecond, but it is really limited to the accuracy of the system clock, which is more typically in the order of 10 milliseconds. In addition, as with all timing functions, there is no guarantee that the process will be able to run immediately when the timer expires. In the struct itimerval, it_value is the current value of the timer, which is decremented depending on type as described above. When it_value is decremented to 0, two things happen: a signal is generated, and it_value is reloaded from it_interval. If the result is 0, no further action occurs; otherwise the system continues decrementing the counter. In this way, one call to setitimer can cause the system to generate continuous signals at a specied interval. The signal that is generated depends on the timer. Heres an overview:
Table 161: setitimer signals

Timer Real time Virtual Profiler Real-time profiler1

1

Signal
SIGALRM SIGVTALRM SIGPROF SIGPROF

Only Solaris 2.x

The only timer youre likely to see is the real time timer. If you dont have it, you can fake it with alarm. In System V.4, setitimer is implemented as a library function that calls an undocumented system call. See The Magic Garden explained: The Internals of UNIX System

5 February 2005 02:09

Chapter 16: Timekeeping

281

V Release 4, by Berny Goodheart and James Cox, for more details. setitimer is used to implement the library routine usleep.

sleep
#include <unistd.h> unsigned sleep (u_int seconds);

The library routine sleep suspends program execution for approximately seconds seconds. It is available on all UNIX platforms.

usleep
usleep is a variant of sleep that suspends program execution for a very short time:
#include <unistd.h> void usleep (u_int microseconds);

usleep sleeps for at least microseconds microseconds. It is supplied on BSD and System V.4 systems as a library function that uses the setitimer system call.

select and poll

If your system doesnt supply any timing function with a resolution of less than one second, you might be able to fake it with the functions select or poll. select can wait for nothing if you ask it to, and since the timeout is specied as a struct timeval (see page 270), you can specify times down to microsecond accuracy. You can use poll in the same way, except that you species its timeout value in milliseconds. For example,
void usleep (int microseconds) { struct timeval timeout; timeout.tv_usec = microseconds % 1000000; timeout.tv_sec = microseconds / 1000000; select (0, NULL, NULL, NULL, &timeout); } or void usleep (int microseconds) { poll (0, NULL, microseconds / 1000); }

5 February 2005 02:09

Header les
When the C language was young, header les were required to dene structures and occasionally to specify that a function did something out of the ordinary like taking a double parameter or returning a float result. Then ANSI C and POSIX came along and changed all that. Header les seem a relatively simple idea, but in fact they can be a major source of annoyance in porting. In particular: ANSI and POSIX.1 have added a certain structure to the usage of header les, but there are still many old-fashioned headers out there. ANSI and POSIX.1 have also placed more stringent requirements on data types used in header les. This can cause conicts with older systems, especially if the author has commited the sin of trying to out-guess the header les. C++ has special requirements of header les. If your header les dont full these requirements, the GNU protoize program can usually x them. There is still no complete agreement on the names of header les, or in which directories they should be placed. In particular, System V.3 and System V.4 frequently disagree as to whether a header le should be in /usr/include or in /usr/include/sys.

ANSI C, POSIX.1, and header files

ANSI C and POSIX.1 have had a far-reaching effect on the structure of system header les. Well look at the changes in the C language in more detail in Chapter 20, Compilers. The following points are relevant to the use of header les: ANSI C prefers to have an ANSI-style prototype for every function call it encounters. If it doesnt nd one, it cant check the function call semantics as thoroughly, and it may issue a warning. Its a good idea to enable all such warnings, but this kind of message makes it difcult to recognize the real errors hiding behind the warnings. In C++, the rules are even stricter: if you dont have a prototype, its an error and your source le doesnt compile.

283

5 February 2005 02:09

284

To do a complete job of error checking, ANSI C requires the prototype in the new, embedded form:
int foo (char *zot, int glarp);

and not
int foo (zot, glarp); char *zot;

Old C compilers dont understand this new kind of prototype. Header les usually contain many denitions that are not part of POSIX.1. A mechanism is needed to disable these denitions if you are compiling a program intended to be POSIX.1 compatible.*

The result of these requirements is spaghetti header les: you frequently see things like this excerpt from the header le stdio.h in 4.4BSD:
/* * Functions defined in ANSI C standard. */ __BEGIN_DECLS void clearerr __P((FILE *)); int fclose __P((FILE *)); #if !defined(_ANSI_SOURCE) && !defined(_POSIX_SOURCE) extern int sys_nerr; /* perror(3) external variables */ extern __const char *__const sys_errlist[]; #endif void perror __P((const char *)); __END_DECLS /* * Functions defined in POSIX 1003.1. */ #ifndef _ANSI_SOURCE #define L_cuserid 9 /* size for cuserid(); UT_NAMESIZE + 1 */ #define L_ctermid 1024 /* size for ctermid(); PATH_MAX */ __BEGIN_DECLS char *ctermid __P((char *)); __END_DECLS #endif /* not ANSI */ /* * Routines that are purely local. */

* Writing your programs to conform to POSIX.1 may be a good idea if you want them to run on as many platforms as possible. On the other hand, it may also be a bad idea: POSIX.1 has very rudimentary facilities in some areas. You may nd it more conning than is good for your program.

5 February 2005 02:09

Chapter 17: Header les

#if !defined (_ANSI_SOURCE) && !defined(_POSIX_SOURCE) __BEGIN_DECLS char *fgetln __P((FILE *, size_t *)); __END_DECLS

285

Well, it does look vaguely like C, but this kind of header le scares most people off. A number of conicts have led to this kind of code: The ANSI C library and POSIX.1 carefully dene a subset of the total available functionality. If you want to abide strictly to the standards, any extension must be agged as an error, even if it would work. The C++ language has a different syntax from C, but both languages share a common set of header les.

These solutions have caused new problems, which well examine in this chapter.

ANSI and POSIX.1 restrictions

Most current UNIX implementations do not conform completely with POSIX.1 and ANSI C, and every implementation offers a number of features that are not part of either standard. A program that conforms with the standards must not use these features. You can specify that you wish your program to be compliant with the standards by dening the preprocessor variables _ANSI_SOURCE or _POSIX_SOURCE, which maximizes the portability of the code. It does this by preventing the inclusion of certain denitions. In our example, the array sys_errlist, (see Chapter 18, Function libraries, page 298), is not part of POSIX.1 or ANSI, so the denition is not included if either preprocessor variable is set. If we refer to sys_errlist anyway, the compiler signies an error, since the array hasnt been declared. Similarly, L_cuserid is dened in POSIX.1 but not in ANSI C, so it is dened only when _POSIX_SOURCE is dened and _ANSI_SOURCE is not dened.

Declarations for C++

C++ has additional requirements of symbol naming: function overloading allows different functions to have the same name. Assemblers dont think this is funny at all, and neither do linkers, so the names need to be changed to be unique. In addition, the names need to somehow reect the class to which they belong, the kind of parameters that the function takes and the kind of value it returns. This is done by a technique called function name encoding, usually called function name mangling. The parameter and return value type information is appended to the function name according to a predetermined rule. To illustrate this, lets look at a simple function declaration:
double Internal::sense (int a, unsigned char *text, Internal &p, ...);

First, two underscores are appended to the name of the function. With the initial underscore we get for the assembler, the name is now _sense__.

5 February 2005 02:09

286

Then the class name, Internal is added. Since the length of the name needs to be specied, this is put in rst: _sense__8Internal. Next, the parameters are encoded. Simple types like int and char are abbreviated to a single character (in this case, i and c. If they have modiers like unsigned, these, too, are encoded and precede the type information. In this case, we get just plain i for the int parameter, and PUc (a Pointer to Unsigned characters for the second parameter: _sense__8InternaliPUc. Class or structure references again cant be coded ahead of time, so again the length of the name and the name itself is used. In this case, we have a reference, so the letter R is placed in front of the name: _sense__8InternaliPUcR8Internal. Finally, the ellipses are specied with the letter e: _sense__8InternaliPUcR8Internale.

For more details on function name mangling, see The Annotated C++ Reference Manual by Margaret Ellis and Bjarne Stroustrup. This difference in naming is a problem when a C++ program really needs to call a function written in C. The name in the object le is not mangled, and so the C++ compiler must not output a reference to a mangled name. Theoretically, there could be other differences between C++ calls and C calls that the compiler also needs to take into account. You cant just assume that a function written in another language adheres to the same conventions, so you have to tell it when a called function is written according to C conventions rather than according to C++ conventions. This is done with the following elegant construct:
extern "C" { char *ctermid (char *); };

In ANSI C, the same declaration would be

char *ctermid (char *);

and in K&R C it would be

char *ctermid ();

It would be a pain to have a separate set of header les for each version. Instead, the implementors dened preprocessor variables which evaluate to language constructs for certain places: __BEGIN_DECLS is dened as extern C { for C++ and nothing otherwise. __END_DECLS is dened as }; for C++ and nothing otherwise. __P(foo) is dened as foo for C++ and ANSI C, and nothing otherwise. This is the reason why the arguments to __P() are enclosed in double parentheses: the outside level of parentheses gets stripped by the preprocessor.

5 February 2005 02:09

Chapter 17: Header les

287

In this implementation, sys/cdefs.h denes these preprocessor variables. What happens if sys/cdefs.h isnt included before stdio.h? Lots of error messages. So one of the rst lines in stdio.h is #include <sys/cdefs.h>. This is not the only place that sys/cdefs.h is included: in this particular implementation, from 4.4BSD, it is included from assert.h, db.h, dirent.h, err.h, fnmatch.h, fstab.h, fts.h, glob.h, grp.h, kvm.h, locale.h, math.h, netdb.h, nlist.h, pwd.h, regex.h, regexp.h, resolv.h, runetype.h, setjmp.h, signal.h, stdio.h, stdlib.h, string.h, time.h, ttyent.h, unistd.h, utime.h and vis.h. This places an additional load on the compiler, which reads in a 100 line denition le multiple times. It also creates the possibility for compiler errors. sys/cdefs.h denes a preprocessor variable _CDEFS_H_ in order to avoid this problem: after the obligatory UCB copyright notice, it starts with
#ifndef _CDEFS_H_ #define _CDEFS_H_ #if defined(__cplusplus) #define __BEGIN_DECLS extern "C" { #define __END_DECLS }; #else #define __BEGIN_DECLS #define __END_DECLS #endif

This is a common technique introduced by ANSI C: the preprocessor only processes the body of the header le the rst time. After that, the preprocessor variable _CDEFS_H_ is dened, and the body will not be processed again. There are a couple of things to note about this method: There are no hard and fast rules about the naming and denition of these auxiliary variables. The result is that not all header les use this technique. For example, in FreeBSD 1.1, the header le machine/limits.h denes a preprocessor variable _MACHINE_LIMITS_H and only interprets the body of the le if this preprocessor variable was not set on entry. BSD/OS 1.1, on the other hand, does not. The same header le is present, and the text is almost identical, but there is nothing to stop you from including and interpreting machine/limits.h multiple times. The result can be that a package that compiles just ne under FreeBSD may fail to compile under BSD/OS. The ANSI standard denes numerous standard preprocessor variables to ensure that header les are interpreted only the rst time they are included. The variables all start with a leading _, and the second character is either another _ or an upper-case letter. Its a good idea to avoid using such symbols in your sources. We could save including sys/cdefs.h multiple times by checking _CDEFS_H_ before including it. Unfortunately, this would establish an undesireable relationship between the two les: if for some reason it becomes necessary to change the name of the preprocessor variable, or perhaps to give it different semantics (like giving it different values at different times, instead of just being dened), you have to go through all the header les that refer to the preprocessor variable and modify them.

5 February 2005 02:09

288

ANSI header files

The ANSI C language denition, also called Standard C, was the rst to attempt some kind of standardization of header les. As far as it goes, it works well, but unfortunately it covers only a comparatively small number of header les. In ANSI C, The only header les you should need to include are assert.h, ctype.h, errno.h, oat.h, limits.h, locale.h, math.h, setjmp.h, signal.h, stdarg.h, stddef.h, stdio.h, stdlib.h, string.h and time.h. You may include headers in any order. You may include any header more than once. Header les do not depend on other header les. Header les do not include other header les.

If you can get by with just the ANSI header les, you wont have much trouble. Unfortunately, real-life programs usually require headers that arent covered by the ANSI standard.

Type information
A large number of system and library calls return information which can be represented in a single machine word. The machine word of the PDP-11, on which the Seventh Edition ran, was only 16 bits wide, and in some cases you had to squeeze the value to get it in the word. For example, the Seventh Edition le system represented an inode number in an int, so each le system could have only 65536 inodes. When 32-bit machines were introduced, people quickly took the opportunity to extend the length of these elds, and modern le systems such as ufs or vxfs have 32 bit inode numbers. These changes were an advantage, but they bore a danger with them: nowadays, you cant be sure how long an inode number is. Current systems really do have different sized elds for inode numbers, and this presents a portability problem. Inodes arent the only thing that has changed: consider the following structure denition, which contains information returned by system calls:
struct process_info { long pid; long start_time; long owner; long log_file; long log_file_pos; short file_permissions; short log_file_major; short log_file_minor; short inode; }

/* /* /* /* /* /* /* /* /*

process number */ time process was started, from time () */ user ID of owner */ file number of log file */ current position in log file */ default umask */ major device number for log file */ minor device number */ inode number of log file */

On most modern systems, the longs take up 32 bits and the shorts take up 16 bits. Because

5 February 2005 02:09

Chapter 17: Header les

289

of alignment constraints, we put the longest data types at the front and the shortest at the end (see Chapter 11, Hardware dependencies, page 158 for more details). And for older systems, these elds are perfectly adequate. But what happens if we port a program containing this structure to a 64 bit machine running System V.4 and vxfs? Weve already seen that the inode numbers are now 32 bits long, and System V.4 major and minor device numbers also take up more space. If you port this package to 4.4BSD, the eld log_file_pos needs to be 64 bits long. Clearly, its an oversimplication to assume that any particular kind of value maps to a short or a long. The correct way to do this is to dene a type that describes the value. In modern C, the structure above becomes:
struct process_info { pid_t pid; time_t start_time; uid_t owner; long log_file; pos_t log_file_pos; mode_t file_permissions; short log_file_major; short log_file_minor; inode_t inode; }

/* /* /* /* /* /* /* /* /*

process number */ time process was started, from time () */ user ID of owner */ file number of log file */ current position in log file */ default umask */ major device number for log file */ minor device number */ inode number of log file */

Its important to remember that these type denitions are all in the mind of the compiler, and that they are dened in a header le, which is usually called sys/types.h: the system handles them as integers of appropriate length. If you dene them in this manner, you give the compiler an opportunity to catch mistakes and generate more reliable code. Check your man pages for the types of the arguments on your system if you run into trouble. In addition, Appendix A, Comparative reference to UNIX data types, contains an overview of the more common types used in UNIX systems.

Classes of header files

If you look at the directory hierarchy /usr/include, you may be astounded by the sheer number of header les, over 400 of them on a typical UNIX system. Fortunately, many of them are in subdirectories, and you usually wont have to worry about them, except for one subdirectory: /usr/include/sys.

/usr/include/sys
In early versions of UNIX, this directory contained the header les used for compiling the kernel. Nowadays, this directory is intended to contain header les that relate to the UNIX implementation, though the usage varies considerably. You will frequently nd les that directly include les from /usr/include/sys. In fact, it may come as a surprise that this is not supposed to be necessary. Often you will also see code like

5 February 2005 02:09

290
#ifdef USG #include <sys/err.h> #else #include <err.h> #endif /* System V */ /* non-System V system */

This simplied example shows what you need to do because System V keeps the header le err.h in /usr/include/sys, whereas other avours keep it in /usr/include. In order to include the le correctly, the source code needs to know what kind of system it is running on. If it guesses wrong (for example, if USG is not dened when it should be) or if the author of the package didnt allow for System V, either out of ignorance, or because the package has never been compiled on System V before, then the compilation will fail with a message about missing header les. Frequently, the decisions made by the kind of code in the last example are incorrect. Some header les in System V have changed between System V.3 and System V.4. If, for example, you port a program written for System V.4 to System V.3, you may nd things like
#include <wait.h>

This will fail in most versions of System V.3, because there is no header le /usr/include/wait.h; the le is called /usr/include/sys/wait.h. There are a couple of things you could do here: You could start the compiler with a supplementary -I/usr/include/sys, which will cause it to search /usr/include/sys for les specied without any pathname component. The problem with this approach is that you need to do it for every package that runs into this problem. You could consider doing what System V.4 does in many cases: create a le called /usr/include/wait.h that contains just an obligatory copyright notice and an #include directive enclosed in #ifdefs:
/* THIS IS PUBLISHED NON-PROPRIETARY SOURCE CODE OF OREILLY */ /* AND ASSOCIATES Inc. */ /* The copyright notice above does not evidence any actual or */ /* intended restriction on the use of this code. */ #ifndef _WAIT_H #define _WAIT_H #include <sys/wait.h> #endif

Problems with header files

Its fair to say that no system is supplied with completely correct system header les. Your system header les will probably suffer from at least one of the following problems: Incorrect naming. The header les contain the denitions you need, but they are not in the place you would expect. Incomplete denitions. Function prototypes or denitions of structures and constants are missing.

5 February 2005 02:09

Chapter 17: Header les

291

Incompatible denitions. The denitions are there, but they dont match your compiler. This is particularly often the case with C++ on systems that dont have a native C++ compiler. The gcc utility program protoize, which is run when installing gcc, is supposed to take care of these differences, and it may be of use even if you choose not to install gcc. Incorrect #ifdefs. For example, the le may dene certain functions only if _POSIX_SOURCE is dened, even though _POSIX_SOURCE is intended to restrict functionality, not to enable it. The System V.4.2 version math.h surrounds M_PI (the constant pi) with
#if (__STDC__ && !defined(_POSIX_SOURCE)) || defined(_XOPEN_SOURCE)

In other words, if you include math.h without dening __STDC__ (ANSI C) or _XOPEN_SOURCE (X Open compliant), M_PI will not be dened. The header les may contain syntax errors that the native compiler does not notice, but which cause other compilers to refuse them. For example, some versions of XENIX curses.h contain the lines:
#ifdef M_TERMCAP # include <tcap.h> /* Use: cc -DM_TERMCAP ... -lcurses -ltermlib */ #else # ifdef M_TERMINFO # include <tinfo.h> /* Use: cc -DM_TERMINFO ... -ltinfo [-lx] */ # else ERROR -- Either "M_TERMCAP" or "M_TERMINFO" must be #defined. # endif #endif

This does not cause problems for the XENIX C compiler, but gcc, for one, complains about the unterminated character constant starting with defined. The header les may be missing. In the course of time, header les have come and gone, and the denitions have been moved to other places. In particular, the denitions that used to be in strings.h have been moved to string.h (and changed somewhat on the way), and termio.h has become termios.h (see Chapter 15, Terminal drivers, page 241 for more details).

The solutions to these problems are many and varied. They usually leave you feeling dissatised: Fix the system header les. This sounds like heresy, but if you have established beyond any reasonable doubt that the header le is to blame, this is about all you can do, assuming you can convince your system administrator that it is necessary. If you do choose this way, be sure to consider whether xing the header le will break some other program that relies on the behaviour. In addition, you should report the bugs to your vendor and remember to re-apply the updates when you install a newer version of the operating system. Use the system header les, but add the missing denitions in local header les, or, worse, in the individual source les. This is a particularly obnoxious solution,

5 February 2005 02:09

292

especially when, as so often, the declarations are not dependent on a particular ifdef. In almost any system with reasonably complete header les there will be discrepancies between the declarations in the system header les and the declarations in the package. Even if they are only cosmetic, they will stop an ANSI compiler from compiling. For example, your system header les may declare getpid to return pid_t, but the package declares it to return int. About the only legitimate use of this style of xing is to declare functions that will really cause incorrect compilation if you dont declare them. Even then, declare them only inside an ifdef for a specic operating system. In the case of getpid, youre better off not declaring it: the compiler will assume the correct return values. Nevertheless, you will see this surprisingly often in packages that have already been ported to a number of operating systems, and its one of the most common causes of porting problems. Make your own copies of the header les and use them instead. This is the worst idea of all: if anything changes in your systems header les, you will never nd out about it. It also means you cant give your source tree to somebody else: in most systems, the header les are subject to copyright.

5 February 2005 02:09

Function libraries
In this chapter, well look at functions normally supplied in libraries with the system. As mentioned on page 151, if you have the sources, it is usually relatively trivial to port a single library function that doesnt require specic kernel functionality. There are hundreds of libraries, of course, and we can look only at those libraries that are available on a large number of systems. In the following sections, well look at: Functions that are found in the standard C library frequently enough that their absence can be a problem. Block memory functions, which modern compilers frequently treat as special cases. Regular expression librariesve of them, all incompatible with each other, starting on page 300. terminfo and termlib, starting on page 307. Well also briey touch on curses.

Standard library functionality

The content of the standard C library libc.a reminds you of the joke The nice thing about standards is that there are so many to choose from. Different systems have very different views on what should and should not be part of the standard library, and a number of systems dont supply the functions at all. In this section, well look at frequently used functions and what to do if your system doesnt have them.

alloca
alloca allocates memory in the stack, as opposed to malloc, which allocates memory in the heap:
void *alloca (size_t size);

This has a signicant speed advantage over malloc: malloc needs to search and update a free space list. alloca typically just needs to change the value of a register, and is thus very fast. It is often included with the package you are compiling, but you may need to set a ag or modify the Makele in order to include it. Versions for VAX, HP 300, i386 and Tahoe
293

5 February 2005 02:09

294

processors are located in the 4.4BSD Lite distribution as lib/libc/<machine>/gen/alloca.s. On the down side, it is a somewhat system-dependent function, and its possible that it might break after a new kernel release. You can almost always replace alloca with malloc, though there may be a performance impact.

bcopy
bcopy is a BSD function that substantially corresponds to memmove:
#include <string.h> void bcopy (const void *src, void *dst, size_t len);

Unlike memcpy, it is guaranteed to move correctly when the source and destination elds overlap. If your system has memmove, you can dene it as:
#define bcopy(s, d, l) memmove (d, s, l)

The operands have a different sequence from those of memmove.

bzero
bzero is a BSD function to clear an area of memory to 0. It is a subset of the standard function memset, and you can dene it as
#define bzero(d, l) memset (d, \0, l)

fnmatch
fnmatch is a routine that matches patterns according to the shell le name rules:
#include <fnmatch.h> int fnmatch (const char *pattern, const char *string, int flags);

fnmatch compares string against pattern. It returns 0 if string matches pattern and FNM_NOMATCH otherwise. The ags in Table 18-1 specify the kind of match to perform:
Table 181: fnmatch ags

Flag FNM_NOESCAPE FNM_PATHNAME FNM_PERIOD

Meaning Interpret the backslash character (\) literally. Slash characters in string must be explicitly matched by slashes in pattern. Leading periods in strings match periods in patterns. Not all versions of fnmatch implement this ag.

5 February 2005 02:09

Chapter 18: Function libraries

295

fnmatch is supplied with later BSD versions only. If you need it, it is in the 4.4BSD Lite distributio as lib/libc/gen/fnmatch.c.

getcwd and getwd

getcwd and getwd both return the name of the current working directory. This is the function behind the pwd command:
#include <stdio.h> char *getcwd (char *buf, size_t size); char *getwd (char *buf);

getwd has the great disadvantage that the function does not know the length of the pathname, and so it can write beyond the end of the buffer. As a result, it has been replaced by getcwd, which species a maximum length for the returned string. You can dene getwd as:
#define getwd(d) getcwd (d, MAXPATHLEN)

MAXPATHLEN is a kernel constant dening the maximum path name length. It is normally dened in /usr/include/sys/param.h.

gethostname and uname

There is no one function call which will return the name of the system on all UNIX platforms. On BSD systems, gethostname returns the current host name:
#include <unistd.h> int gethostname (char *name, int namelen);

gethostname returns a null-terminated string if the space dened by namelen allows. This function is supported in System V.4, but not in standard versions of System V.3 and XENIX. On System V systems, the system call uname returns a number of items of information about the system, including the name. It is also supported as a library function by most BSD systems.
#include <sys/utsname.h> sys/utsname.h defines struct utsname { char sysname [9]; char nodename [9]; char release [9]; char version [9]; char machine [9]; }; int uname (struct utsname *name);

/* /* /* /* /*

Internal system name */ External system name */ Operating system release */ Version of release */ Processor architecture */

The systems that do support uname apply a different meaning to the eld sysname. For example, consider the output of the following program, which was compiled and run on Interactive UNIX/386 System V.3 Version 2.2 and BSD/386 version 1.1, both running on an Intel

5 February 2005 02:09

296

486 platform:
#include <sys/utsname.h> main () { struct utsname myname; uname (&myname); printf ("sysname %s nodename %s release %s version %s machine %s\n", myname.sysname, myname.nodename, myname.release, myname.version, myname.machine); } $ uname On the System V.3 machine: sysname adagio nodename adagio release 3.2 version 2 machine i386 $ uname On the BSD/386 machine: sysname BSD/386 nodename allegro release 1.1 version 0 machine i386

System V puts the node name in sysname, whereas BSD uses it for the name of the operating system. This information is by no means complete: in particular, neither version tells you explicitly whether the system is running System V or BSD, and there is no indication of the vendor at all on the System V system.

index
index searches the string s forwards for the rst occurrence of the character c. If it nds one, it returns a pointer to the character. Otherwise it returns NULL. It is essentially the same as the ANSI function strchr, and you can dene it as:
#define index strchr

malloc
malloc has always been around, but the semantics have changed in the course of time. In the Seventh Edition and XENIX, a call to malloc with length 0 returned a valid pointer, whereas later versions return a NULL pointer, indicating an error. As a result, programs that ran on older versions might fail on more recent implementations.

memmove
memmove copies an area of memory:
#include <string.h> void *memmove (void *dst, const void *src, size_t len);

This is the same function as memcpy, except that memmove is guaranteed to move overlapping data correctly. Except for the parameter sequence, it is the same as bcopy (see page 294). If you dont have either function, you can nd bcopy in the 4.4BSD library source

5 February 2005 02:09

Chapter 18: Function libraries

297

(lib/libc/string/bcopy.c), as well as versions in assembler: lib/libc/vax/string/memmove.s lib/libc/hp300/string/bcopy.s lib/libc/tahoe/string/bcopy.s A generic version of memmove in C is in the GNU C library in sysdeps/generic/memmove.c. See Appendix E, Where to get sources to locate all these sources. Note also the comments about memory move functions on page 299.

remove
#include <stdio.h> int remove (const char *path);

On BSD systems, remove is a synonym for the system call unlink. This means that it makes sense to use it only for les. On System V.4 systems, it is slightly more complicated: if called for a le, it does the same thing as unlink, for directories it does the same thing as rmdir.

rindex
rindex (reverse index) searches the string s for the last occurrence of character c and returns a pointer to the character if it is found, and NULL otherwise. It is essentially the same function as strrchr, and you can dene it as:
#define rindex strrchr

snprintf and vsnprintf

snprintf and vsnprintf are versions of sprintf and vsprintf that limit the length of the output string:
int int int int sprintf (char *str, const char *format, ...); snprintf (char *str, size_t size, const char *format, ...); vsprintf (char *str, char *format, va_list ap); vsnprintf (char *str, size_t size, const char *format, va_list ap);

The argument size species the maximum length of the output string, including the trailing \0. These functions are supplied in 4.4BSD Lite as usr/src/lib/libc/stdio/snprintf.c and usr/src/lib/libc/stdio/vsnprintf.c. Alternatively, you can remove the second parameter and use sprintf or vsprintf instead.

strcasecmp and strncasecmp

strcasecmp and strncasecmp perform an unsigned comparison of two ASCII strings ignoring case (they consider a to be the same thing as A):

5 February 2005 02:09

298
#include <string.h> int strcasecmp (const char *s1, const char *s2); int strncasecmp (const char *s1, const char *s2, size_t len);

strncasecmp differs from strcasecmp by comparing at most len characters. Both functions stop comparing when a NUL character is detected in one of the strings. You can nd both functions in the 4.4BSD Lite distribution (lib/libc/string/strcasecmp.c).

strdup
strdup allocates memory with malloc and copies a string to it:
#include <string.h> char *strdup (const char *str);

It is included in the 4.4BSD Lite distribution (lib/libc/string/strdup.c).

strerror and sys_errlist

strerror returns an error message string for a specic error:
#include <string.h> extern char *sys_errlist []; extern int sys_nerr; char *strerror (int errnum);

errnum is the number of the error; strerror returns a pointer to a text for the error, or NULL if none is found. Most library implementations also dene sys_errlist, an array of description strings for errors, and sys_nerr, the total number of error messages, in other words, the number of messages in sys_errlist. If you dont nd this function anywhere in your man pages, dont give up: its frequently hidden in an unexpected library. For example, NonStop UX version B22 doesnt dene or document sys_errlist anywhere, but it is in libc.a all the same. The implementation of strerror is trivial:
char *strerror (int errnum) { if (errnum < sys_nerr) return sys_errlist [errnum]; else { static char bogus [80]; sprintf (bogus, "Unknown error: %d", errnum); return bogus; } }

Dont assume that your system doesnt have sys_errlist just because you cant nd a

5 February 2005 02:09

Chapter 18: Function libraries

299

denition in the header les. Many systems install it via the back door because packages such as X11 use them. The safest way to nd out is to search the system libraries. The shell script ndf, described in Chapter 21, Object les and friends, page 374, will do this for you.

stricmp and strnicmp

These are somewhat uncommon alternatives to strcasecmp and strncasecmp which are supplied on some systems (see page 297).

Block memory access functions

Many programs spend the bulk of their execution time moving areas of memory about or comparing them. The C language supplies a rich set of functions, such as memcpy, memmove, strcpy, strchr, strlen, and friends. Unfortunately, their performance frequently leaves something to be desired. Many C libraries still write the move in C. You can write memcpy as:
char *memcpy (char *d, char *s, int len) { char *dest = d; while (len--) *d++ = *s++; return dest; }

On an Intel 386 architecture, gcc compiles quite a tight little loop with only 7 instructions,* but it also requires another 15 instructions to set up the function environment and remove it again. In addition, the calling sequence memcpy (bar, foo, 10) might compile in 5 instructions. Many machines supply special instructions for block memory operations, but even those that dont can do it faster without a function call. The block memory functions are thus ideal candidates for inline functions that the compiler can optimize. Many compilers now do so, including gcc and the System V.4 CCS compilers. In this situation, the compiler can recognize that there are only a few bytes to be moved, and that they are word-aligned, so it can use native load and store operations. When you enable optimization, gcc can compiles the memcpy (bar, foo, 10) into only 6 simple instructions: the loop has disappeared completely, and we just have 3 load and 3 store instructions. This approach isnt appropriate for moves of 1000 bytes, of course. Here, the compiler uses 4 instructions to set up a block move instruction, for a total of 5 instructions. These examples are typical of what a smart compiler can do if it has inside information about what the function does. Normally this information is compiled into the compiler, and it doesnt need a header le to know about the function. This can have a number of consequences for you: The compiler knows the parameter types for the function. If you dene the function differently, you get a possibly confusing error message:

* See Chapter 21, Object les and friends, page 377 for a discussion of parameter passing.

5 February 2005 02:09

300
memcpy.c:3: warning: conflicting types for built-in function memcpy

If you get this message, you can either ignore it or, better, remove the denition. The compiler knows anyway. When debugging, you cant just put a breakpoint on memcpy. There is no such function, or if it has been included to satisfy references from modules compiled by other compilers, it may not be called when you expect it to. If you have a program written for a compiler that knows about the block memory functions, you may need to add denitions if your compiler doesnt support them.

Regular expression routines

Regular expressions are coded descriptions of text strings. They are used by editors and utilities such as grep for searching and matching actual text strings. Theres nothing very special about routines that process regular expressions, but there is no agreement about standards and there is no agreement about how to write a regular expression. The only regular thing about them is the regularity with which programs fail to link due to missing functions. There are at least ve different packages, and the names are similar enough to cause signicant confusion. In all cases, the intention of the routines is the same: A compilation function converts a string representation of a regular expression into an internal representation that is faster to interpret. A search function performs the search.

In addition, the Eighth Edition regex package has support for a replacement function based on the results of a previous search. Regular expression syntax also comes in two avours: The documentation of the older syntax usually states that it is the same syntax that ed uses. ed is an editor that is now almost completely obsolete,* so its good to know that the stream editor sed, which is still in current use, uses the same syntax. The newer syntax is the same that egrep uses. It is similar to that of ed, but includes a number of more advanced expressions.

If you get software that expects one package, but you have to substitute another, you should expect changes in behaviour. Regular expressions that worked with the old package may not work with the new one, or they may match differently. A particularly obvious example is the use of parentheses to group expressions. All forms of regular expressions perform this grouping, but the old syntax requires the parentheses to be escaped: \(expr\), whereas the new syntax does not: (expr).
* ed does have its uses, though. If you have serious system problems (like /usr crashed), its nice to have a copy of ed on the root le system. Its also useful when your only connection to the machine is via a slow modem line: over a 2400 bps line, redrawing a 24x80 screen with vi or emacs takes 8 seconds, and things are a lot faster with ed.

5 February 2005 02:09

Chapter 18: Function libraries

301

Apart from the intention of the functions, they also perform their task in very different ways. They might store compiled program in an area that the caller supplies, they might malloc it, or they might hide it where the user cant nd it. In some cases, the compiled expression is stored in a struct along with other information intended for use by the calling functions. In others this information is not returned at all, and in others again it is returned in global arrays, and in others it is returned via optional arguments. Translating from one avour to another takes a lot of time. Three packages are generally available: Henry Spencers Eighth Edition package, the 4.4BSD POSIX.2 version, and the GNU POSIX.2 version. See Appendix E, Where to get sources for sources of these packages. If you do have to port to a different regular expression package, choose a POSIX.2 implementation. Although it is by far the most complicated to understand, it is probably the only one that will be around in a few years. Regular expressions are used for two purposes: searching and replacing. When replacing one regular expression with another, its nice to be able to refer to parts of the expression. By convention, you dene these subexpressions with parentheses: the expression foo\(.*\)bar\(.*\)baz denes two such subexpressions. The regular expression will match all strings containing the texts foo, bar, and baz with anything in between them. The rst marked subexpression is the text between foo and bar, and the second one is the text between bar and baz.

regexpr
The regexpr routines have traditionally been supplied with System V and XENIX systems. They were originally part of the ed editor and thus implement the ed style of regular expressions. Despite the name, there is no function called regexpr. The routines themselves are normally in a library libgen.a. In addition, for some reason many versions of System V and XENIX include the complete source to the functions in the header le regexp.h, whereas the header le regexpr.h contains only the declarations. There are three routines:
#include <regexpr.h> extern char *loc1, *loc2, *locs; extern int nbra, regerrno, reglength; extern char *braslist [], *braelist []; char *compile (const char *instring, char *expbuf, char *endbuf); int step (const char *string, char *expbuf); int advance (const char *string, char *expbuf);

compile compiles the regular expression instring. The exact behaviour depends on the value of expbuf. If expbuf is NULL, compile mallocs space for the compiled version of the expression and returns a pointer to it. If expbuf is non-NULL, compile places the compiled form there if it ts between expbuf and endbuf, and it returns a pointer to the rst free byte. If the regular expression does not t in this space, compile aborts. If the compilation succeeds, the length of the compiled expression is stored in the

5 February 2005 02:09

302

global variable reglength. If the compilation fails, compile returns NULL and sets the variable regerrno to one of the values in Table 18-2:
Table 182: regcomp error codes

Error code 11 16 25 36 41 42 43 44 45 46 49 50

Meaning Range endpoint too large. Bad number. \digit out of range. Illegal or missing delimiter. No remembered search string. \( \) imbalance. Too many \(. More than 2 numbers given in \{ \}. } expected after \. First number exceeds second in \{ \}. [ ] imbalance. Regular expression overow.

step compares the string string with the compiled expression at expbuf. It returns non-zero if it nds a match, and 0 if it does not. If there is a match, the pointer loc1 is set to point to the rst matching character, and loc2 is set to point past the last character of the match. If the regular expression contains subexpressions, expressions bracketed by the character sequences \( and \), step stores the locations of the start and end of each matching string in the global arrays braslist (start) and braelist (end). It stores the total number of such subexpressions in nbra. advance has the same function as step, but it restricts its matching to the beginning of the string. In other words, a match always causes loc1 to point to the beginning of the string.

regcmp
regcmp is another regular expression processor used in System V and XENIX. Like regexpr, they implement the ed style of regular expressions with some extensions. They are also normally part of the library libgen.a.
#include <libgen.h> char *regcmp (const char *string1, /* char *string2 */ ... (char *) 0);

5 February 2005 02:09

Chapter 18: Function libraries

char *regex (const char *re, const char *subject, /* char *ret0, *ret1, ... *retn */ ... ); extern char *__loc1;

303

regcmp can take multiple input arguments, which it concatenates before compilation. This can be useful when the expression is supplied on a number of input lines, for example. It always mallocs space for its compiled expression, and returns a pointer to it. regex searches for the string subject in the compiled regular expression re. On success, it returns a pointer to the next unmatched character and sets the global pointer __loc1 to the address of the rst character of the match. Optionally, it returns up to ten strings at ret0 and the parameters that follow. You specify them with the $n regular expression element discussed below.

The regular expression syntax is slightly different from that of ed and sed: The character $ represents the end of the string, not the end of the line. Use \n to specify the end of the line. You can use the syntax [a-f] to represent [abcdef]. You can use the syntax x+ to represent one or more occurrences of x. You can use the syntax {m}, where m is an integer, to represent that the previous subexpression should be applied m times. You can use the syntax {m,}, where m is an integer, to represent that the previous subexpression should be applied at least m times. You can use the syntax {m,u}, where m and u are integers, to represent that the previous subexpression should be applied at least m and at most u times. The syntax (exp) groups the characters exp so that operators such as * and + work on the whole expression and not just the preceding character. For example, abcabcabcabc matches the regular expression (abc)+, and abcccccc matches abc+. The syntax (exp)$n, where n is an integer, matches the expression exp, and returns the address of the matched string to the call parameter retn of the call to regex. It will even try to return it if you didnt supply parameter retn, so its good practice to supply all the parameters unless you are in control of the regular expressions.

regex: re_comp and re_exec

regex is the somewhat confusing name given to the functions re_comp and re_exec, which were introduced in 4.0BSD. Note particularly that there is no function called regex, and that the name is spelt without a nal p. regex also implements ed-style regular expressions. There are two functions:

5 February 2005 02:09

304
char *re_comp (char *sp); int re_exec (char *p1);

re_comp compiles the regular expression sp and stores the compiled form internally. On successful compilation, it returns a NULL pointer, and on error a pointer to an error message. re_exec searches the string p1 against the internally stored regular expression. It returns 1 if the string p1 matches the last compiled regular expression, 0 if the string p1 fails to match the last compiled regular expression, and -1 if the compiled regular expression is invalid.

No public-domain versions of regex are available, but its relatively simple to dene them in terms of POSIX.2 regex.

Eighth edition regexp

The Eighth edition regexp package has gained wider popularity due to a widely available implementation by Henry Spencer of the University of Toronto. It consists of the functions regcomp, regexec, regsub and regerror:
#include <regexp.h> regexp * regcomp (const char *exp); int regexec (const regexp *prog, const char *string); void regsub (const regexp *prog, const char *source, char *dest); void regerror (const char *msg);

In contrast to earlier packages, Eighth edition regexp implements egrep-style regular expressions. Also in contrast to other packages, the compiled form of the regular expression includes two arrays which regexec sets for the convenience of the programmer: char *startp [] is an array of start addresses of up to nine subexpressions (expressions enclosed in parentheses), and char *endp [] is an array of the corresponding end addresses. The subexpressions are indexed 1 to 9; startp [0] refers to the complete expression. regcomp compiles the regular expression exp and stores the compiled version in an area that it mallocs. It returns a pointer to the compiled version on success or NULL on failure. regexec matches the string string against the compiled regular expression prog. It returns 1 if a match was found and 0 otherwise. In addition, it stores the start and end addresses of the rst ten parenthesized subexpressions in prog->startp and prog->endp. regsub performs a regular expression substitution, a function not offered by the other packages. You use it after regcomp nds a match and stores subexpression start and end information in startp and endp. It copies the input string source to dest, replacing expressions of the type &n, where n is a single digit, by the substring dened by startp [n] and endp [n]. regerror determines the action to be taken if an error is detected in regcomp, regexec or regsub. The default regerror prints the message and terminates the program. If you want, you can replace it with your own routine.

5 February 2005 02:09

Chapter 18: Function libraries

305

POSIX.2 regex
As if there werent enough regular expression libraries already, POSIX.2 also has a version of regex. It is intended to put an end to the myriad other avours of regular expressions, and thus supplies all the functionality of the other packages. Unfortunately, it re-uses the function names of Eighth Edition regexp. This is the only similarity: the POSIX.2 functions take completely different parameters. The header le of the 4.4BSD package starts with
#ifndef _REGEX_H_ #define _REGEX_H_ /* never again */

#ifdef _REGEXP_H_ BAD NEWS -- POSIX regex.h and V8 regexp.h are incompatible #endif

The Eighth Edition regexp.h contains similar code, so if you accidentally try to use both, you get an error message when you try to compile it. The POSIX.2 regex package offers a seemingly innite variety of ags and options. It consists of the functions regcomp, regexec, regerror and regfree. They match regular expressions according to the POSIX.2 regular expression format, either ed format (so-called basic regular expressions, the default) or egrep format (extended regular expressions). The 4.4BSD implementations refer to them as obsolete regular expressions and modern regular expressions respectively. You choose the kind of expression via ag bits passed to the compilation function regcomp. Alternatively, you can specify that the string is to be matched exactly no characters have any special signicance any more. Here are the functions:
#include <sys/types.h> #include <regex.h> int regcomp (regex_t *preg, const char *pattern, int cflags); int regexec (const regex_t *preg, const char *string, size_t nmatch, regmatch_t pmatch [], int eflags); size_t regerror (int errcode, const regex_t *preg, char *errbuf, size_t errbuf_size); void regfree (regex_t *preg);

regcomp compiles the regular expression pattern and stores the result at preg. It returns 0 on success and an error code on failure. cflags modies the way in which the

5 February 2005 02:09

306

compilation is performed. There are a number of ags, listed in Table 18-3:

Table 183: cflags bits for regcomp

Flag bit REG_BASIC REG_EXTENDED REG_NOSPEC

Function Compile basic ("obsolete") REs. This is the default. Compile extended ("modern") REs. Compile a literal expression (no special characters). This is not specied by POSIX.2. You may not combine REG_EXTENDED and REG_NOSPEC. Compile an expression that ignores case. Compile to report only whether the text matched, dont return subexpression information. This makes the search faster. Compile for newline-sensitive matching. By default, a newline character does not have any special meaning. With this ag, and $ match beginnings and ends of lines, and expressions bracketed with [] do not match new lines. Specify that the expression ends at re_endp, thus allowing NUL characters in expressions. This is not dened in POSIX.2

REG_ICASE REG_NOSUB REG_NEWLINE

REG_PEND

regexec matches the string string against the compiled regular expression preg. If nmatch is non-zero and pmatch is non-NULL, start and end information for up to nmatch subexpressions is returned to the array pmatch. regexec also supports a number of ags in eflags. They are described in Table 18-4:
Table 184: eflags bits for regexec

Flag bit REG_NOTBOL REG_NOTEOL REG_STARTEND

Function Do not match the beginning of the expression with . Do not match the end of the expression wtih $. Specify that the string starts at string + pmatch[0].rm_so and ends at string + pmatch[0].rm_eo. This can be used with cflags value REG_PEND to match expressions containing NUL characters.

regerror is analogous to the C library function perror: it converts the error code errcode for regular expression preg into a human-readable error message at errbuf, up to a maximum of errbuf_size bytes.

As in Eighth Edition regexp, regcomp returns additional information about the expression:

5 February 2005 02:09

Chapter 18: Function libraries

307

If you compile the expression with the REG_PEND bit set in cflags, you can set re_endp to point to the real end of the regular expression string supplied to regcomp, thus allowing NUL characters in the regular expression. After compilation, regcomp sets re_nsub to the number of subexpressions it found. For each subexpression, regexec can return start and end address information if you call it with appropriate parameters.

In addition, regexec stores information about matched subexpressions in a structure of type regmatch_t, unless you specify the ag REG_NOSUB. This contains at least the elds rm_so and rm_eo, which are offsets from the start of the string of the rst and last character of the match. They are of type regmatch_t. No less than three versions of POSIX.2 regex are generally available: Henry Spencers regex is included in the 4.4BSD distribution, and the GNU project has the older regex and newer rx. See Appendix E, Where to get sources.

termcap and terminfo

When full-screen editors started to appear in the late 70s, Bill Joy at UCB wrote ex, out of which grew vi. One of the problems he had was that just about every terminal has a different command set. Even commands as simple as positioning the cursor were different from one terminal to the next. In order to get the editor to work over a range of terminals, he devised termcap, the terminal capabilities database, which, with a few access routines, allowed a program to determine the correct control sequences for most common functions. A few years later, while writing the game rogue, Ken Arnold extended the routines and created a new package, curses, which offered higher-level functions. Finally, USG improved on curses and introduced terminfo, a new implementation of the same kind of functionality as termcap. It was designed to address some of the weaknesses of termcap, and as a result is not compatible with it. More programs use termcap than terminfo: terminfo is usually restricted to System V systems, and termcap has been around for longer, and is available on most systems. Some programs use both, but there arent very many of them. There are a number of gotchas waiting for you with termcap, termlib and curses: Termcap isnt perfect, to put it mildly: it relies on a marginally human-readable denition le format, which means that the access routines are slow. When AT&T incorporated termcap into System V, they addressed this problem and created terminfo, with a socalled compiled representation of the terminal data. Both termcap and terminfo are passive services: your program has to explicitly ask for them in order for them to work. Many programs dont use either, and many use only one or the other. Though the BSD curses package offered a bewildering number of functions, users asked for more. When AT&T incorporated curses into System V, they added signicant enhancements. Curses goes a level of complexity beyond termcap and terminfo, and supplies a huge number of functions to perform all kinds of functions on a terminal. Its

5 February 2005 02:09

308

like a can of worms: avoid opening it if you can, otherwise refer to UNIX Curses Explained by Berny Goodheart. If you have a BSD system and need System V curses, you can try the ncurses package (see Appendix E, Where to get sources). BSD versions of UNIX have still not incorporated terminfo or the additional curses routines, although they have been available for some time. In this section, well look at the differences in the implementations and what can be done about them. For more information, read Programming with curses, by John Strang, for a description of BSD curses, Termcap and Terminfo, by John Strang, Tim OReilly and Linda Mui for a description of termcap and terminfo, and UNIX Curses Explained, by Berny Goodheart, for a description of both versions of curses.

termcap
The heart of termcap is the terminal description. This may be specied with an environment variable, or it may be stored in a le containing denitions for a number of terminals. There is no complete agreement on the location of this le: If the termcap routines nd the environment variable TERMCAP, and it doesnt start with a slash (/), they try to interpret it as a termcap entry. If the termcap routines nd the environment variable TERMCAP, and it does start with a slash, they try to interpret it as the name of the termcap le. If TERMCAP isnt specied, the location depends on a constant which is compiled into the termcap library. Typical directories are /etc, /usr/lib, and /usr/share. Dont rely on nding only one of these les: its not uncommon to nd multiple copies on a machine, only one of which is of any use to you. If the system documentation forgets to tell you where the termcap le is located, you can usually nd out with the aid of strings and grep. For example, BSD/OS gives
$ strings libtermcap.a |grep /termcap .termcap /usr/share/misc/termcap

and SCO UNIX gives

$ strings libtermcap.a |grep /termcap /etc/termcap /etc/termcap

The le termcap contains entries for all terminals known to the system. On some systems it can be up to 200 kB in length. This may not seem very long, but every program that uses termcap must read it until it nds the terminal denition: if the terminal isnt dened, it reads the full length of the le. Heres a typical entry:
vs|xterm|vs100|xterm terminal emulator (X Window System):\ :AL=\E[%dL:DC=\E[%dP:DL=\E[%dM:DO=\E[%dB:IC=\E[%d@:UP=\E[%dA:\ :al=\E[L:am:\ :bs:cd=\E[J:ce=\E[K:cl=\E[H\E[2J:cm=\E[%i%d;%dH:co#80:\

5 February 2005 02:09

Chapter 18: Function libraries

:cs=\E[%i%d;%dr:ct=\E[3k:\ :dc=\E[P:dl=\E[M:\ :im=\E[4h:ei=\E[4l:mi:\ :ho=\E[H:\ :is=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;3;4;6l\E[4l:\ :rs=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;3;4;6l\E[4l\E<:\ :k1=\EOP:k2=\EOQ:k3=\EOR:k4=\EOS:kb=H:kd=\EOB:ke=\E[?1l\E>:\ :kl=\EOD:km:kn#4:kr=\EOC:ks=\E[?1h\E=:ku=\EOA:\ :li#65:md=\E[1m:me=\E[m:mr=\E[7m:ms:nd=\E[C:pt:\ :sc=\E7:rc=\E8:sf=\n:so=\E[7m:se=\E[m:sr=\EM:\ :te=\E[2J\E[?47l\E8:ti=\E7\E[?47h:\ :up=\E[A:us=\E[4m:ue=\E[m:xn: v2|xterms|vs100s|xterm terminal emulator, small window (X Window System):\ :co#80:li#24:tc=xterm: vb|xterm-bold|xterm with bold instead of underline:\ :us=\E[1m:tc=xterm: # # vi may work better with this termcap, because vi # doesnt use insert mode much vi|xterm-ic|xterm-vi|xterm with insert character instead of insert mode:\ :im=:ei=:mi@:ic=\E[@:tc=xterm:

309

The lines starting with the hash mark (#) are comments. The other lines are terminal capability denitions: each entry is logically on a single line, and the lines are continued with the standard convention of terminating them with a backslash (\). As in many other old UNIX les, elds are separated by colons (:). The rst eld of each description is the label, the name of the terminal to which the denition applies. The terminal may have multiple names, separated by vertical bars (|). In our example, the rst entry has the names vs, xterm and vs100. The last part of the name eld is a description of the kind of terminal. In the second entry, the names are vi, xterm-ic and xterm-vi. Both 4.4BSD termcap and System V terminfo recommend the following conventions for naming terminals: Start with the name of the physical hardware, for example, hp2621. Avoid hyphens in the name. Describe operational modes or conguration preferences for the terminal with an indicator, separated by a hyphen. Use the following sufxes where possible: Sufx
-w -am -nam -n -na -np -rv

Meaning Wide mode (more than 80 columns) With automatic margins (usually default) Without automatic margins Number of lines on screen No arrow keys (leave them in local) Number of pages of memory Reverse video

Example
vt100-w vt100-am vt100-nam aaa-60 concept100-na concept100-4p concept100-rv

5 February 2005 02:09

310

The following elds describe individual capabilities in the form capability=denition. The capabilities are abbreviated to two characters, and case is signicant. See Programming with curses, by John Strang, for a list of the currently dened abbreviations and their meaning. Depending on the capability, the denition may be a truth value (true or false), a number, or a string. For example, The rst entry for vs (AL=\E[%dL) states that the capability AL (insert n new blank lines) can be invoked with the string \E[%dL. \E represents an ESC character. The characters [ and L are used literally. The program uses sprintf to replace the %d with the number of lines to insert. The next entry, am, has no parameter. This is a boolean or truth value, in this case meaning that the terminal supports automatic margins. The presence of the capability means that it is true, the absence means that it is false. The entry co#80 species a numeric parameter and states that the capability co (number of columns) is 80.

There is almost nothing in the syntax of the denition le that requires that a particular capability have a particular type, or even which capabilities exist: this is simply a matter of agreement between the program and the capabilities database: if your program wants the capability co, and wants it to be numeric, it calls tgetnum. For example, the following code checks rst the information supplied by ioctl TIOCGWINSZ (see Chapter 15, Terminal drivers, page 259), then the termcap entry, and if both of them are not dened, it defaults to a congurable constant:
if (! (maxcols = winsize.ws_col) && (! (maxcols = tgetnum ("co"))) ) maxcols = MAXCOLS;

The only exception to this rule is the capability tc, which refers to another capability. In the example above, the entry for vi and friends consists of only 5 entries, but the last one is a tc entry that refers to the vs entry above. This lack of hard and fast rules means that termcap is extensible: if you have a terminal that can change the number of colours which can be displayed at one time, and for some reason you want to use this feature, you might dene a termcap variable XC specifying the string to output to the terminal to perform this function. The danger here is, of course, that somebody else might write a program that uses the variable XC for some other purpose.

termcap functions: termlib

Along with termcap come library functions, normally in a library called libtermcap.a, though a number of systems include them in libc.a. They help you query the database and perform user-visible functions such as writing to the terminal. There arent many functions, but they are nonetheless confusing enough:
char PC; char *BC; char *UP; /* padding character */ /* backspace string */ /* Cursor up string */

5 February 2005 02:09

Chapter 18: Function libraries

short ospeed; /* terminal output speed */

311

tgetent (char *bp, char *name); tgetnum (char *id); tgetflag (char *id); char *tgetstr (char *id, char **sbp); char *tgoto (char *cm, int destcol, int destline); tputs (register char *cp, int affcnt, int (*outc) ());

Before you start, you need two areas of memory: a 1 kB temporary buffer for the termcap entry, which we call buf, and a buffer for the string capabilities that you need, which we call sbuf. The initialization function tgetent lls sbuf with the termcap entry, and the function tgetstr transfers strings from buf to sbuf. After initialization is complete, buf can be deallocated. In addition, you need a char pointer sbp which must be initialized to point to sbuf. tgetstr uses it to note the next free location in sbuf. If you dont specify a specic termcap string as the value of the TERMCAP environment variable, you need to know the name of your terminal in order to access it in the termcap le. By convention, this is the value of the TERM environment variable, and you can retrieve it with the library function getenv. tgetent searches the termcap le for a denition for the terminal called name and places the entry into the buffer, which must be 1024 bytes long, at buf. All subsequent calls use this buffer. The confusing thing is that they do not explicitly reference the buffer: tgetent saves its address in a static pointer internal to the termcap routines. tgetent returns 1 on success, 0 if the terminal name was not found, or -1 if the termcap database could not be found. tgetnum looks for a numeric capability id and returns its value if found. If the value is not found, it returns -1. tgetflag looks for a boolean capability id and returns 1 if it is present and 0 otherwise. tgetstr looks for a string capability id. If it is found, it copies it into the buffer pointed to by the pointer at *sbp. It then updates sbp to point past the string. It returns the address of the string in sbuf if found, or NULL if not. This method ensures that the strings are null-terminated (which is not the case in buf), and that they are stored efciently (though a 1 kB buffer is no longer the overhead it used to be). tgoto generates a positioning string to move the cursor to column destcol and line destline using the cm (cursor position) capability. This capability contains format specications that tgoto replaces with the representation of the actual row and column numbers. It returns a pointer to the positioning string, which again is stored in a static buffer in the package. It also attempts to avoid the use of the characters \n, CTRL-D or CTRL-@. The resulting string may contain binary information that corresponds to tab characters. Depending on how it is set up, the terminal driver may replace these tab characters with blanks, which is obviously not a good idea. To ensure that this does not happen, turn off TAB3 on a termio or termios system (see Chapter 15, Terminal drivers, page 243) or reset XTABS in sgttyp.sg_flags with the old terminal driver (see page 240).

5 February 2005 02:09

312

This is all that tgoto does. It does not actually output anything to the terminal. tputs writes the string at cp to the screen. This seems unnecessary, since write and fwrite already do the same thing. The problem is that the output string cp may contain padding information for serial terminals, and only tputs interprets this information correctly. affcnt is the number of lines on the screen affected by this output, and outc is the address of a function that outputs the characters correctly. Often enough, this is something like putchar.

Problems with termcap

Sooner or later youll run into problems with termcap. Here are some of the favourite ones:

Missing description
It could still happen that there isnt a description for your terminal in the termcap data base. This isnt the problem it used to be, since most modern terminals come close to the ANSI standard. In case of doubt, try ansi or vt100. This is, of course, not a good substitute for complete documentation for your terminal.

Incomplete description
Its much more likely that you will nd a terminal description for your terminal, but its incomplete. This happens surprisingly often. For example, the xterm denition supplied in X11R5 has 56 capabilities, and the denition supplied with X11R6 has 85. xterm hasnt changed signicantly between X11R5 and X11R6: the capabilities were just missing from the entry in X11R5. Frequently youll nd that a feature youre looking for, in particular the code generated by a particular function key, is missing from your termcap entry. If nothing else helps, you can nd out what codes the key generates with od:
$ od -c display stdin in character format [[11[[12[[13[[14 RETURN 0000000 033 [ 1 1 033 [ 1 2 033 [ 1 3 0000020 033 [ 1 4 \n 0000025

In this example, I pressed the keys F1, F2, F3 and F4 on an xterm: this is what echos on the rst line. od doesnt display anything until its read completes, so I pressed RETURN to show the text. It shows that the sequences generated are: 033 (ESC, which is represented as \E in termcap entries). [1 and the number of the function key and a tilde ().

These sequences can then be translated into the termcap capabilities:

k1=\E[11:k2=\E[12:k3=\E[13:k4=\E[14:

5 February 2005 02:09

Chapter 18: Function libraries

313

Incorrect description
If we look at the previous example more carefully, well notice something strange: these capabilities arent the same as the ones in the example for xterm on page 308. Whats wrong with this picture? A good question. Both under X11R5 and X11R6, xterm on an Intel architecture gives you the codes shown above. The codes for F5 to F10 are as shown in the termcap entry, but the entries for F1 to F4 are just plain wrong. I dont know of any way to generate them with xterm. This is typical of termcap entries: if you run into trouble, rst make sure that your descriptions are correct.

Obsolete information
Another interesting thing about the xterm example is that it tells you the size of the terminal: co#80 says that this terminal has 80 columns, and li#65 says that it has 65 lines. This information can be an approximation at best, since X11 allows you to resize the window. Most modern systems supply the SIGWINCH signal, which is delivered to the controlling process when a window is resized (see Chapter 15, Terminal drivers, page 259). This information is just plain misleading, but theres a lot of it in just about any termcap le. The 4.4BSD man page ags a number of capabilities that are considered obsolete, including things as the character used to backspace or the number of function keys.

terminfo
terminfo is the System V replacement for termcap. At rst sight it looks very similar, but there are signicant differences: Instead of reading the termcap le, the terminfo routines require a compiled version. Instead of storing all entries in a single le, terminfo stores each entry in a different le, and puts them in a directory whose name is the initial letter of the terminal name. For example, the terminfo description for xterm might be stored in /usr/lib/terminfo/x/xterm. The substitution syntax has been signicantly expanded: in termcap, only tgoto could handle parameter substitution (see page 311); in terminfo, the substitution syntax is more general and offers many more features. The program tic (terminfo compiler) compiles terminfo descriptions. The programs infocmp, which is supplied with System V, and untic, which is part of ncurses, dump compiled terminfo entries in source form.

As an example of a terminfo denition, lets look at the denition for an xterm. This should contain the same information as the termcap entry on page 308:
xterm|xterm-24|xterms|vs100|xterm terminal emulator (X Window System), is2=\E7\E[r\E[m\E[?7h\E[?1;3;4;6l\E[4l\E8\E>, rs2=\E7\E[r\E[m\E[?7h\E[?1;3;4;6l\E[4l\E8\E>, am, bel=G, cols#80, lines#24, clear=\E[H\E[2J, cup=\E[%i%p1%d;%p2%dH,

5 February 2005 02:09

314
csr=\E[%i%p1%d;%p2%dr, cud=\E[%p1%dB, cud1=\n, cuu=\E[%p1%dA, cuu1=\E[A, cub=\E[%p1%dD, cub1=\b, cuf=\E[%p1%dC, cuf1=\E[C, el=\E[K, ed=\E[J, home=\E[H, ht=I, ind=J, cr=M, km, smir=\E[4h, rmir=\E[4l, mir, smso=\E[7m, rmso=\E[m, smul=\E[4m, rmul=\E[m, bold=\E[1m, rev=\E[7m, blink@, sgr0=\E[m, msgr, enacs=\E)0, smacs=N, rmacs=O, smkx=\E[?1h\E=, rmkx=\E[?1l\E>, kf1=\EOP, kf2=\EOQ, kf3=\EOR, kf4=\EOS, kf5=\E[15, kf6=\E[17, kf7=\E[18, kf8=\E[19, kf9=\E[20, kf10=\E[21, kf11=\E[23, kf12=\E[24, kf13=\E[25, kf14=\E[26, kf15=\E[28, kf16=\E[29, kf17=\E[31, kf18=\E[32, kf19=\E[33, kf20=\E[34, kfnd=\E[1, kich1=\E[2, kdch1=\E[3, kslt=\E[4, kpp=\E[5, knp=\E[6, kbs=\b, kcuu1=\EOA, kcud1=\EOB, kcuf1=\EOC, kcub1=\EOD, meml=\El, memu=\Em, smcup=\E7\E[?47h, rmcup=\E[2J\E[?47l\E8, sc=\E7, rc=\E8, il=\E[%p1%dL, dl=\E[%p1%dM, il1=\E[L, dl1=\E[M, ri=\EM, dch=\E[%p1%dP, dch1=\E[P, tbc=\E[3g, xenl, xterm-65|xterm with tall window 65x80 (X Window System), lines#65, use=xterm, xterm-bold|xterm with bold instead of underline (X Window System), smul=\E[1m, use=xterm, # # vi may work better with this entry, because vi # doesnt use insert mode much xterm-ic|xterm-vi|xterm with insert character instead of insert mode, smir@, rmir@, mir@, ich1=\E[@, ich=\E[%p1%d@, use=xterm,

The entries look very similar, but there are a few minor differences: The names for the capabilities may be up to 5 characters long. As a result, the names are different from the termcap names. Capabilities are separated by commas instead of colons. Denitions may be spread over several lines: there is no need to terminate each line of a denition with a \. The last character in each entry must be a comma (,). If you remove it, you will thoroughly confuse tic.

5 February 2005 02:09

Chapter 18: Function libraries

315

terminfo functions
terminfo has a number of functions that correspond closely to termlib. They are also the lowlevel routines for curses:
#include <curses.h> #include <term.h> TERMINAL *cur_term; int setupterm (char *term, int fd, int *error); int setterm (char *term); int set_curterm (TERMINAL *nterm); int del_curterm (TERMINAL *oterm); int restartterm (char *term, int fildes, int *errret); char *tparm (char *str, long int p1 ... long int p9); int tputs (char *str, int affcnt, int (*putc) (char)); int putp (char *str); int vidputs (chtype attrs, int (*putc) (char)); int vidattr (chtype attrs); int mvcur (int oldrow, int oldcol, int newrow, int newcol); int tigetflag (char *capname); int tigetnum (char *capname); int tigetstr (char *capname);

Terminfo can use an environment variable TERMINFO, which has a similar function to TERMCAP: it contains the name of a directory to search for terminfo entries. Since terminfo is compiled, there is no provision for stating the capabilities directly in TERMINFO. setupterm corresponds to the termcap function tgetent: it reads in the terminfo data and sets up all necessary capabilities. The parameter term is the name of the terminal, but may be NULL, in which case setupterm calls getenv to get the name of the terminal from the environment variable TERM. fd is the le descriptor of the output terminal (normally 1 for stdout), and error is an error status return address. setterm is a simplied version of setupterm: it is equivalent to setupterm (term, 1, NULL). setupterm allocates space for the terminal information and stores the address in the global pointer cur_term. You can use set_curterm to set it to point to a different terminal entry in order to handle multiple terminals. del_curterm deallocates the space allocated by setupterm for the terminal oterm. restartterm performs a subset of setupterm: it assumes that the cur_term is valid, but that terminal type and transmission speed may change. tparm substitutes the real values of up to 9 parameters (p1 to p9) into the string str. This can be used, for example, to create a cursor positioning string like tgoto, but it is much more exible. tputs is effectively the same function as termcap puts described on page 312.

5 February 2005 02:09

316

putp is effectively tputs (str, stdout, putchar). vidputs sets the terminal attributes for a video terminal. vidattr is effectively vidputs (attr, putchar). mvcur provides optimized cursor motion depending on the terminal capabilities and the relationship between the current cursor position (oldrow, oldcol) and the new position (newrow, newcol). tigetflag, tigetnum and tigetstr correspond to the termcap functions tgetnum, tgetflag and tgetstr described on page 311.

printcap
Termcap was developed in the days where terminal did not always mean a display terminal, and the capabilities include a number of functions relating to hardcopy terminals. Nevertheless, they are not sufcient to describe the properties of modern printers, and BSD systems have developed a parallel system called printcap, which is a kind of termcap for printers. It is used primarily by the spooling system. Printcap differs from termcap in a number of ways: The printcap capabilities are stored in a le called /etc/printcap, even in systems where termcap is stored in some other directory. The syntax of the printcap le is identical to the syntax of termcap, but the capabilities differ in name and meaning from termcap capabilities. There is no environment variable corresponding to TERMCAP. There are no user-accessible routines. The functions used by the spool system actually have the same names as the corresponding termcap functions, but they are private to the spooling system. In a BSD source tree, they are located in the le usr.sbin/lpr/common_source/printcap.c.

A better way to look at printcap is to consider it part of the spooling system, but occasionally youll need to modify /etc/printcap for a new printer.

5 February 2005 02:09

Make
Nowadays, only the most trivial UNIX package comes without a Makele, and you can assume that the central part of building just about any package is:
$ make

We wont go into the details of how make works hereyou can nd this information in Managing projects with make, by Andrew Oram and Steve Talbott. In this chapter, well look at the aspects of make that differ between implementations. Well also take a deeper look at BSD make, because it is signicantly different from other avours, and because there is very little documentation available for it.

Terminology
In the course of evolution of make, a change in terminology has taken place. Both the old and the new terminology are in current use, which can be confusing at times. In the following list, well look at the terms we use in this book, and then relate them to others which you might encounter: A rule looks like:
target: dependencies command command

A target is the name by which you invoke a rule. make implicitly assumes that you want to create a le of this name. The dependencies are the les or targets upon which the target depends: if any of the dependencies do not exist, or they are newer than the current target le, or the corresponding target needs to be rebuild, then the target will be rebuilt (in other words, its commands will be executed). Some versions of make use the terms prerequisite or source to represent what we call dependencies. The commands are single-line shell scripts that get executed in sequence if the target needs to be rebuilt.

317

5 February 2005 02:09

318

variables are environment variables that make imports, explicitly named variables, or implicit variables such as $@ and $<. Variables used to be called macros. They arent really macros, since they dont take parameters, so the term variable is preferable. BSD make uses the term local variable for implicit variables. As we will see, they dont correspond exactly. SunOS uses the term dynamic macros for implicit variables.

Additional make features

A number of versions of make offer additional features beyond those of the version of make described in Managing projects with make. In the following sections, well look at: Internal variables Variables with special meanings Targets with special meanings Including other source les from the Makele Conditional execution Variations on assignments to variables Functions Multiple targets

Internal variables
All versions of make supply internal variables, but the list differs between individual implementations. Well defer their discussion until we discuss BSD make, on page 324.

Variables with special meanings

A number of normal variables have taken on special meanings in some versions of make. Heres an overview: VPATH is a list of directory names to search for les named in dependencies. It is explicitly supported in GNU make, where it applies to all le searches, and is also supported, but not documented, in some versions of System V.4. GNU make also supports a directive vpath. MAKE is the name with which make was invoked. It can be used to invoke subordinate makes, and has the special property that it will be invoked even if you have specied the -n ag to make, indicating that you just want to see the commands that would be executed, and you dont want to execute them. In all modern versions of make, MAKEFLAGS is a list of the ags passed to make. make takes the value of the environment variable MAKEFLAGS, if it exists, and adds the command line arguments to it. It is automatically passed to subordinate makes.

5 February 2005 02:09

Chapter 19: Make

319

SHELL is the name of a shell to be used to execute commands. Note that many versions of make execute simple commands directly, so you may nd that this doesnt have any effect unless you include a shell metacharacter like ;.

The exact semantics of these variables frequently varies from one platform to anotherin case of doubt, read your system documentation.

Special targets
All versions of make dene a number of targets that have special meanings. Some versions dene additional targets: .BEGIN is a target to be executed before any other target. It is supported by BSD make. .INIT is a target to be executed before any other target. It is supported by SunOS and Solaris make. .END is a target to be executed after all other targets have been executed. It is supported by BSD make. .DONE is a target to be executed after all other targets have been executed. It is supported by SunOS and Solaris make. .FAILED is a target to be executed after all other targets have been executed. It is supported by SunOS and Solaris make. .INTERRUPT is a target to be executed if make is interrupted. It is supported by BSD make. .MAIN is the default target to be executed if no target was specied on the command line. If this target is missing, make will execute the rst target in the Makele. It is supported by BSD make. .MAKEFLAGS is an alternate method to supply ags to subordinate makes. It is supported by BSD make. .PATH is an alternate method to specify a search path for les not found in the current directory. It is supported by BSD make. .MUTEX is used in System V.4 to synchronize parallel makes. GNU make uses the target .PHONY to indicate targets that do not create les, such as clean and install. If by chance you have a le install in your directory, make will determine that make install does not need to be executed, since install is up to date. If you use GNU make, you can avoid this problem with:
.PHONY: all install clean

If you dont have GNU make, you can usually solve the problem with

5 February 2005 02:09

320
all install clean: .FORCE install commands .FORCE:

In this example, .FORCE looks like a special target, as it is meant to. In fact, the name is not important: you just need a name that doesnt correspond to a real le. In addition to special targets, BSD make also has special sources (in other words, special dependencies). Well look at them on page 327.

include directive
Many modern makes allow you to include other les when processing the Makele. Unfortunately, the syntax is very variable: In GNU make, the syntax is simply include filename. In BSD make, the syntax is .include <filename> or .include filename". The syntax resembles that of the C preprocessor: the rst form searches only the system directories, the second form searches the current directory before searching the system directories. In SunOS, Solaris and System V.4 make, the syntax is include filename, but the text include must be at the beginning of the line. SunOS and Solaris make automatically include a le make.rules in the current directory if it exists. Otherwise they include the le /usr/share/lib/make/make.rules.

Conditional execution
A number of versions of make support conditional execution of commands. GNU make has commands reminiscent of the C preprocessor:
ifeq (${CC},gcc} ${CC} -traditional -O3 -g $*.c -c -o $< else ${CC} -O $*.c -c -o $< endif

BSD make has a different syntax, which also vaguely resembles the C preprocessor. Apart from standard .if, .else and .endif, BSD make also provides an .ifdef directive and additional operators analogous to #if defined: .if make (variable) checks whether variable is a main target of make (in other words, if it was mentioned on the command line that invoked make). .if empty (variable) tests whether variable represents the empty string. .if exists (variable) tests whether the le variable exists.

5 February 2005 02:09

Chapter 19: Make

321

.if target (variable) tests whether variable represents a dened target.

SunOS and Solaris have so-called conditional macros:

foo bar baz:= CC = mycc

This tells make that the variable (macro) CC should be set to mycc only when executing the targets foo, bar, and baz.

Other forms of variable assignment

Simply expanded variables
make normally expands variables until no variable references remain in the result. Take the following Makele, for example:
CFLAGS = $(INCLUDE) $(OPT) OPT = -g -O3 INCLUDE= -I/usr/monkey -I/usr/dbmalloc all: @echo CFLAGS: ${CFLAGS}

If you run make, you will get:

$ make CFLAGS: -I/usr/monkey -I/usr/dbmalloc -g -O3

On the other hand, you cant change the denition to:

CFLAGS = $(CFLAGS) -I/usr/monkey

If you do this, you will get:

$ make makefile:7: *** Recursive variable CFLAGS references itself (eventually). Stop.

make would loop trying to expand $(CFLAGS). GNU make solves this with simply expanded variables, which go through one round of expansion only. You specify them with the assignment operator := instead of the usual =. For example:
CFLAGS = -g -O3 CFLAGS := $(CFLAGS) -I/usr/monkey

In this case, CFLAGS expands to -g -O3 -I/usr/monkey.

define directive
You frequently see multi-line shell script fragments in make rules. Theyre ugly and errorprone, because in conventional make, you need to put this command sequence on a single line with lots of backslashes and semicolons. GNU make offers an alternative with the dene directive. For example, to check for the existence of a directory and create it if it doesnt exist, you might normally write

5 February 2005 02:09

322
${INSTDIR}: if [ ! -d $@ ]; then \ mkdir -p $@; \ fi

With GNU make, you can dene this as a command:

define makedir if [ ! -d $@ ]; then mkdir -p $@ fi endef ${INSTDIR}: ${makedir}

Override variable definitions

Conventional versions of make have three ways to dene a make variable. In order of precedence, they are: 1. Dene it on the command line used to invoke make:
$ make CFLAGS="-g -O3"

2. 3.

Dene it in the Makele. Dene it in an environment variable. This is all the more confusing because most shells allow you to write the environment variable on the same line as the invokation of make:
$ CFLAGS="-g -O3" make

This looks almost identical to the rst form, but the precedence is lower. The command line option has the highest priority. This is usually a good idea, but there are times when you want the declaration in the Makele to take precedence: you want to override the denition on the command line. GNU make allows you to specify it with the override directive. For example, you might want to insist that the optimization level be limited to -O2 if youre generating debugging symbols. In GNU make, you can write:
override CFLAGS=-O2

Functions
As well as variables, GNU make supplies builtin functions. You call them with the syntax ${function arg,arg,arg}. These functions are intended for text manipulation and have names like subst, findstring, sort, and such. Unfortunately there is no provision for dening your own functions.

5 February 2005 02:09

Chapter 19: Make

323

Multiple targets
All forms of make support the concept of multiple targets. They come in two avours: Single-colon targets, where the target name is followed by a single colon. Each target of the same name may specify dependenciesthis is how Makele dependencies are specied but only one rule may have commands. If any of the dependencies require the target to be rebuilt, then these commands will be executed. If you supply commands to more than one rule, the behaviour varies: some versions of make will print a warning or an error message, and generally they execute only the last rule with commands. Under these circumstances, however, BSD make executes the rst rule with commands. Double-colon targets have two colons after the target name. Each of these is independent of the others: each may contain commands, and each gets executed only if the dependencies for that rule require it. Unfortunately, if multiple rules need to be executed, the sequence of execution of the rules is not dened. Most versions of make execute them in the sequence in which they appear in the Makele, but it has been reported that some versions of BSD make execute in reverse order, which breaks some Imakeles.

BSD make
With the Net/2 release, the Computer Sciences Research Group in Berkeley released a completely new make with many novel features. Most BSD avoured software that has come out in the last few years uses it. Unfortunately, it contains a number of incompatibilities with other makes. It is part of the 4.4BSD Lite distribution see Appendix E, Where to get sources for further detailsand includes hardcopy documentation, which refers to it as PMake. This name does not occur anywhere else, though you may see the name bsdmake. Weve already seen some of the smaller differences between BSD make and other avours. In the following sections well look at some more signicant differences. On page 327 well investigate the features of BSD make designed to make conguration easier.

Additional rule delimiter

There is a third delimiter between target and dependency in rules. Apart from the single and double colon, which have the same meaning as they do with other makes, there is a ! delimiter. This is the same as the single colon delimiter in that the dependencies are the sum of all dependencies for the target, and that only the rst rule set gets executed. However, the commands are always executed, even if all the dependencies are older than the target.

Assignment operators
BSD make supplies ve different types of variable assignment: = functions as in other versions of make: the assignment CFLAGS = -g unconditionally sets CFLAGS to -g.

5 February 2005 02:09

324

+= adds to a denition. If CFLAGS was set as in the previous example, writing CFLAGS += -O3 results in a new value -g -O3. ?= assigns a value only if the variable is currently undened. This can be used to set default values. := assigns and expands immediately. This is the same as the GNU make := assignment. != expands the value and passes it to a shell for execution. The result from the shell is assigned to the variable after changing newline characters to spaces.

Variables
BSD make has claried the denitions of variables somewhat. Although there is nothing really new in this area, the terminology is arranged in a more understandable manner. It denes four different kinds of variables, the rst three of which correspond to the kinds of variable assignment in other makes. In order of priority, they are: Environment variables global variables (just called variables in other avours of make) command line variables local variables, which correspond roughly to implicit variables in other makes.

BSD make allows the use of the implicit variable symbols ($@ and friends), but doesnt recommend it. They dont match very well, anyway, so it makes sense not to use them. Local variables are really variables that make predenes. Table 19-1 compares them to traditional make variables:
Table 191: make local variables Traditional

BSD .ALLSRC, $>

Meaning The list of all dependencies ("sources") for this target. (GNU make) The list of all dependencies of the current target. Only the member name is returned for dependencies that represent an archive member. Otherwise this is the same as BSD .ALLSRC.

$@ $$@

.ARCHIVE .TARGET, $@

The name of the current target. If the target is an archive le member, the name of the archive le. The complete name of the current target, even if it represents an archive le.1

5 February 2005 02:09

Chapter 19: Make Table 191: make local variables (continued) Traditional BSD Meaning .IMPSRC, $< $<

325

The implied source, in other words the name of the source le (dependency) implied in an implicit rule. The name of the current dependency that has been modied more recently than the target. Traditionally, it can only be used in sufx rules and in the .DEFAULT entry, but most modern versions of make (except BSD make) allow it to be used in normal rules as well.

$%

.MEMBER

The name of an archive member. For example, if the target name is libfoo.a(bar.o), $@ evaluates to libfoo.a and $% evaluates to bar.o. Supported by GNU, SunOS and System V.4 make. The dependencies for this target that were newer than the target.2 The raw name of the current dependency, without sufx, but possibly including directory components. Can only be used in sufx rules.

$? $*

.OODATE, $?

${*F} ${*D}

.PREFIX, $*

The raw le name of the current dependency. It does not contain any directory component. The directory name of the current dependency. For example, if $@ evaluates to foo/bar.o, ${@D} will evaluate to foo. Supported by GNU, SunOS and System V.4 make.

.CURDIR

The name of the directory in which the top-level make was started.

$$@ can only be used to the right of the colon in a dependency line. Supported by SunOS and System V.4 make. 2 Confusingly, BSD make refers to these dependencies as out of date, thus the name of the variable.

Variable substitution
In BSD make, variable substitution has reached a new level of complexity. All versions of make support the syntax ${SRC:.c=.o}, which replaces a list of names of the form foo.c bar.c baz.c with foo.o bar.o baz.o.. BSD make generalizes this syntax is into ${variable[:modifier[: . . . ]]}. In the following discussion, BSD make uses the term word where we would normally use the term parameter. In particular, a le name is a word. modier is an upper case letter:

5 February 2005 02:09

326

E replaces each word in the variable with its sufx. According to the documentation, H strips the last component from each word in the variable. A better denition is: it returns the directory name of each le name. If the original le name didnt have a directory name, the result is set to . (current directory). Mpattern selects those words from the variable that match pattern. pattern is a globbing pattern such as is used by shells to specify wild-card le names. Npattern selects those words from the variable that dont match pattern. R replaces each word in the variable with everything but its sufx. S/old/new/ replaces the rst occurrence of the text old with new. The form S/old/new/g replaces all occurrences. T replaces each word in the variable with its last component, in other words with the le name part.

This is heavy going, and its already more than the documentation tells you. The following example shows a number of the features:
SRCS = foo.c bar.c baz.cc zot.pas glarp.f src/mumble.c util/grunt.f LANGS = ${SRCS:E} DIRS = ${SRCS:H} OBJS = ${SRCS:T} CSRCS = ${SRCS:M*.c} PASSRCS = ${SRCS:M*.pas} FSRCS = ${SRCS:M*.f} PROGS = ${SRCS:R} PROFS = ${CSRCS:S/./_p./g:.c=.o} all: @echo @echo @echo @echo @echo @echo @echo @echo Languages: ${LANGS} Objects: ${OBJS} Directories: ${DIRS} C sources: ${CSRCS} Pascal sources: ${PASSRCS} Fortran sources: ${FSRCS} Programs: ${PROGS} Profiled objects: ${PROFS}

If you run it, you get:

$ make Languages: c c cc pas f c f Objects: foo.c bar.c baz.cc zot.pas glarp.f mumble.c grunt.f Directories: . . . . . src util C sources: foo.c bar.c src/mumble.c Pascal sources: zot.pas Fortran sources: glarp.f util/grunt.f Programs: foo bar baz zot glarp src/mumble util/grunt Profiled objects: foo_p.o bar_p.o src/mumble_p.o

5 February 2005 02:09

Chapter 19: Make

327

Special sources
In addition to special targets, BSD make includes special sources (recall that source is the word that it uses for dependencies). Here are the more important special sources: .IGNORE, .SILENT and .PRECIOUS have the same meaning as the corresponding special targets in other versions of make. .MAKE causes the associated dependencies to be executed even if the ags -n (just list commands, dont perform them) or -t (just update timestamps, dont perform make) are specied. This enables make to perform subsidiary makes even if these ags are specied. If this seems a strange thing to want to do, consider that the result of the main make could depend on subsidiary makes to such an extent that it would not even make sense to run make -n if the subsidiary makes did not run correctlyfor example, if the subsidiary make were a make depend. .OPTIONAL tells make that the specied dependencies are not crucial to the success of the build, and that make should assume success if it cant gure out how to build the target.

Specifying dependencies
We have seen that the bulk of a well-written Makele can consist of dependencies. BSD make offers the alternative of storing these les in a separate le called .depend. This avoids the problem of different avours of makedepend missing the start of the dependencies and adding them again.

BSD Makefile configuration system

One of the intentions of BSD make is to make conguration easier. A good example of how much difference it makes is in the Makeles for gcc. In its entirety, the top-level Makele is:
SUBDIR= cc cpp lib cc1 libgcc cc1plus cc1obj #libobjc .include <bsd.subdir.mk>

The complete Makele in the subdirectory cc1 (the main pass of the compiler) reads
# @(#)Makefile 6.2 (Berkeley) 2/2/91

PROG= gcc1 BINDIR= /usr/libexec SRCS= c-parse.c c-lang.c c-lex.c c-pragma.c \ c-decl.c c-typeck.c c-convert.c c-aux-info.c \ c-iterate.c CFLAGS+= -I. -I$(.CURDIR) -I$(.CURDIR)/../lib YFLAGS= NOMAN= noman .if exists(${.CURDIR}/../lib/obj)

5 February 2005 02:09

328
LDADD= DPADD= .else LDADD= DPADD= .endif -L${.CURDIR}/../lib/obj -lgcc2 ${.CURDIR}/../lib/obj/libgcc2.a -L${.CURDIR}/../lib/ -lgcc2 ${.CURDIR}/../lib/libgcc2.a

LDADD+= -lgnumalloc DPADD+= ${LIBGNUMALLOC} .include <bsd.prog.mk>

The standard release Makele for gcc is about 2500 lines long. Clearly a lot of work has gone into getting the BSD Makeles so small. The clue is the last line of each Makele:
.include <bsd.subdir.mk>

or
.include <bsd.prog.mk>

These les are supplied with the system and dene the hardware and software used on the system. They are normally located in /usr/share/mk, and you can modify them to suit your local preferences. This conguration mechanism has little connection with the new BSD make. It could equally well have been done, for example, with GNU make or System V make. Unfortunately, the signicant incompatibilities between BSD make and the others mean that you cant just take the conguration les and use them with other avours of make. The BSD system places some constraints on the Makele structure. To get the best out of it, you may need to completely restructure your source tree. To quote bsd.README:
Its fairly difcult to make the BSD .mk les work when youre building multiple programs in a single directory. Its a lot easier [to] split up the programs than to deal with the problem. Most of the agony comes from making the obj directory stuff work right, not because we switch to a new version of make. So, dont get mad at us, gure out a better way to handle multiple architectures so we can quit using the symbolic link stuff.

On the other hand, its remarkably easy to use BSD make conguration once you get used to it. Its a pity that the make itself is so incompatible with other makes: although the system is good and works well, its usually not worth restructuring your trees and rewriting your Makeles to take advantage of it. There are a couple of other points to note about the conguration method: make depend is supported via an auxiliary le .depend, which make reads after reading the Makele. The conguration les are included at the end of the Makele. This is due to the way that BSD make works: unlike other makes, if multiple targets with a single colon exist, only the rst will be executed, but if multiple declarations of the same variable exist, only the last one will take effect.

The conguration les consist of one le, sys.mk, which make automatically reads before

5 February 2005 02:09

Chapter 19: Make

329

doing anything else, and a number of others, one of which is usually included as the last line in a Makele. These are usually: bsd.prog.mk for a Makele to make an executable binary. bsd.lib.mk for a Makele to make a library. bsd.subdir.mk to make binaries or libraries in subdirectories of the current directory. In addition, another le bsd.doc.mk is supplied to make hardcopy documentation. In keeping with the Cinderella nature of such parts of a package, no other le refers to it. If you want to use it, you include it in addition to one of the other three. This is required only for hardcopy documentation, not for man pages, which are installed by the other targets.

sys.mk
sys.mk contains global denitions for all makes. make reads it in before looking for any Makeles. The documentation states that it is not intended to be modied, but since it contains default names for all tools, as well as default rules for makes, there is every reason to believe that you will want to change this le: theres no provision to override these denitions anywhere else. How you handle this dilemma is your choice. I personally prefer to change sys.mk (and put up with having to update it when a new release comes), but you could create another le bsd.own.mk, like FreeBSD does, and put your personal choices in there. The last line of the FreeBSD sys.mk is
.include <bsd.own.mk>

With this method you can override the denitions in sys.mk with the denitions in bsd.own.mk. Its up to you to decide whether this is a better solution.

bsd.prog.mk
bsd.prog.mk contains denitions for building programs. Table 19-2 lists the targets that it denes:
Table 192: bsd.prog.mk targets Target all clean cleandir depend

Purpose Build the single program ${PROG}, which is dened in the Makele. remove ${PROG}, any object les and the les a.out, Errs, errs, mklog, and core. remove all of the les removed by the target clean and also the les .depend, tags, obj, and any manual pages. make the dependencies for the source les, and store them in the le .depend.

5 February 2005 02:09

330 Table 192: bsd.prog.mk targets (continued) Target Purpose install

install the program and its manual pages. If the Makele does not itself dene the target install, the targets beforeinstall and afterinstall may also be used to cause actions immediately before and after the install target is executed. run lint on the source les. create a tags le for the source les.

lint tags

In addition, it supplies default denitions for the variables listed in Table 19-3. The operator ?= is used to ensure that they are not redened if they are already dened in the Makele (see page 324 for more details of the ?= operator).
Table 193: variables dened in bsd.prog.mk

Variable
BINGRP BINOWN BINMODE CLEANFILES

Purpose Group ownership for binaries. Defaults to bin. Owner for binaries. Defaults to bin. Permissions for binaries. Defaults to 555 (read and execute permission for everybody). Additional les that the clean and cleandir targets should remove. bsd.prog.mk does not dene this variable, but it adds the le strings to the list if the variable SHAREDSTRINGS is dened. Additional library dependencies for the target ${PROG}. For example, if you write DPADD=${LIBCOMPAT} ${LIBUTIL} in your Makele, the target depends on the compatibility and utility libraries. Dependent sourcesa list of source les that must exist before compiling the program source les. Usually for a building a conguration le that is required by all sources. Not all systems dene this variable. The C library. Defaults to /lib/libc.a. The 4.3BSD compatibility library. Defaults to /usr/lib/libcompat.a. The curses library. Defaults to /usr/lib/libcurses.a. The crypt library. Defaults to /usr/lib/libcrypt.a. The dbm library. Defaults to /usr/lib/libdbm.a. The des library. Defaults to /usr/lib/libdes.a. The lex library. Defaults to /usr/lib/libl.a.

DPADD

DPSRCS

LIBC LIBCOMPAT LIBCURSES LIBCRYPT LIBDBM LIBDES LIBL

5 February 2005 02:09

Chapter 19: Make Table 193: variables dened in bsd.prog.mk (continued)

331

Variable
LIBKDB LIBKRB LIBM LIBMP LIBPC LIBPLOT LIBTELNET LIBTERM LIBUTIL SRCS STRIP

Purpose Defaults to /usr/lib/libkdb.a. Defaults to /usr/lib/libkrb.a. The math library. Defaults to /usr/lib/libm.a. Defaults to /usr/lib/libmp.a. Defaults to /usr/lib/libpc.a. Defaults to /usr/lib/libplot.a. Defaults to /usr/lib/libtelnet.a. Defaults to /usr/lib/libterm.a. Defaults to /usr/lib/libutil.a. List of source les to build the program. Defaults to ${PROG}.c. If dened, this should be the ag passed to the install program to cause the binary to be stripped. It defaults to -s.

The variables in Table 19-4 are not dened in bsd.prog.mk, but will be used if they have been dened elsewhere:
Table 194: variables used by bsd.prog.mk Variable COPTS HIDEGAME LDADD LDFLAGS LINKS

Purpose Additional ags to supply to the compiler when compiling C object les. If dened, the binary is installed in /usr/games/hide, and a symbolic link is created to /usr/games/dm. Additional loader objects. Usually used for libraries. Additional loader ags. A list of pairs of le names to be linked together. For example LINKS= ${DESTDIR}/bin/test ${DESTDIR}/bin/[ links /bin/test to /bin/[. If set, make does not try to install man pages. This variable is dened only in bsd.prog.mk, and not in bsd.lib.mk or bsd.man.mk. The name of the program to build. If not supplied, nothing is built. List of source les to build the program. If SRC is not dened, its assumed to be ${PROG}.c.

NOMAN PROG SRCS

5 February 2005 02:09

332 Table 194: variables used by bsd.prog.mk (continued) Variable Purpose SHAREDSTRINGS SUBDIR

If dened, the Makele denes a new .c.o rule that uses xstr to create shared strings. A list of subdirectories that should be built as well as the targets in the main directory. Each target in the main Makele executes the same target in the subdirectories. Note that the name in this le is SUBDIR, though it has the same function as the variable SUBDIRS in bsd.subdir.mk.

There are a couple more points to note: If the le ../Makele.inc exists, it is included before the other denitions. This is one possibility for specifying site preferences, but of course it makes assumptions about the source tree structure, so its not completely general. The le bsd.man.mk is included unless the variable NOMAN is dened. Well take another look at bsd.man.mk on page 333.

bsd.lib.mk
bsd.lib.mk contains denitions for making library les. It supplies the same targets as bsd.prog.mk, but denes or uses a much more limited number of variables:
Table 195: Variables dened or used in bsd.lib.mk

Variable
LDADD LIB

Purpose Additional loader objects. The name of the library to build. The name is in the same form that you nd in the -l option to the C compilerif you want to build libfoo.a, you set LIB to foo. Target installation directory for libraries. Defaults to /usr/lib. Library group owner. Defaults to bin. Library owner. Defaults to bin. Library mode. Defaults to 444 (read access for everybody). Target directory for lint libraries. Defaults to /usr/libdata/lint. If set, only standard libraries are built. Otherwise (the default), both standard libraries (libfoo.a) and proling libraries (libfoo_p.a) are built.* List of source les to build the library. Unlike in bsd.prog.mk, there is no default value.

LIBDIR LIBGRP LIBOWN LIBMODE LINTLIBDIR NOPROFILE SRCS

5 February 2005 02:09

Chapter 19: Make

333

Given the choice of compiling foo.s or foo.c, bsd.lib.mk chooses foo.s. Like bsd.prog.mk, it includes bsd.man.mk. Unlike bsd.prog.mk, it does this even if NOMAN is dened.

bsd.subdir.mk
bsd.subdir.mk contains denitions for making les in subdirectories. Since only a single program target can be made per directory, BSD-style directory trees tend to have more branches than others, and each program is placed in its own subdirectory. For example, if I have three programs foo, bar and baz, I might normally write a Makele with the rule
all: foo: bar: baz: foo bar baz foo.c foobar.h conf.h bar.c foobar.h zot.h conf.h baz.c baz.h zot.h conf.h

As we have seen, this is not easy to do with the BSD conguration scheme. Instead, you might place all the les necessary to build foo in the subdirectory foo, and so on. You could then write
SUBDIRS = foo bar baz .include <bsd.subdir.mk>

foo/Makele could then contain

PROG = foo DPADD = foo.c foobar.h conf.h .include <bsd.prog.mk>

bsd.subdir.mk is structured in the same way as bsd.prog.mk. Use bsd.prog.mk for making les in the same directory, and bsd.subdir.mk for making les in subdirectories. If you want to do both, use bsd.prog.mk and dene SUBDIR instead of SUBDIRS.

bsd.man.mk
bsd.man.mk contains denitions for installing man pages. It is included from bsd.prog.mk and bsd.lib.mk, so the target and variables are available from both of these les as well. It denes the target maninstall, which installs the man pages and their links, and uses or denes the

A proling library is a library that contains additional code to aid prolers, programs that analyze the CPU usage of the program. We dont cover proling in this book.

5 February 2005 02:09

334

variables described in Table 19-6:

Table 196: Variables dened or used by bsd.man.mk

Variable
MANDIR

Meaning The base path of the installed man pages. Defaults to /usr/share/man/cat. The section number is appended directly to MANDIR, so that a man page foo.3 would be installed in /usr/share/man/cat3/foo.3. The group that owns the man pages. Defaults to bin. The owner of the man pages. Defaults to bin. The permissions of the installed man pages. Defaults to 444 (read permission for anybody). The subdirectory into which to install machine specic man pages. For example, i386 specic pages might be installed under /usr/share/man/cat4/i386. In this case, MANSUBDIR would be set to /i386. (n has the values 1 to 8). Manual page names, which should end in .[1-8]. If no MANn variable is dened, MAN1=${PROG}.1 is assumed. A list of pairs of names for manual page links. The rst lename in a pair must exist, and it is linked to the second name in the pair.

MANGRP MANOWN MANMODE MANSUBDIR

MANn MLINKS

bsd.own.mk
Not all variants of the BSD conguration system usebsd.own.mk. Where it is supplied, it contains default permissions, and may be used to override denitions in sys.mk, which includes it.

bsd.doc.mk
bsd.doc.mk contains denitions for formatting hardcopy documentation les. It varies signicantly between versions and omits even obvious things like formatting the document. It does, however, dene the variables in Table 19-7, which can be of use in your own Makele:
Table 197: Variables dened in bsd.doc.mk

Variable
PRINTER BIB COMPAT

Meaning Not a printer name at all, but an indicator of the kind of output format to be used. This is the argument to the troff ag -T. Defaults to ps (PostScript output). The name of the bib processor. Defaults to bib. Compatibility mode ag for groff when formatting documents with Berkeley me macros. Defaults to -C.

5 February 2005 02:09

Chapter 19: Make Table 197: Variables dened in bsd.doc.mk (continued)

335

Variable
EQN GREMLIN GRIND INDXBIB PAGES PIC REFER

Meaning How to invoke the eqn processor. Defaults to eqn -T${PRINTER}. The name of the gremlin processor. Defaults to grn. The name of the vgrind processor. Defaults to vgrind -f. Name of the indxbib processor. Defaults to indxbib. Specication of the page range to output. Defaults to 1-. Name of the pic processor. Defaults to pic. Name of the refer processor. Defaults to refer.

5 February 2005 02:09

Compilers
The central tool in building software is the compiler. In UNIX, the compiler is really a collection of programs that compile sources written in the C language. In this chapter, well consider the following topics: The way the C language has evolved since its introduction and some of the problems that this evolution has caused. C++, an evolution of C. The way the compiler is organized. How to use the individual parts of the compiler separately, in particular the assembler and the linker.

Well defer how the assembler and the linker work until the next chapterto understand them, we rst need to look at object les in more detail. There are, of course, lots of other languages besides C, but on a UNIX platform C is the most important. Even if you use another language, some of the information in this chapter will be of use to you: many other languages output C source code.

The C language
The C language has evolved a lot since its appearance in the early 70s. It started life as a Real Mans language, cryptic, small, tolerant of obscenities almost to the point of encouraging them, but now it has passed through countless committees and has put on weight and become somewhat sedate, pedantic and much less tolerant. Along with this, of course, it has developed a whole lot of idiosyncracies that plague the life of the portable software writer. First, lets take a look at the avours that are commonly available.

337

5 February 2005 02:09

338

Kernighan and Ritchie

Kernighan and Ritchie or K&R is the name given to the dialect of C described in the rst edition of The C programming language by Brian Kernighan and Dennie Ritchie. This was the rst book to describe the C language, and has become something of a bible. In 1988, a second edition appeared, which describes an early version of ANSI C, not K&R C. The K&R dialect is now completely obsolete, though many older versions of UNIX C resemble it. Compared to ANSI C (also called Standard C), it lacks a number of features, and has a few incompatibilities. In particular, strings were always allocated separately for each instance, and so could be modied if desired. For example, you could encounter code like this:
complain (msg) char *msg; { char *message = "Nothing to complain about\n"; if (msg) /* parameter supplied? */ strcpy (message, msg); /* yes, save in message */ puts (message); /* say what we have to say */ }

When the parameter msg is non-NULL, it is copied into the string message. If you call this function with a NULL message, it will display the last message again. For example:
complain (NULL); prints Nothing to complain about complain ("Bad style"); prints Bad style complain (NULL); prints Bad style

This may fail with modern C compilers: The ANSI Standard says that string constants are not writable, but real-world compilers differ in the way they handle this situation.

UNIX C
A period of over ten years elapsed between the publication of K&R and the nal adoption of the ANSI C standard. During this time, the language didnt stand still, but there was no effective standards document beyond K&R. The resultant evolution in the UNIX environment is based on the Portable C Compiler rst described in the paper Portability of C Programs and the UNIX System published by S. C. Johnson and Dennis Ritchie in 1978, and is frequently referred to as UNIX C. This is not a standard, or even a series of standardsits better to consider it a set of marginally compatible extensions to K&R C. You can nd more information in The evolution of CPast and Future by L. Rosler, but you cant rely on the degree to which your particular compiler (or the one for which your software package was written) agrees with that description. From a present-day standpoint, its enough to know that these extensions exist, and otherwise treat the compilers like K&R. In case of doubt, the documentation that comes with the compiler is about the only even remotely reliable help. Heres a brief summary of the sort of things that had changed by the time The evolution of CPast and Future appeared:

5 February 2005 02:09

Chapter 20: Compilers

339

Optional function prototyping similar to that of ANSI C was introduced. One difference exists: if a function accepts a variable number of parameters, UNIX C uses the form
int printf (char *format, );

whereas ANSI C uses the form

int printf (char *format, ...);

The enum type species a way to dene classes of constants. For example, traditionally I could write:
#define RED 0 #define GREEN 1 #define BLUE 2 int colour; int x; colour = BLUE; x = RED;

With enums, I can write

enum colours {red, green, blue}; enum texture {rough, smooth, slimy}; enum colours colour; enum texture x; colour = blue; x = red;

This syntax is intended to make error checking easier. As you can see in the second example, there seems to be something wrong with the assignment to x, which was not evident in the K&R example. The compiler can see it too, and should complain, although many modern compilers compile the second program without any comment. In addition, the symbols are visible to the compiler. This means that the debugger can use them as well: preprocessor macros never make it to the code generation pass of the compiler, so the debugger doesnt know about them. The keyword const was added to specify that a variable may not be changed in the course of program execution. The preprocessor directive #elif was added. The preprocessor pseudofunction defined (identifier) was added. The data type void was added.

5 February 2005 02:09

340

ANSI C
In 1989, the C programming language was nally standardized by the American National Standards Institute (ANSI) as standard X3.159-1989. In the following year it was adopted by the International Standards organization (ISO) as standard ISO/IEC 9899:1990. There are minor textual differences in the two standards, but the language dened is the same in each. The existence of two standards is good for a certain amount of confusion: some people call it ANSI C, some call it Standard C, and I suppose you could call it ISO C, though I havent heard that name. I call it ANSI C because the name is more specic: the word Standard doesnt make it obvious which standard is implied. The following discussion is intended to show the differences between ANSI C and older versions. Its not intended to teach you ANSI Csee Practical C Programming, by Steve Oualline, and the POSIX Programmers Guide by Donald A. Lewine for that information. ANSI C introduced a large number of changes, many of them designed to help the compiler detect program errors. You can nd a reasonably complete list in Appendix C of K&R. Here are the most important from the standpoint of porting software: A number of changes have been made in the preprocessor. Well look at these on page 342. The keywords void, signed and const were adopted from the Portable C compiler. The keyword volatile was added to tell an optimizer not to assume that the value of the variable will stay the same from one reference to another. Variables normally stay unchanged unless you execute an explicit assignment statement or call a function, and most optimizers rely on this behaviour. This assumption may not hold true if a signal interrupts the normal course of execution, or if you are sharing the memory with another process. If the value of a variable might change without your inuence, you should declare the variable volatile so that the optimizer can handle it correctly. We saw an example of this kind of problem in Chapter 13, Signals, page 200. You can state the type of numeric constants explicitly: for example, you can write a long constant 0 as 0L, and a double 0 would be 0D. Implicit string literal concatenation is allowed the following two lines are completely equivalent:
"first string" "second string" "first stringsecond string"

K&R C allows only the second form. void pointers are allowed. Previous versions of C allowed the type void, but not pointers to objects of that type. You use a void pointer to indicate the the object you are pointing to is of indeterminate type. Before you can use the data, you need to cast it to a specic data type. In strict ANSI C, you must declare or dene functions before you call them. You use a function declaration to tell the compiler that the function exists, what parameters it takes,

5 February 2005 02:09

Chapter 20: Compilers

341

and what value (if any) it returns. A function denition is the code for the function, and includes the declaration. Strict ANSI C function denitions and declarations include function protyping, which species the nature of each parameter, though most implementations allow old-style denitions. Consider the following function denition in K&R C:
foobar (a, b, c, d) char *c; struct baz *a; { body }

This denition does not specify the return type of the function; it may or may not return int. The types of two of the parameters are specied, the others default to int. The parameter type speciers are not in the order of the declaration. In ANSI C, this would become:
void foobar (struct baz *a, int b, char *c, int d) { body }

This denition states all types explicitly, so we can see that foobar does not, in fact, return any value. The same syntax can also be used to declare the function, though you can also abbreviate it to:
void foobar (struct baz *, int, char, int);

This helps catch one of the most insidious program bugs: consider the following code, which is perfectly legal K&R:
extern foobar ();/* define foobar without parameters */ int a, b; /* two integers */ struct baz *c; /* and a struct baz */ foobar (a, b, c);/* call foobar (int, int, struct baz *) */

In this example, I have supplied the parameters to foobar in the wrong sequence: the struct baz pointer is the rst parameter, not the third. In all likelihood, foobar will try to modify the struct baz, and will use the value of apossibly a small integer to do this. If I call foobar without parameters, the compiler wont notice, but by the time I get my almost inevitable segmentation violation, foobar will probably have overwritten the stack and removed all evidence of how the problem occurred.

5 February 2005 02:09

342

Differences in the ANSI C preprocessor

At rst sight, the C preprocessor doesnt seem to have changed between K&R C and ANSI C. This is intentional: for the casual user, everything is the same. When you scratch the surface, however, you discover a number of differences. The following list reects the logical sequence in which the preprocessor processes its input. A method called trigraphs represents characters not found in the native character set of some European countries. The following character sequences are considered identical:
Table 201: ANSI C trigraphs

character # [ \ ] { | }

trigraph ??= ??( ??/ ??) ?? ??< ??! ??> ??-

To show what this means, lets look at a possibly barely recognizable program:
??=include <unistd.h> main () ??< printf ("Hello, world??/n"); ??>

Not surprisingly, most programmers hate the things. To quote the gcc manual: You dont want to know about this brain-damage. Many C compilers, including the GNU C compiler, give you the opportunity to turn off support for trigraphs, since they can bite you when youre not expecting them. Any line may end with \, indicating that it should be splicedin other words, the preprocessor removes the \ character and the following newline character and joins the line to the following line. K&R C performed line splicing only during the denition of preprocessor macros. This can be dangerous: trailing blanks can nullify the meaning of the \ character, and its easy to oversee one when deleting lines that follow it. Unlike UNIX C, formal macro parameters in strings are not replaced by the actual parameters. In order to be able to create a string that includes an actual parameter, the operator # was introduced. A formal parameter preceded by a # is replaced by the actual parameter surrounded by string quotes. Thus

5 February 2005 02:09

Chapter 20: Compilers

#define foo(x) open (#x) foo (/usr/lib/libc.a);

343

will be replaced by
open ("/usr/lib/libc.a");

In many traditional versions of C, you could have got the same effect from:
#define foo(x) open ("x") foo (/usr/lib/libc.a);

In K&R C, problems frequently arose concatenating two parameters. Since both parameter names are valid identiers, you cant just write one after the other, because that would create a new valid identifer, and nothing would be substituted. For example, consider the X11 macro Concat, which joins two names together to create a complete path name from a directory and a le name:
Concat(dir, file);

I obviously cant just write

#define Concat(dir, file) dirfile

because that will always just give me the text dirfile, which isnt much use. The solution that the X Consortium used for K&R C was:
#define Concat(dir,file)dir/**/file

This relies on the fact that most C compilers derived from the portable C compiler simply remove comments and replace them by nothing. This works most of the time, but there is no basis for it in the standard, and some compilers replace the sequence /**/ with a blank, which breaks the mechanism. ANSI C introduced the operator ## to address this problem. ## removes itself and any white space (blanks or tab characters) to either side. For ANSI C, Imake.tmpl denes Concat as
#define Concat(dir,file)dir##file

The #include directive now allows the use of preprocessor directives as an argument. imake uses this to #include the <vendor>.cf le. Conditional compilation now includes the #elif directive, which signicantly simplies nested conditional compilation. In addition, a number of logical operators are available: || and && have the same meaning as in C, and the operator defined checks whether its operand is dened. This allows code like:
#if defined BSD || defined SVR4 || defined ULTRIX foo #elif defined SVR3 bar #endif

If you want, you can surround the operand of defined with parentheses, but you dont need to.

5 February 2005 02:09

344

The use of the preprocessor directive #line, which had existed in previous versions of C, was formalized. #line supports preprocessors that output C codesee page 88 for an example. #line tells the compiler to reset the internal line number and le name used for error reporting purposes to the specied values. For example if the le bar.c contains just
#line 264 "foo.c" slipup!

the compiler would report the error like this:

$ gcc -O bar.c -o bar foo.c:264: parse error before ! gnumake: *** [bar] Error 1

Although the error was really detected on line 2 of bar.c, the compiler reports the error as if it had occurred on line 264 of foo.c. The line slipup! suggests that it is there to draw attention to an error. This is a fairly common technique, though its obviously just a kludge, especially as the error message requires you to look at the source to gure out what the problem is. ANSI C introduced another directive to do the Right Thing. Instead of slipup!, I can enter:
#error Have not finished writing this thing yet

This produces (from gcc)

$ make bar gcc -O bar.c -o bar foo.c:270: #error Have not finished writing this thing yet gnumake: *** [bar] Error 1

I couldnt write Havent, because that causes gcc to look for a matching apostrophe (). Since there isnt one, it would die with a less obvious message, whether or not an error really occurred. To quote the Standard:
A preprocessor line of the form # pragma token-sequenceopt causes the processor to perform an implementation-dependent action. An unrecognized pragma is ignored.

This is not a Good Thing. Implementation-dependent actions are the enemy of portable software, and about the only redeeming fact is that the compiler ignores an unrecognized pragma. Since almost nobody uses this feature, you can hope that your compiler will, indeed, ignore any pragmas it nds.

Assertions
Assertions provide an alternative form of preprocessor conditional expression. They are specied in the form

5 February 2005 02:09

Chapter 20: Compilers

#assert question (answer)

345

In the terminology of the documentation, this asserts (states) that the answer to question is answer. You can test it with the construct:
#if #question(answer) ... #endif

The code between #if and #endif will be compiled if the answer to question is answer. An alternative way to use this facility is in combination with the compiler directive -Aquestion(answer). This method is intended for internal use by the compiler: typically, it tells the compiler the software and platform on which it is running. For example, compiling bar.c on UNIXWare 1.0 with gcc and the -v ag reveals:
/usr/local/lib/gcc-lib/i386-univel-sysv4.2/2.4.5/cpp \ -lang-c -v -undef -D__GNUC__=2 -Di386 -Dunix -D__svr4__ \ -D__i386__ -D__unix__ -D__svr4__ -D__i386 -D__unix \ -D__svr4__ -Asystem(unix) -Acpu(i386) -Amachine(i386) \ bar.c /usr/tmp/cca000Nl.i

The -A ags passed by gcc to the preprocessor specify that this is a unix system and that the cpu and machine are both i386. It would be nice if this information stated that the operating system was svr4, but unfortunately this is not the default for System V.4. gcc has also retrotted it to System V.3, where the assertion is -Asystem(svr3), which makes more sense, and to BSD systems, where the assertion is -Asystem(bsd).

C++
C++ is an object-oriented evolution of C that started in the early 80s, long before the ANSI C standard evolved. It is almost completely upwardly compatible with ANSI C, to a large extent because ANSI C borrowed heavily from C++, so we dont run much danger by considering it the next evolutionary step beyond ANSI C. The last thing I want to do here is explain the differences between ANSI C and C++: The Annotated C++ Reference Manual, by Margaret A. Ellis and Bjarne Stroustrup, spends nearly 450 very carefully written pages dening the language and drawing attention to its peculiarities. From our point of view, there is not too much to say about C++. One of the more popular C++ translators is AT&Ts cfront, which, as the name suggests, is a front-end preprocessor that generates C program code as its output. Although this does not make the generated code any worse, it does make debugging much more difcult. Since C++ is almost completely upwards compatible from ANSI C, a C++ compiler can usually compile ANSI C. This assumes well-formed ANSI C programs: most ANSI C compilers accept a number of anachronisms either with or without warnings for example, K&R-style function denitions. The same anachronisms are no longer part of the C++ language, and cause the compilation to fail. C++ is so much bigger than C that it is not practicable to even think about converting a C++ program to C. Unless there are some really pressing reasons, its a whole lot easier to get hold

5 February 2005 02:09

346

of the current version of the GNU C compiler, which can compile both C and C++ (and Objective C, if youre interested). C and C++ have different function linking conventions. Since every C++ program calls C library functions, there is potential for errors if you use the wrong calling convention. We looked at this aspect in Chapter 17, Header les, on page 286.

Other C dialects
Before the advent of ANSI C, the language was ported to a number of non-UNIX architectures. Some of these added incompatible extensions. Many added incompatible library calls. One area is of particular interest: the Microsoft C compiler, which was originally written for MS-DOS. It was subsequently adapted to run under XENIX and SCO UNIX System V. Since our emphasis is on UNIX and UNIX-like systems, well talk about the XENIX compiler, though the considerations also apply to SCO UNIX and MS-DOS. The most obvious difference between the XENIX C compiler and most UNIX compilers is in the ags, which are described in Appendix B, Compiler ags, but a couple of architectural limitations have caused incompatibilities in the language. Well look at them in the following section.

Intel 8086 memory models

The original MS-DOS compiler ran on the Intel 8086 architecture. This architecture has 1 MB of real memory, but the addresses are only 16 bits long. In order to address memory, each machine instruction implicitly adds the contents one of four segment registers to the address, so at any one time the machine can address a total of 256 kB of memory. In order to address more memory, the C implementation denes a 32 bit pointer type, the so-called far address, in software. Accessing memory via a far pointer requires reloading a segment register before the access, and is thus signicantly slower than access via a 16-bit near address. This has a number of consequences: Near addresses are simply offsets within a segment: if the program expects it to point to a different segment, it will access the wrong data. Far pointers are 32 bits wide, containing the contents of the segment register in one half and the offset within the segment in the other half. The segment register contains bits 4 through 19 of a 20-bit address, and the offset contains bits 0 through 15. To create an absolute address from a far pointer, the hardware performs effectively
struct fp { short segment_reg; /* 16 bits, bits 4 through 19 of address */ short offset; /* 16 bits, bits 0 through 15 of address */ } long abs_address = (fp.segment_reg << 4) + fp.offset;

As a result, many possible far pointer contents that could resolve to the same address. This complicates pointer comparison signicantly. Some implementations solved this problem by declaring huge pointers, which are normalized 20-bit addresses in 32-bit words.

5 February 2005 02:09

Chapter 20: Compilers

347

Along with three pointer types, MS-DOS C uses a number of different executable formats. Each of them has default pointer sizes associated with them. You choose your model by supplying the appropriate ag to the compiler, and you can override the default pointer sizes with the explicit use of the keywords near, far or (where available) huge: The tiny model occupies a single segment and thus can always use near addresses. Apart from the obvious compactness of the code, this model has the advantage that it can be converted to a .COM le. The small model occupies a single data segment and a single code segment. Here, too, you can always use near pointers, but you need to be sure youre pointing into the correct segment. The medium model (sometimes called middle model) has multiple code segments and a single data segment. As a result, code pointers are far and data pointers are near. The compact model is the inverse of the medium model. Here, code is restricted to one segment, and data can have multiple segemnts. Static data is restricted to a single segment. As a result, code pointers are near and data pointers are far. The large model can have multiple code and multiple data segments. Static data is restricted to a single segment. All pointers are far. The huge model is like the large model except that it can have multiple static data segments. The name is unfortunate, since it suggests some connection with huge pointers. In fact, the huge model uses far pointers.

What does this mean to you? If youre porting from MS-DOS to UNIX, you may run into these keywords near, far and huge. This isnt a big deal: you just need to remove them, or better still, dene them as an empty string. You may also nd a lot of pointer checking code, which will probably get quite confused in a UNIX environment. If you do nd this kind of code, the best thing to do is to ifdef it out (#ifndef unix). If youre converting from UNIX to MS-DOS, things can be a lot more complicated. Youll be better off using a 32-bit compiler, which doesnt need this kind of kludge. Otherwise you may have to spend a considerable amount of time guring out the memory architecture most suitable for your package.

Other differences in MS-DOS

MS-DOS compilers grew up in a very different environment from UNIX. As a result, a number of detail differences exist. None of them are very serious, but its good to be forewarned: They do not adhere to the traditional UNIX organization of preprocessor, compiler, assembler and loader. They dont use the assembler directly, though they can usually output assembler code for use outside the compilation environment.

5 February 2005 02:09

348

The assembler code output by MS-DOS compilers is in the standard Intel mnemonics, which are not compatible with UNIX assemblers. Many MS-DOS compilers combine the preprocessor and the main compiler pass, which makes for faster compilation and less disk I/O. Many rely on the Microsoft linker, which was not originally written for C, and which has signicant limitations. Many MS-DOS compilers still run in real mode, which limits them to 640K code and data. This is a severe limitation, and it is not uncommon to have to modify programs in order to prevent the compiler from dying of amnesia. This leads to a different approach with header les, in particular: in UNIX, its common to declare everything just in case, whereas in MS-DOS it may be a better idea to not declare anything unless absolutely necessary.

Compiler organization
The traditional UNIX compiler is derived from the Portable C Compiler and divides the compilation into four steps, traditionally called phases or passes, controlled by the compiler control program cc. Most more modern compilers also adhere to this structure: 1. 2. 3. 4. The preprocessor, called cpp, reads in the the source les and handles the preprocessor directives (those starting with #) and performs macro substitution. The compiler itself, usually called cc1, reads in the preprocessor output and compiles to assembler source code. In SunOS, this pass is called ccom. The assembler as reads in this output and assembles it, producing an object le. The loader takes the object le or les and links them together to form an executable. To do so, it also loads a low-level initialization le, normally called crt0.o, and searches a number or libraries.

cc usually performs these passes invisibly. The intermediate outputs are stored in temporary les or pipes from one pass to the next. It is possible, however, to call the passes directly or to tell cc which pass to execute well look at how to do that in the next section. By convention, a number of sufxes are used to describe the intermediate les. For example, the GNU

5 February 2005 02:09

Chapter 20: Compilers

349

C compiler recognizes the following sufxes for a program foo:

Table 202: C compiler intermediate les

le foo.c foo.cc foo.cxx foo.C foo.i foo.ii foo.m foo.h foo.s foo.S foo.o

contents unpreprocessed C source code unpreprocessed C++ source code unpreprocessed C++ source code unpreprocessed C++ source code preprocessed C source code preprocessed C++ source code Objective C source code C header le assembler source code assembler code requiring preprocessing object le

created by compiler?

yes yes

yes yes

Heres what you need to do to go through the compilation of foo.c to the executable foo, one pass at a time:
$ $ $ $ gcc gcc gcc gcc -E foo.c -o foo.i -S foo.i -c foo.s foo.o -o foo preprocess compile assemble link

There are slight variations in the form of the commands: if you dont tell the preprocessor where to put the output le, gcc writes it to stdout. Other preprocessors may put a special sufx on the base le name, or if you specify the -o ag, the compiler might put it in the le you specify. If you dont tell the linker where to put the output le, it writes to a.out. Compiling an object le from an assembler le is the same as compiling from a source le or a preprocessed legcc decides what to do based on the sufx of the input le. You can also run any combination of contiguous passes like this:
$ $ $ $ $ $ gcc gcc gcc gcc gcc gcc -S -c -o -c -o -o foo.c foo.c foo foo.c foo.i foo foo.i foo foo.s preprocess and compile preprocess, compile and assemble preprocess, compile, assemble, link compile and assemble compile, assemble, link assemble and link

The location of the C compiler is, unfortunately, anything but standardized. The control program cc is normally in /usr/bin, or occasionally in /bin, but the other components might be stored in any of the following: /usr/lib, /usr/ccs/lib (System V.4), /usr/lib/cmplrs/cc (MIPS) or /usr/local/lib/gcc-lib (gcc on most systems).

5 February 2005 02:09

350

Other compiler organizations

Some modern compilers have additional passes. Some optimizers t between the compiler and the assembler: they take the output of the compiler and output optimized code to the assembler. An extreme example is the MIPS compiler, which has a total of 8 passes: The preprocessor cpp, the front end cc1, the ucode* linker uld, the procedure merge pass umerge, the global optimizer uopt, the code generator ugen, the assembler as1, and the linker ld. Despite this apparent complexity, you can consider this compiler as if it had only the traditional four passes: the ve passes from the front end up to the code generator perform the same function as the traditional cc1.

The C preprocessor
You can use the preprocessor cpp for other purposes than preprocessing C source code: it is a reasonably good macro processor, and it has the advantage that its functionality is available on every system with a C compiler, though in some cases it is available only via the C compiler. It is one of the mainstays of imake, and occasionally packages use it for other purposes as well. There are two ways to invoke cpp: you can invoke it with cc and the -E ag, or you can start it directly. If at all possible, you should start it via cc rather than running it directly. On some systems you cant rely on cc to pass the correct ags to cpp. You also cant rely on all versions of cpp to use the same agsyou cant even rely on them to be documented. You can nd a list comparing the more common preprocessor ags in Appendix B, Compiler ags, page .

Which compiler to use

Most systems still supply a C compiler, and normally this is the one you would use. In some cases, bugs in the native system compiler, compatibility problems, or just the fact that you dont have the normal compiler may lead to your using a different compiler. This situation is becoming more common as software manufacturers unbundle their compilers. Using a different compiler is not necessarily a Bad Thing, and can frequently be an improvement. In particular, gcc, the GNU C compiler from the Free Software Foundation, is very popularits the standard C compiler for a number of systems, including OSF/1, 4.4BSD, and Linux. It can do just about everything except run in minimal memory, and it has the advantage of being a well-used compiler: chances are that somebody has compiled your package with gcc before, so you are less likely to run into trouble with gcc than with the native compiler of a less-known system. In addition, gcc is capable of highly optimized code, in many cases signicantly better than the code created by the native compiler. Compilers are becoming more standardized, and so are the bugs you are liable to run into. If you have the choice between compiling for K&R or ANSI, choose ANSI: the K&R ags may
* ucode is a kind of intermediate code used by the compiler. It is visible to the user, and you have the option of building and using ucode libraries.

5 February 2005 02:09

Chapter 20: Compilers

351

use features that were not universally implemented, whereas the ANSI versions tend to pay more attention to the standard. If you do run into a bug, chances are someone has seen it before and has taken steps to work around it. In addition, compiling for ANSI usually means that the prototypes are declared in ANSI fashion, which increases the chance of subtle type conicts being caught. Some things that neither you nor the Makele may expect are: gcc compiles both K&R (-traditional) and ANSI dialects. However, even some software supplied by the Free Software Foundation breaks when compiled with gcc unless the -traditional ag is used. Many compilers do not compile correctly when both optimization and debugging information are specied (-O and -g ags), though most of them recognize the fact and turn off one of the ags. Even if the compiler ostensibly supports both ags together, bugs may prevent it from working well. For example, gcc version 2.3.3 generated invalid assembler output for System V.4 C++ programs if both ags were specied. Even when compilers do create debugging information from an optimizing compilation, the results can be confusing due to the action of the optimizer: The optimizer may remove variables. As a result, you may not be able to set or display their values. The optimizer may rearrange program ow. This means that single-stepping might not do what you expect, and you may not be able to set breakpoints on certain lines because the code there has been eliminated. Some optimizers remove stack frames,* which makes for faster code, particularly for small functions. gcc will do this with the -O3 option.

Stack frame removal in particular makes debugging almost impossible. These arent bugs, theyre features. If they cause problems for you, you will need to recompile without optimization. Some compilers limit the length of identiers. This can cause the compiler to treat two different identiers as the same thing. The best thing to do if you run into this problem is to change the compiler: modern compilers dont have such limits, and a compiler that does is liable to have more tricks in store for you. With a System V compiler, you might nd:
$ cc -c frotzel.c -o frotzel.o cc: Error: -o would overwrite frotzel.o

System V compilers use the ag -o only to specify the name of the nal executable, which must not coincide with the name of the object le. In many Makeles from the BSD world, on the other hand, this is the standard default rule for compiling from .c to .o.
* See Chapter 21, Object les and friends, page 377, for further information on stack frames.

5 February 2005 02:09

352

All C compilers expect at least some of their ags in a particular sequence. The documentation is frequently hazy about which operands are sequence-sensitive, or what interactions there are between specic operands.

The last problem bears some more discussion. A well-documented example is that the linker searchs library specications (the -l option) in the sequence in which they are specied on the compiler invocation linewell investigate that in more detail in Chapter 21, Object les and friends, page 373. Heres an example of another operand sequence problem:
$ cc foo.c -I../myheaders

If foo.c refers to a le bar.h in the directory ../myheaders, some compilers wont nd the header because they dont interpret the -I directive until after they try to compile foo.c. The man page for System V.4 cc does state that the compiler searches directories specied with -I in the order in which they occur, but it does not relate this to the sequence of operands and le names.

5 February 2005 02:09

Object les and friends

Object les are a special kind of le which store compiled programs. Normally, you manipulate them only as part of the build process, where you can treat them as a black box: you dont need to know what they look like inside. Sometimes, however, some aspects of the true nature of object les become apparentfor example: Your program bombs out with a segmentation violation, and when you check up you discover that it was trying to write to a valid storage locationso why did it bomb out? It might be that the location was in the text segment, a part of the address space that is read-only. You want to debug a program, and nd that the debugger refuses to look at it, because it doesnt have any symbols whatever that may mean. You recompile programs and run out of disk spacefor some reason, the object les are suddenly ten times the size that they used to be.

The information in this chapter is some of the most technical in the whole book, which is why Ive left it to the end. We look at a number of topics that are related only by their dependence on object les. So far, the inter-platform differences weve seen have been the result of a choice made by the software people who implemented the system. In this chapter, we come a whole lot closer to the hardware you can almost feel the clocks tick and the pipelines ll. You denitely see instructions execute. Youll nd it an interesting look below covers that are usually locked shut. A number of programs manipulate the object les either because thats their purposefor example, assemblers or linkers or because they want to play tricks to install more comfortably. For example, emacs and T X both write themselves out as object les during the build E process. If anything goes wrong with these programs, you need to open the black box and look inside. In this chapter, well examine the tools that manipulate object les and some of the background information that you need to know to solve problems. There arent many programs that manipulate object les. The kernel uses absolute object les when creating a processthis is the most frequent use of an object le. In addition, the assembler creates them from assembly sources. In most UNIX systems, this is the only
353

6 February 2005 00:57

354

program that creates object les from scratch. The linker or link editor joins object les together to form a larger object le, and debuggers access specic debugging information in the object le. These are the only programs that have intimate understanding of the object le format. A number of smaller programs do relatively trivial things with object les: The archiver ar is normally used for archiving binary les, but it does not know very much about their contents. The name list display program nm displays the symbol table or name list of an object le or an archive of object les. Well look at the symbol table in more detail on page 363. size displays size information from an object le. strings displays printable strings in an object le. strip removes unnecessary information from an object le. The kernel process model that the object le supports. The assembler, including some of the syntax, the symbol table, relocation, and debugging symbols. The linker, including the way it searches libraries, and some of the problems that can occur during linking. The internal structure of function libraries, and how this affects what they can and cannot do. How emacs and T X dump themselves as object les. E How exec starts programs.

In the rest of this chapter, well look at the following topics:

Object formats
The purpose of object les is to make it as easy as possible to start a process, so it makes sense to look at the process image in memory rst. Modern UNIX systems run on stackbased systems with virtual memory. We touched on the concept of virtual memory in Chapter 11, Hardware dependencies, on page 155. Since UNIX is a multiprogramming system, it is possible for more than one process to run from a single object le. These facts have a signicant inuence on the way the system manages processes. For each process, the system allocates at least three segments in which program code and data is stored: A text segment, which contains the executable code of the program and read-only data. Modern systems create code where the program may not write to its text segment it is so-called pure text. This has two signicant advantages for the memory manager: rst, all processes in the system that are run from this program can share the same text segment, which signicantly reduces the memory requirements when, say, 20 copies of a shell are running. In addition, since the text is not modied, the memory management

6 February 2005 00:57

Chapter 21: Object les

355

routines never need to swap it out to disk. The copy on disk is always the same as the copy in memory. This also means that the copy on disk can be the copy in the object le: it does not take up any space in the swap partition. Older systems also provided for impure text segments that could be modied by the program. This usage is obsolete, but it is still supported by modern systems. A data segment. This consists of two parts: Global data that has been initialized in the program. This data can be modied, of course, so it takes up space in the swap partition, but the rst time the page is referenced, the memory manager must load it from the object le. bss* data, non-initialized global data. Since the data is not initialized, it does not need to be loaded from a le. The rst time the page is referenced, the memory manager just creates an empty data page. After that, it gets paged to the swap partition in the same way as initialized data.

A stack segment. Like bss data, the stack segment is not initialized, and so is not stored in the object le. Unlike any of the other segments, it does not contain any xed addresses: at the beginning of program execution, it is almost empty, and all data stored in it is relative to the top of the stack or another stack marker. Well look at stack organization in more detail on page 377. In addition, many systems have library segments. From the point of view of memory management, these segments are just additional text and data segments, but they are loaded at run time from another object le, the library le.

Older systems without virtual memory stored the data segment below the stack segment with a gap in between, the so-called break. The stack grew down into the break as the result of push or call instructions, and the data segment grew up into the break as the result of system calls brk and sbrk (set break). This additional space in the data segment is typically used for memory allocated by the library call malloc. With a virtual memory system, the call to sbrk is no longer necessary, but some versions of UNIX still require it, and all support it. Table 21-1 summarizes this information:

* The name comes from the assembler directive bss (Block Starting with Symbol), which was used in older assemblers to allocate uninitialized memory and allocate the address of the rst word to the label of the directive. There was also a directive bes (Block Ending with Symbol) which allocated the address of the last word to the label.

6 February 2005 00:57

356 Table 211: Kinds of segments

Property In object le Access Paged out Fixed size

Text Segment yes r-x no yes

Initialized Data yes rwyes yes

bss Data no rwyes maybe

Stack Segment no rwyes no

Object les contain the information needed to set up these segments. Before we continue, we should be aware of a terminology change: The object le for a process is called a program. The images of process segments in an object le are called sections. The a.out format is the oldest, and has remained essentially unchanged since the Seventh Edition. It supplies support for a text section and a data section, as well as relocation information for both sections. It is used by XENIX and BSD systems. The COFF (Common Object File Format) was introduced in System V, and offers an essentially unlimited number of segments, including library segments. It is now obsolescent, except for use in Microsoft Windows NT. The ELF (Executable and Linking Format) format was introduced for System V.4. From our point of view, it offers essentially the same features as COFF. ELF shows promise as the executable format of the future, since it greatly simplies the use of shared libraries. Currently the Linux project is moving from a.out to ELF.

There are three main object le formats in current use:

With the exception of library segments, theres not much to choose between the individual object formats, but the internal structures and the routines that handle them are very different. Lets take an a.out header from a BSD system as an example. The header le sys/exec.h denes:
struct exec { long a_magic; unsigned long a_text; unsigned long a_data; unsigned long a_bss; unsigned long a_syms; unsigned long a_entry; unsigned long a_trsize; unsigned long a_drsize; }; /* a_magic */ #define OMAGIC

/* /* /* /* /* /* /* /*

magic number */ text segment size */ initialized data size */ uninitialized data size */ symbol table size */ entry point */ text relocation size */ data relocation size */

0407 /* old impure format */

6 February 2005 00:57

Chapter 21: Object les

#define NMAGIC #define ZMAGIC #define QMAGIC 0410 /* read-only text */ 0413 /* demand load format */ 0314 /* compact demand load format */

357

This header includes: A magic number. This species the exact kind of le (for example, whether it is relocatable or absolute). The program le can interpret this magic number and report the kind of object le. The length of the text section, an image of the text segment. The text section immediately follows the header. The length of the data section, an image of the initialized global data part of the data segment as we have seen, bss data does not need to be stored in the object le. The data section immedately follows the text section. The length of the bss data. Since the bss data is not initialized, no space is needed for it in the object le. The length of the symbol table. The symbol table itself is stored after the data section. The entry point, the address in the text segment at which execution is to start.

The lengths of the text and data relocation tables, which are stored after the symbol table. If you look at the above list of contents carefully, youll notice that there are no start addresses for the segments, and there isnt even any mention of the stack segment. The start address of the text and data segments is implicit in the format, and its frequently difcult information to gure out. On 32 bit machines, the text segment typically starts at a low address, for example 0 or 0x1000.* The data segment may start immediately after the text segment (on the following page), or it might start at a predetermined location such as 0x40000000. The stack segment is usually placed high in the address space. Some systems place it at 0x7fffffff, others at 0xefffffff. The best way to nd out these addresses is to look through the address space of a typical process with a symbolic debugger. The magic number is worth closer examination: I said that it occupies the rst two bytes of the header, but in our example it is a long, four bytes. In fact, the magic number is used in two different contexts: The rst two bytes in the le are reserved for the magic number in all systems. The information in these bytes should be sufcient to distinguish the architecture. The following two bytes may contain additional information for specic systems, but it is often set to 0.

* Why 0x1000? Its a wonderful debugging aid for catching NULL pointers. If the rst page of memory is not mapped, youll get a segmentation violation or a bus error if you try to access data at that address

6 February 2005 00:57

358

The assembler
Assembly is the second oldest form of programming*. It is characterized by being specic about the exact instructions that the machine executes, which makes an assembler program much more voluminous than a higher level language. Nevertheless, there is nothing difcult about it, its just tedious. Assembler programming involves two aspects that dont have much in common: The instruction set of the machine in question. The best source of information for this aspect is the hardware description of the machine. Even if you get an assembler manual for the machine, it will not be as authoratative as the hardware description. The syntax of the assembler. This is where the problems start: rst, little documentation is available, and secondly, assembler syntax diverges greatly, and the documentation you get may not match your assembler.

The i386 is a particularly sorry example of incompatible assembler syntax. The UNIX assemblers available for the i386 (at least three of them, not even compatible with each other) use modied forms of the old UNIX as syntax, whereas all books about the assembler and the hardware of the i386 use a syntax related to the Microsoft assembler MASM. They dont even agree on such basic things as the names of the instructions and the sequence of the operands. Although nowadays it is used almost only for assembling compiler output, as frequently offers features specically intended for human programmers. In particular, most assemblers support some kind of preprocessing: they may use the macro preprocessor m4 or the C preprocessor when processing assembler source. See the description of the ags in Appendix C, Assembler directives and ags, page 415, for more information.

Assembler syntax
Assembler syntax is a relatively involved topic, but there are some general rules that apply to just about every assembler. In this section, well see how to ght our way through an assembler listing. Assemblers are line-oriented: each instruction to the assembler is placed on a separate line. An instruction line consists of four parts: If the optional label is present, the assembler assigns a value to it. For most instructions, the value is the current value of the location counter, the relative address of the instruction in the current section. In UNIX, if the label is present it is followed by a colon (:). Other assemblers frequently require that only labels start at the beginning of the line, and recognize them by this fact.

* The oldest form of programming, of course, used no computational aids whatsoever: in some form or another, the programmer wrote down direct machine code and then entered into memory with a loader or via front-panel switches. Assembly added the symbolic character to this operation.

6 February 2005 00:57

Chapter 21: Object les

359

The assembler usually translates the source le in a single pass. This means that when it encounters the name of a label that is further down in the source le, it cannot know its value or even if it exists. Some assemblers require that the name of the label be followed with the letter b (backwards) for labels that should have already been seen in the text, and f (forwards) for labels that are further down. In order to avoid ambiguity, these assemblers also require that the labels be all digits. Many other assemblers also support this syntax, so 1b is not a good name for a label. The next eld is the instruction. In this context, assembler instructions are commands to the assembler, and may be either directives, which tell the assembler to do something not directly related to emitting code, or machine instructions, which emit code. In UNIX, directives frequently start with a period (.). The third eld contains the operands for the instruction. Depending on the instruction, they may not be required. The fourth eld is a comment eld. It is usually delimited by a hash mark (#).

The operands of instructions that take a source operand and a destination operand are usually specied in the sequence src, dest. Register names are usually preceded with a % sign. Literal values are usually preceded with a $ sign.

For example, consider the instruction:

fred: movl $4,%eax # example

This instruction emits a movl instruction, which moves the literal* value 4 to the register eax. The symbol fred is set to the address of the instruction. We cant go into all the details of the assembly language for all machines, but the descriptions in Appendix C, Assembler directives and ags, page 415, will hopefully give you enough insight to be able to read existing assembler source, though youll need more information before you can write it. One of the few reasonably detailed as manuals is Using as, by Dean Elsner and Jay Fenlason, which is included as part of the GNU binutils distribution.

Assembler symbols
Local symbols dene instruction addresses. High-level constructs in C such as if, while and switch require a number of jump (go to) instructions in assembler, and the compiler must generate labels for the instructions. Local symbols are also used to label literal data constants such as strings. Global symbols dened in the source. The word global has different meanings in C and assembler: in C, it is any symbol dened in the data or text segments, whether or not it is
* movl means move long, not move literal. In this particular assembler, we know that it is a literal value because of the $ symbol, just as we know that eax is a register name because it is preceded by a % sign.

6 February 2005 00:57

360

visible outside the module. In assembler, a global symbol is one that is visible outside the module. There are a couple of points to note here: C local variables are generated automatically on the stack and do not retain their names after compilation. They do not have a xed location, since their position on the stack depends on what was already on the stack when the function was called. If the function is recursive, they could even be in many different places on the stack at the same time. As a result, there is nothing that the assembler or the linker can do with the symbols, and the compiler discards them. There is a possibility of conict between the local symbols generated by the compiler and global symbols declared in the program. Most compilers avoid this conict by prepending an underscore (_) to all symbols dened in the program, and not using the underscore for local symbols. Others solve the problem by prepending local symbols with a period (.), which is not legal in a C identier.

To see how this all works, lets take the following small program and look at different aspects of what the compiler and assembler do with it in the next few sections:
Example 211:
char global_text [] = "This is global text in the data area"; void inc (int *x, int *y) { if (*x) (*x)++; else (*y)++; puts (global_text); /* this is an external function */ puts ("Thats all, folks"); }

We compile this program on a BSD/OS machine using gcc version 2.5.8, with maximum optimization and debugging symbols:
$ gcc -O2 -g -S winc.c

The -S ag tells the compiler control program to stop after running the compiler. It stores the assembly output in winc.s, which looks like this:
Example 212:
.file "winc.c" gcc2_compiled.: ___gnu_compiled_c: .stabs "/usr/lemis/book/porting/grot/",100,0,0,Ltext0 name of the source directory .stabs "winc.c",100,0,0,Ltext0 name of the source file .text select text section Ltext0: internal label: start of text .stabs "int:t1=r1;-2147483648;2147483647;",128,0,0,0 .stabs "char:t2=r2;0;127;",128,0,0,0 ... a whole lot of standard debugging output omitted

6 February 2005 00:57

Chapter 21: Object les Example 212: (continued)

.stabs "void:t19=19",128,0,0,0 .globl _global_text specify an externally defined symbol .data select data section .stabs "global_text:G20=ar1;0;36;2",32,0,1,0 debug info for global symbol _global_text: variable label .ascii "This is global text in the data area " and text .text select text section LC0: .ascii "Thats all, folks " .align 2 start on a 16 bit boundary .globl _inc define the function inc to be external _inc: start of function inc .stabd 68,0,3 debug information: start of line 3 pushl %ebp movl %esp,%ebp movl 8(%ebp),%eax movl 12(%ebp),%edx .stabd 68,0,4 debug information: start of line 4 LBB2: cmpl $0,(%eax) je L2 .stabd 68,0,5 debug information: start of line 5 incl (%eax) jmp L3 .align 2,0x90 L2: .stabd 68,0,7 debug information: start of line 7 incl (%edx) L3: .stabd 68,0,8 debug information: start of line 8 pushl $_global_text call _puts .stabd 68,0,9 debug information: start of line 9 pushl $LC0 call _puts .stabd 68,0,10 debug information: start of line 10 LBE2: leave ret .stabs "inc:F19",36,0,3,_inc debug information for inc .stabs "x:p21=*1",160,0,2,8 debug information for x .stabs "y:p21",160,0,2,1 debug information for y .stabs "x:r21",64,0,2,0 .stabs "y:r21",64,0,2,2 .stabn 192,0,0,LBB2 .stabn 224,0,0,LBE2

361

Well look at various aspects of this output in the next few sections. For now, we should notice: As advertised, the names of the global symbols global_text, inc and puts have been changed to _global_text, _inc and _puts.

6 February 2005 00:57

362

The compiler has created the local symbols Ltext0, LC0, LBB2, LBE2, L2 and L3. Clearly it likes to start the names of local symbols with the letter L, and distinguish them with numbers at the end. But what has happened to L1, for example? The compiler generated it, but the optimizer optimized it away. If you compile this same program without the optimizer, the labels will all still be there. The compiler has assigned the local symbol LC0 to the string "Thats all, folks" so that the assembler can refer to it. The variables x and y have disappeared, since they exist only on the stack.

Relocation information
Example 21-2 shows another dilemma that aficts the linker: the program is not complete. It refers to the external symbol _puts, and in addition it does not have a main function: the only way to use it is for a function in another object le to call _inc. In order to do this, we need to give the linker additional information: Information about the names of the external symbols that the object le references (_puts in our example). Information about symbols dened in the object le that can be referenced by other object les (_global_text and _inc in our example). Information about where external symbols are referenced in the object code. Information about where locations in the text and data segments are referenced in the object code.

Why do we need to know where internal locations are referenced? The linker takes the text and data sections from a large number of object les and makes a single text section and a single data section out of them. The locations of the original sections from the individual object les differ from one occasion to the next, but the addresses in the nal executable must reect the correct references. If an instruction refers to an address in the data or text section or an external symbol, the assembler cant just put the address of the item in the instruction, since the address is allocated by the linker. Instead, it places the offset from the beginning of the text or data section or from the external symbol into the instruction or data word, and generates a relocation record in the output le. These relocation records contain the following information: The address of the data that needs to be relocated. From the linkers point of view, the data may be an instruction, in which case it will need to modify only the address portion of the instruction, or it may be a pointer, in other words an indirect address. The length of the data. For a data pointer, this is the length of the pointer. For an instruction, it is the length of the address eld of the instruction. Some systems have strange instruction formats, and this can become quite complicated. Information about the section in which the data will be located. This could be the current text or data section, or it could be a reference to an external symbol.

6 February 2005 00:57

Chapter 21: Object les

363

For an external symbol, a pointer to information about the symbol.

Object les contain separate relocation tables for each section that can contain address data at least the text and data sections. Referring again to Example 21-2, we see that the compiler has output .text and .data directives. These are used to tell the assembler in which section it should put the output that follows. It also supplies relocation information for the output le.

String table and name list

Amongst other things, the relocation information includes a signicant number of strings. These are stored in the string table, which is simply a list of strings terminated with a NUL (\0) character. Other parts of the object le refer to strings by their offset in the string table. As we saw in Example 21-2, the assembler has a directive (.globl in this example) that outputs information about externally visible symbols, such as global_text. Some assemblers need to be told about external references (such as _puts in this example), and others dont, like the GNU assembler gas used here. For both external denitions and external references, gas outputs an entry for the symbol table to the output le with information about the symbol. This symbol table is one of the better-known parts of an object le, and is usually called the name list. The structure differs strongly from one avour of UNIX to the next, but all namelists contain the following information: The index of the symbols name in the string table. The type of the symbol (undened, absolute, text, data, bss, common). The value of the symbol, if it has one (undened symbols dont, of course).

The library function nlist accesses the symbol table and returns a symbol table entry. The call is
#include <nlist.h> int nlist (const char *filename, struct nlist *nl);

This function has confusing semantics: the symbol table structure struct nlist does not contain the name of the symbol. Instead, it contains a pointer to the name of the symbol. On disk, the symbol is located in the string list, but in your program you supply the strings in advance. For the System V.4 ELF format, the structure is
struct nlist { char *n_name; long n_value; short n_scnum; unsigned short n_type; char n_sclass; char n_numaux; };

/* /* /* /* /* /*

name of symbol */ value of symbol */ section number */ type and derived type */ storage class */ number of auxiliary entries */

6 February 2005 00:57

364

To use the nlist function, you create an array of struct nlist entries with n_name set to the symbols you are looking for. The last entry contains a null string to indicate the end of the list. nlist searches the symbol table and lls in information for the symbols it nds, and sets all elds except n_name to 0 if it cant nd the string. The return value differs from one system to another: If lename doesnt exist, or if it isnt an object le, nlist returns -1. If all symbols were found, nlist returns 0. If some symbols were not found, BSD nlist returns the number of symbols not found. System V nlist still returns 0.

Examining symbol tables: the nm program

You can display the complete symbol table of an object le or an archive with the program nm. Invoke it simply with
$ nm filename

nm output is frequently used by tools such as shell scripts used during building. This can be a problem, since the format of the printout depends strongly on the object le format. In the following sections well look at the differences between nm output for a.out, COFF and ELF les.

nm display of a.out format

With an a.out le, nm output looks like:
$ nm /usr/lib/libc.a syscall.o: 00000028 T _syscall U cerror sigsuspend.o: 00000030 T _sigsuspend U cerror this is the object file name

The lines with the le name and colon tell you the name of the archive member (in other words, the object le) from which the following symbols come. The other lines contain a value (which may be missing if it is not dened), a type letter, and a symbol name. Thats all there is to a.out symbols. As we will see, a.out handles debugging information separately. On the other hand, this means that the type letters are reasonably easy to remember. Upper case represents global symbols, lower case represents local symbols. Table 21-2 gives an overview:

6 February 2005 00:57

Chapter 21: Object les Table 212: a.out symbol types

365

Type letter A B C D f T U

Meaning symbol table entries (see the -a ag). absolute symbol (not relocatable) bss segment symbol common symbol data segment symbol le name (always local) text segment symbol undened

nm display of COFF format

By contrast,, COFF gives something like this:
$ nm /usr/lib/libc.a Symbols from /lib/libc.a[printf.o]: Name printf.c DGROUP printf _doprnt _iob | | | | | Value Class | file | 0|static| 0|extern| 0|extern| 0|extern| this is the object file name Type Size | | | | | Line Section | | | | | | |.data |.text | |

These columns have the following meaning: Name is the name of the symbol. Value is the value of the symbol. Class is the storage class of the symbol. There are a large number of storage classes, and the System V.3 man pages dont describe them. See Understanding and using COFF, by Gintaras R. Gircys, for a complete list. The one that interests us is extern (externally dened symbols). In conjunction with Class, Type describes the type of symbol more accurately, when it is needed. The symbols were looking at dont need a more accurate description. Size species the size of the entry. This is used in symbolic debug information. Line is line number information, which is also used for symbolic debug information. Section is the section to which the symbol belongs. In our example, the familiar .text and .data occur, but this could be any of the myriad COFF section names.

6 February 2005 00:57

366

nm display of ELF format

The differences between COFF and ELF are less obvious:
$ nm /lib/libc.so.1 Symbols from /lib/libc.so.1: [Index] Value Size Type Bind Other Shndx 0|FILE |LOCL |0 0|SECT |LOCL |0 12|OBJT 4|OBJT 0|FILE 384|FUNC 72|FUNC 128|FUNC 464|FUNC 4|OBJT |LOCL |LOCL |LOCL |LOCL |LOCL |LOCL |LOCL |LOCL |0 |0 |0 |0 |0 |0 |0 |0 |ABS |1 |12 |12 |ABS |8 |8 |8 |8 |15 Name |../../libc.so.1 | |libdirs |rt_dir_list |dlfcns.c |dl_delete |lmExists |appendLm |dl_makelist |dl_tail

[1] | 0| [2] | 148| ... skipping [32] | 208976| [33] | 208972| [34] | 0| [35] | 57456| [36] | 56240| [37] | 56112| [38] | 56320| [39] | 214960|

These columns have the following meanings: Index is simply the index of the symbol in the symbol list. Value is the value of the symbol. Size is the size of the associated object in bytes. Type is the type of the object. This can be NOTY (no type speced), OBJT (a data object), FUNC (executable code), SECT (a section name), or FILE (a le name). Bind species the scope of the symbol. GLOB species that the symbol is global in scope, WEAK species a global symbol with lower precendene, and LOCL species a local symbol. Other is currently unused and contains 0. Shndx may be ABS, specifying an absolute symbol (in other words, not relocatable), COMMON species a bss block, and UNDEF species a reference to an external symbol. A number in this eld is an index into the section table of the section to which the symbol relates. nm doesnt tell you which section this is, so the information is not very useful.

Problems with nm output

As we have seen, the output of nm depends a lot on the object le format. You frequently see shell scripts that use nm to look inside a le and extract informationthey will go seriously wrong if the object le format is not what they expect. If there isnt an alternative script to look at your kind of object le, you will have to modify it yourself.

6 February 2005 00:57

Chapter 21: Object files

367

Debugging information
Symbolic debuggers have a problem: they relate to the object le, but they want to give the impression that they are working with the source le. For example, the program addresses that interest you are source le line numbers, not absolute addresses, and you want to refer to variables by their names, not their addresses. In addition, you expect the debugger to know the types of variables, so that when you ask the debugger to display a variable of type char *, it displays a string, and when you ask it to display a float, you get the correct numeric value. The object le structures we have seen so far dont help too much. Information is available for global symbols, both in the text and data sections, but the type information is not detailed enough to tell the debugger whether a data variable is a char * or a float. The symbol table information contains no information at all about local variables or line numbers. In addition, the symbol table information goes away at link time, so it wouldnt be of much help anyway. For these reasons, separate data structures for debugging information were introduced. In the a.out format, they have nothing to do with the rest of the le. In COFF and ELF, they are more integrated, but debugging information has one thing in common in all object formats: it is at the end of the le so that it can be easily removed when it is no longer wanted debugging information can become very large. Its not uncommon to see debugging information increase the size of an executable by a factor of 5 or 10. In extreme cases, such as in libraries, it can become 50 times as big. Frequently youll see a make all that creates an executable with debugging symbols, and a make install that installs the same executable but removes the debugging symbols. This process is called stripping, and can be done by the program strip or by install with the -s ag. In order to do this, it makes sense for the debugging information to be in its own section at the end of the le, and this is how all object le formats solve the problem. Debugging information is supplied in the assembler source in the form of directives. In Example 21-2, which is from an assembler designed to create a.out relocatables, this job is done by the .stabs, .stabn and .stabd directives. These directives are discussed in more detail in Appendix C, Assembler directives and ags, on page 421. Lets look at the directives in our example: At the beginning of the le there are a lot of .stabs directives dening standard data types, so many that we have omitted most of them. The compiler outputs these directives even if the data type isnt used in the program, and theyre handy to have in case you want to cast to this data type when debugging. Throughout the le you nd individual .stabd 68 directives. These specify that the line number specied in the last parameter starts at this point in the text. At the end of the function _inc, information about the function itself and the variables associated with it appear in further .stabs directives. Finally, information about the block structure of the function appears in the .stabn directive.

6 February 2005 00:57

368

This information is very dependent on the object le format. If you need more information, the best source is the accompanying system documentation.

The linker
You usually encounter the linker as the last pass of the C compiler. As the name ld implies, the linker was called the loader in the Seventh Edition, though all modern systems call it a link editor.* Traditionally, the compiler compiles object les, and then runs the linker to create an executable program. This is logical, since a single executable is always composed of multiple object les, whereas there is a one-to-one relationship between source les and object modules. The most important function performed by the linker is symbol resolution. To understand this, we need to dene a few terms: The symbol list, sometimes called a symbol table, is an internal data structure where the linker stores information about all symbols whose name it has encountered. It contains the same kind of information about the symbol as we saw in struct nlist on page 363. An undened symbol is only partially undened: we know at least its name, but some part of its value is unknown.

Initially, the symbol list is empty, but every le that is included adds to the list. A number of cases can occur: The le refers to an undened symbol. In this case, if the linker has not yet seen this symbol, it enters it into the symbol list and starts a list of references to it. If the symbol is already in the symbol list, the linker adds a reference to the symbols reference list. The le refers to a symbol that has already been dened. In this case, the linker simply performs the required relocation. The le denes a symbol. There are three possibilities here: If the symbol has not been encountered before, it is just added to the symbol list. If the symbol is already marked as undened, the linker updates the symbol information and performs the required relocation for each element in the reference list. If the symbol is known and dened, it is now doubly dened. The linker prints an error message, and will not create an output le.

At the same time as it creates the symbol list, the linker copies data and text sections into the areas it has allocated for them. It copies each individual section to the current end of the area. The symbol list entries reect these addresses.
* Properly, the loader was the part of the operating system that loaded a program into memory prior to execution. Once, long before the advent of UNIX, the two functions were almost synonymous. Even if you supply only a single object le yourself, you need the C startup code in crt0.o and library modules from system libraries such as libc.a.

6 February 2005 00:57

Chapter 21: Object les

369

Function libraries
Many of the functions you use in linking an executable program are located in function libraries, a kind of object le archive built by ar. The linker knows about the format of ar archives and is able to extract the object les from the archive. The resultant executable contains code from the object les specied on the command line and from the object les found in the libraries. The functions in the libraries are just like any others you may include. They run in user context, not kernel context, and are stored in libraries simply for convenience. We can consider three groups: The standard* C library, normally /usr/lib/libc.a. This library contains at least the functions needed to link simple C programs. It may also contain functions not directly connected with the C language, such as network interface functionsBSD does it this way. Additional libraries supporting system functions not directly concerned with the C programming language. Networking functions may also fall into this category System V does it this way. Libraries supporting third party packages, such as the X11 windowing system.

Library search
You can specify object les to the linker in two different ways: you specify that an object le is to be included in the output or that a library le is to be searched by specifying its name on the command line. The library search is one of the most powerful functions performed by the linker. Instead of including the complete library in the output le, the linker checks each object le in the library for denitions of currently undened symbols. If this is the case, it includes the object le, and not the library. This has a number of implications: The linker includes only object les that dene symbols referenced by the program, so the program is a lot smaller than it would be if you included the complete library. We dont want to include anything that isnt required, so each object le usually denes a single function. In some rare cases, it may dene a small number of related functions that always get included together. Each object le may refer to other external symbols, so including one le in an archive may require including another one. If you compile a library with symbols, each single-function object le will contain debugging information for all the external information dened in the header les. This information is usually many times the size of the function text and data. Once the library has been searched, the linker forgets it. This has important consequences which well examine on page 373.

For reasons shrouded in history, you dont specify the path name of the library leinstead
* Note the lower-case use of the word standard. Whether or not the library conforms to the ANSI/ISO C Standard, it is a standard part of a software development system.

6 February 2005 00:57

370

you tell the linker the names of directories that may contain the libraries you are looking for, and a coded representation of the library name. For example, if you want to include /opt/lib/libregex.a in your search path, you would include -L/opt/lib -lregex in your compiler or linker call: -L/opt/lib tells the linker to include /opt/lib in the list of directories to search. -lregex tells the linker to search for the le libregex.a in each of the directories to search.

This can be a problem if you have four les /usr/lib/libfoo.a, /usr/lib/libbar.a, /opt/lib/libfoo.a and /opt/lib/libbar.a, and you want to search only /opt/lib/libfoo.a and /usr/lib/libbar.a. In this case, you can name the libraries explicitly. To keep the pain of linking executables down to tolerable levels, the compiler control program (usually cc) supplies a few library paths and specications for freenormally the equivalent of -L/usr/lib -lc, which at least nds the library /usr/lib/libc.a, and also supplies the path to all other libraries in /usr/lib. You need only specify additional paths and libraries. Occasionally this behaviour is undesirable: what if you deliberately want to exclude the standard libraries, like if youre building an executable for a different version of the operating system? Some compilers give you an option to forget these libraries. For example, in gcc it is -nostdlib. Like most aspects of UNIX, there is no complete agreement on where to store library les, but most systems come close to the following arrangement: /usr/lib contains the basic system libraries as well as startup code like crt0.o and friends, which are bound in to supply low-level support for the C language. Well look at this in the next section. Some of these les used to be stored in /lib. Nowadays /lib tends either not to be present or, for compatibilitys sake, it is a symlink to /usr/lib. System V.4 systems place BSD compatibility libraries in /usr/ucblib*. Many of these functions duplicate functions in /usr/lib. /usr/X11/lib, /usr/X/lib, /usr/lib/X11, /usr/lib/X11R6 and others are some of the places that the X11 libraries might be hidden. This directory probably contains all the parts of X11 and related code that are on your system.

Shared libraries
Some libraries can be very big. The X11R6 version libX11.a, the standard X11 functions, runs to 630 kB on BSD/OS. The Motif library libXm.a is nearly 1.4 MB in size. This can lead to enormous executables, even if the program itself is relatively smallthe 500 kB Hello world syndrome. Since these functions are used in many programs, many copies of a function may be active at any one time in the system. For example, just about every program
* UCB stands for the University of California at Berkeley, the home of the Berkeley Software Distributions. Youll frequently nd BSD-derived software stored in directories whose names start with the letters ucb.

6 February 2005 00:57

Chapter 21: Object les

371

uses the function printf, which with its auxiliary functions can be quite big. To combat this, modern UNIX avours support shared libraries: the library itself is no smaller, but it is in memory only once. Two different library schemes are in current use: static shared libraries* and dynamic shared libraries. Static shared libraries contain code which has been linked to run at a specic address, which means that you could have difculties if your program refers to two libraries with overlapping address ranges, or if you use a different version of the library with functions at slightly different addresses. Dynamic libraries get round this problem by linking at run time, which requires a dynamic linker. Unless youre building shared libraries, a topic beyond the scope of this book, you dont need to worry about the difference between the two. If you do nd yourself in the situation where you need to build shared libraries, your best source of information is your operating system documentation. A shared library performs two different functions: When you link your program, it supplies information about the locations of the functions and data in the library. Some systems, such as SunOS 4, supply a stub le with a name like libc.sa.1.9. Since it does not contain the code of the functions, it is relatively small on SunOS 4.1.3, it is 7996 bytes long. Other systems, such as System V.4, only supply a single library le with a name like libc.so. The linker only includes enough information for the dynamic loader to locate the functions in the library le at run time. At run time, it supplies the functions and data. On all systems, the le name is of the form libc.so.1.9

Its important to ensure that you use the same library to perform these two actions. If a function or a data structure changes between versions of the library, a program written for a different version may work badly or crash. This is a common problem: most programs are distributed in executable form, and thus contain preconceived notions about what the library looks like. Since were linking the program ourselves, we should not run in to this problem. If you do run into problems, you can always fall back to static (unshared) libraries.

Other linker input

In addition to the user-specied object les and libraries, the C programming language requires a few auxiliary routines to set up its run-time environment. These are stored in one or more auxiliary object les in a place known to the compiler, usually /usr/lib. For example, in the System V.4 example above, we can see how the compiler control program starts the linker if we use the -v ag to the compiler:
$ gcc -v -u baz -o foo foo.o -L. -lbaz -lbar /usr/ccs/bin/ld -V -Y P,/usr/ccs/lib:/usr/lib -Qy -o foo -u baz /usr/ccs/lib/crt1.o /usr/ccs/lib/crti.o /usr/ccs/lib/values-Xa.o /opt/lib/gcc-lib/i386-unknown-sysv4.2/2.5.8/crtbegin.o -L. -L/opt/lib/gcc-lib/i386-unknown-sysv4.2/2.5.8 -L/usr/ccs/bin -L/usr/ccs/lib -L/opt/lib foo.o -lbaz -lbar -lgcc -lc

* Dont confuse static shared libraries with the term static libraries, which are traditional, non-shared libraries.

6 February 2005 00:57

372
/opt/lib/gcc-lib/i386-unknown-sysv4.2/2.5.8/crtend.o /usr/ccs/lib/crtn.o -lgcc

The same example in BSD/OS species the les /usr/lib/crt0.o, foo.o, -lbar, -lbaz, -lgcc, -lc and -lgcconly fractionally more readable. This example should make it clear why almost nobody starts the linker directly.

Merging relocatable files

Occasionally you want to merge a number of object les into one large le. Weve seen one way of doing that: create an object le library with ar. You can also use the linker to create an object le. Which you choose depends on why you want to make the le. If you are creating a function library, use ar. As we have seen, the linker includes individual object les from the archive. It also happily includes a relocatable object created by a previous invocation of the linker, but in this case it includes the complete object, even if you dont need all the functions. You dont often need to create relocatable objects with the linker: the only real advantage over a library is that the resultant object is smaller and links faster. If you want to do it, you specify a ag, normally -r. For example,
$ ld -r foo.o bar.o baz.o -o foobarbaz.o

This links the three object les foo.o, bar.o and baz.o and creates a new object le foobarbaz.o that contains all the functions and data in the three input les.

Problems with the link editor

Once you have compiled all your objects, youre still not home. There are plenty of things that can go wrong with the linkage step. In this section well look at some of the more common problems.

Invalid linker flags

You normally invoke the linker via the compiler rather than calling it directly. This is a good idea, as we saw in the previous section. If you have a Makele with an explicit linker call, and you run into trouble with linker ags, and the system documentation doesnt help, consider replacing the linker invocation with a compiler invocation.

Invalid object files

Occasionally, the package you are building may already contain object les. Its unlikely that you can to use them, but make is far too simplistic to notice the difference, and the result is usually some kind of message from the linker saying that it cant gure out what kind of le this is. If youre in doubt, use the le command:
$ file *.o gram.o: 386 executable not stripped main.o: ELF 32-bit LSB relocatable 80386 Version 1 scan.o: sparc executable not stripped

6 February 2005 00:57

Chapter 21: Object les

util.o: 80386 COFF executable not stripped - version 30821

373

Here are four different kinds of object les in the same directory. Occasionally, you will see les like this that are there for a good reason: due to license reasons, there are no corresponding sources, and there will be one object for each architecture that the package supports. In this example, however, the le names are different enough that you can be reasonably sure that these les are junk left behind from previous builds. If the object les are still there after a make clean, you should remove them manually (and x the Makele).

Suboptimal link order

We have seen that the linker takes all objects it nds and puts their code and data into the code and data segments in the order in which they appear. From the point of view of logic ow, this works ne, but it can have signicant performance implications on modern machines. You might nd that 95% of the execution time of a program is taken up by 5% of the code. If this code is located contiguously, it will probably t into the cache of any modern machine. If, on the other hand, it is scattered throughout memory, it will require much more cache, possibly more than the machine can supply. This can result in a dramatic drop in performance. Most linkers do not help you much in arranging functions. The simplest way is to put one function in a le, like you do in an archive, and specify them in sequence in the linker invocation. For example, if you have ve functions foo, bar, baz, zot, and glarp, and you have determined that you need three functions next to each other in the sequence foo, glarp and zot, you can invoke the linker with:
$ cc -o foobar foo.o glarp.o zot.o bar.o baz.o

Missing functions
The UNIX library mechanism works well and is reasonably standardized from one platform to the next. The main problem you are likely to encounter is that the linker cant nd a function that the program references. There can be a number of reasons for this: The symbol may not be a function name at all, but a reference to an undened preprocessor variable. For example, in xfm version 1.1, the source le FmInfo.c contains:
if (S_ISDIR(mode)) type = "Directory"; else if (S_ISCHR(mode)) type = "Character special file"; else if(S_ISBLK(mode)) type = "Block special file"; else if(S_ISREG(mode)) type = "Ordinary file"; else if(S_ISSOCK(mode)) type = "Socket"; else if(S_ISFIFO(mode)) type = "Pipe or FIFO special file";

sys/stat.h denes the macros of the form S_ISfoo. They test the le mode bits for specic le types. System V does not dene S_ISSOCK (the kernel doesnt have sockets), so

6 February 2005 00:57

374

a pre-ANSI compiler assumes that S_ISSOCK is a reference to an external function. The module compiles correctly, but the linker fails with an undened reference to S_ISSOCK. The obvious solution here is conditional compilation, since S_ISSOCK is a preprocessor macro, and you can test for it directly with #ifdef:
type = "Ordinary file"; #ifdef S_ISSOCK else if(S_ISSOCK(mode)) type = "Socket"; #endif else if(S_ISFIFO(mode))

The function is in a different library, and you need to specify it to the linker. A good example is the networking code we mentioned on page 369: a reference to socket will link just ne with no additional libraries on a BSD platform, but on some versions of System V.3 you will need to specify -linet, and on System V.4 and other versions of System V.3 you will need to specify -lsocket. The ndf script can help here. It uses nm to output symbol information from the les specied in LIBS, and searches the output for a function denition whose name matches the parameter supplied. The search parameter is a regular expression, so you can search for a number of functions at once. For example, to search for strcasecmp and strncasecmp, you might enter:
$ findf str.*casecmp /usr/lib/libc.a(strcasecmp.o): _strcasecmp /usr/lib/libc.a(strcasecmp.o)): _strncasecmp /usr/lib/libc_p.a(strcasecmp.po): _strcasecmp /usr/lib/libc_p.a(strcasecmp.po)): _strncasecmp

Because of the differences in nm output format, ndf looks very different on BSD systems and on System V. You may nd that you need to modify the script to work on your system. Example 21-3 shows a version for 4.4BSD:
Example 213:
LIBS="/usr/lib/lib* /usr/X11R6/lib/lib*" nm $LIBS 2>/dev/null \ | awk -v fun=$1 \ /\// {file = $1}; /[\/].*:/{member = $1}; $3 fun && $2 /T/ { sub (":$", "", file); ; sub (":$", "):", member); print file "(" member "\t" $3}

On a system like System V.4, which uses ELF format, the corresponding script is in Example 21-4:
Example 214:
LIBS="/usr/lib/lib* /usr/X11R6/lib/lib*" nm $LIBS 2>/dev/null \ | sed s:|: :g \ | gawk -v fun=$1 \ /Symbols from/ {file = $3}; $8 fun && $4 /FUNC/ { print file member "\t" $8 }

6 February 2005 00:57

Chapter 21: Object les Example 214: (continued)

375

Some versions of System V awk have difculty with this script, which is why this version uses GNU awk. The function is written in a different language, and the internal name differs from what the compiler expected. This commonly occurs when you try to call a C function from C++ and forget to tell the C++ compiler that the called function is written in C. We discussed this in Chapter 17, Header les, page 285. The function is part of the package, but has not been compiled because a conguration parameter is set incorrectly. For example, xpm, a pixmap conversion program, uses strcasecmp. Knowing that it is not available on all platforms, the author included the function in the package, but it gets compiled only if the Makele contains the compiler ag -DNEED_STRCASECMP. The function is supplied in a library within the package, but the Makele is in error and tries to reference the library before it has built it. You wouldnt expect this ever to happen, since it is the purpose of Makeles to avoid this kind of problem, but it happens often enough to be annoying. Its also not always immediately obvious that this is the cause if you suspect that this is the reason, but are not sure, the best thing is to try to build all libraries rst and see if that helps. The function is really not supplied in your system libraries. In this case, you will need to nd an alternative. We looked at this problem in detail in Chapter 18, Function libraries. The rst reference to a symbol comes after the linker has searched the library in which it is located.

Lets look at the last problem in more detail: when the linker nishes searching a library, it continues with the following le specications. It is possible that another le later in the list will refer to an object le contained in the library which was not included in the executable. In this case, the symbol will not be found. Consider the following three les: foo.c
main () { bar ("Hello"); }

bar.c
void bar (char *c) { baz (c); }

baz.c
void baz (char *c) { puts (c); }

6 February 2005 00:57

376

We compile them to the corresponding object les, and then make libraries libbar.a and libbaz.a, which contain just the single object le bar.o and baz.o respectively. Then we try to link:
$ gcc -c foo.c $ gcc -c bar.c $ gcc -c baz.c $ ar r libbaz.a baz.o $ ar r libbar.a bar.o $ gcc -o foo foo.o -L. -lbaz -lbar Undefined first referenced symbol in file baz ./libbar.a(bar.o) ld: foo: fatal error: Symbol referencing errors. No output written to foo $ gcc -o foo foo.o -L. -lbar -lbaz $

In the rst link attempt, the linker included foo.o, then searched libbaz.a and didnt nd anything of interest. Then it went on to libbar.a and found it needed the symbol baz, but by that time it was too late. You can solve the problem by putting the reference -lbar before -lbaz. This problem is not even as simple as it seems: although its bad practice, you sometimes nd that libraries contain mutual references. If libbar.a also contained an object le zot.o, and baz referred to it, you would have to link with:
$ gcc -o foo foo.o -L. -lbar -lbaz -lbar

An alternative seems even more of a kludge: with the -u ag, the linker will enter an undened symbol in its symbol table. In this example, we could also have written
$ gcc -u baz -o foo foo.o -L. -lbaz -lbar $

Dumping to object files

Some programs need to perform signicant processing during initialization. For example, emacs macros are written in emacs LISP, and they take some time to load. Startup would be faster if they were already in memory when the program is started. The only normal way to have them in memory is to compile them in, and its very difcult to initialize data at compile time as intricately as a program like emacs does it at run time. The solution chosen is simple in concept: emacs does it once at run time. Then it dumps itself to disk in object le format: it copies the text section directly from its own text area, since there is no way it can be changed, and it writes the data section from its current data area, including all of what used to be bss. It doesnt need to copy the stack section, since it will be recreated on initialization. This rather daring approach works surprisingly well as long as emacs knows its own object le format. UNIX doesnt provide any way to nd out, since there is normally no reason why a program should know its own object le format. The result can be problems when porting a package like this to a system with a different object format: the port runs ne until the rst

6 February 2005 00:57

Chapter 21: Object les

377

executable dumps, but the dumped executable does not have a format that the kernel can recognize. Other programs that use this technique are gcl (GNU common LISP) and TEX.

Process initialization and stack frames

In Chapter 12, Kernel dependencies, page 168, we examined the myriad avours of exec. They all pass arguments and environment information to the newly loaded program. From a C program viewpoint, the arguments are passed as a parameter to main, and the environment is just there for the picking. In this section well look more closely at what goes on between exec and main. In order to understand this, we need to look more closely at parameter passing.

Stack frames
Most modern machines have a stack-oriented architecture, even if the support is rather rudimentary in some cases. Everybody knows what a stack is, but here well use a more restrictive denition: a stack is a linear list of storage elements, each relating to a particular function invocation. These are called stack frames. Each stack frame contains The parameters with which the function was invoked. The address to which to return when the function is complete. Saved register contents. Variables local to the function. The address of the previous stack frame.

With the exception of the return address, any of these elds may be omitted.* Typical stack implementations supply two hardware registers to address the stack: The stack pointer points to the last used word of the stack. The frame pointer points to somewhere in the middle of the stack frame.

The resultant memory image looks like:

* Debuggers recognize stack frames by the frame pointer. If you dont save the frame pointer, it will still be pointing to the previous frame, so the debugger will report that you are in the previous function. This frequently happens in system call linkage functions, which typically do not save a stack linkage, or on the very rst instruction of a function, before the linkage has been built. In addition, some optimizers remove the stack frame.

6 February 2005 00:57

378

Function arguments Return address Old value of frame pointer Automatic variables Temporary storage Function arguments Return address Old value of frame pointer Automatic variables Stack pointer Temporary storage Stack frame 0 Stack frame 1

Frame pointer

Figure 211. Function stack frame

The individual parts of the stack frames are built at various times. In the following sections, well see how the stack gets set up and freed.

Setting up the initial parameters

exec builds the initial stack. The exact details are implementation-dependent, but most come close to the way BSD/OS does it, so well look at that. The stack is always allocated at a xed point in memory, 0xefbfe000 in the case of BSD/OS, and grows downwards, like the stacks on almost every modern architecture. At the very top of stack, is structure with information for the ps program:
struct ps_strings { char **ps_argv; int ps_argc; char **ps_envp; int ps_nenv; };

/* /* /* /*

first of 0 the number first of 0 the number

or of or of

more argument pointers */ argument pointers */ more environment pointers */ environment pointers */

This structure is supplied for convenience and is not strictly necessary. Many systems, for example FreeBSD, do not dene it. Next, exec places on the stack all environment variable strings, followed by all the program arguments. Some systems severely limit the maximum size of these stringswe looked at the problems that that can cause in Chapter 5, Building the package, page 74. After the variable strings come two sets of NULL-terminated pointers, the rst to the environment variables, the second to the program arguments.

6 February 2005 00:57

Chapter 21: Object les

379

Finally comes the number of arguments to main, the well-known parameter argc. At this point, the stack looks like:
ps information Environment variables

Program arguments NULL Environment pointers NULL more argument pointers argv [1] argv [0] Stack pointer Figure 212. Stack frame at start of program argc

At this point, all the data for main is on the stack, but its not quite in the form that main needs. In addition, theres no return address. But where could main return to? All this work has been done in the kernel by exec: we need to return to a place in the program. These problems are solved by the function start, the real beginning of the program: it calls main, and then calls exit with the return value from main. Before doing so, it may perform some runtime initializations. A minimal start function looks like this stripped down version of GNU libc start.c, which is the unlikely name of the source le for crt0.o:
static void start (int argc, char *argp) { char **argv = &argp; /* set up a pointer to the first argument pointer */ __environ = &argv [argc + 1]; /* The environment starts just after argv */ asm ("call L_init"); /* call the .init section */ __libc_init (argc, argv, __environ); /* Do C and C++ library initializations */ exit (main (argc, argv, __environ)); /* Call the user program */ }

The asm directive is used for C++ initializationwell look at that on page 380. But whats this? start calls main with three parameters! The third is the address of the environment variable pointers. This is one of the best kept secrets in UNIX: main really has three parameters:

6 February 2005 00:57

380
int main (int argc, char *argv [], char *envp []);

It isnt documented anywhere, but its been there at least since the Seventh Edition and its unlikely to go away, since there isnt really any other good place to store the environment variables. By the time we have saved the stack linkage in main, the top of the stack looks like:

*argv [0]
argc dummy return dummy frame pointer

start stack frame main stack frame

**environ **argv
argc Frame pointer Stack pointer return to start saved frame pointer

Figure 213. Stack frame after entering main

Initializing C++ programs

What weve seen here is not enough for C++: before entering main, the program may need to initialize global class instances. Since this is system library code, it cant know what global classes exist. The solution depends on the system: System V systems place this information in a special section, .init. The initialization le crtn.o contains a default .init section containing a single return instruction. If a C++ program has global initializers, it will create an .init section to initialize them. If any object module before ctrn.o has an .init section, it will be included before the .init section in ctrn.o. During program initialization, the function start calls the .init section to execute the global constructorsthis is the purpose of the asm directive on page 379. Systems based on a.out formats do not have this luxury. Instead, they compile special code into main to call the appropriate constructors.

The difference between these two approaches can be important if you are debugging a C++ program which dies in the global constructors.

6 February 2005 00:57

Chapter 21: Object files

381

Stack growth during function calls

Now that we have an initial stack, lets see how it grows and shrinks during a function call. Well consider the following simple C program compiled on the i386 architecture:
foo (int a, int b) { int c = a * b; int d = a / b; printf ("%d %d0, c, d); } main (int { int x = int y = foo (y, } argc, char *argv []) 4; 5; x);

The assembler code for the calling sequence for foo in main is:
pushl -4(%ebp) pushl -8(%ebp) call _foo addl $8,%esp value of x value of y call the function and remove parameters

Register ebp is the base pointer, which we call the frame pointer. esp is the stack pointer. The push instructions decrement the stack pointer and then place the word values of x and y at the location to which the stack pointer now points. The call instruction pushes the contents of the current instruction pointer (the address of the instruction following the call instruction) onto the stack, thus saving the return address, and loads the instruction pointer with the address of the function. We now have:

argc return to start Frame pointer saved frame pointer local var x local var y parameter a parameter b Stack pointer return to main

main stack frame foo stack frame

Figure 214. Stack frame after call instruction

The called function foo saves the frame pointer (in this architecture, the register is called ebp, for extended base pointer), and loads it with the current value of the stack pointer register esp.

6 February 2005 00:57

382
_foo: pushl %ebp movl %esp,%ebp save ebp on stack and load with current value of esp

At this point, the stack linkage is complete, and this is where most debuggers normally set a breakpoint when you request on to be placed at the entry to a function. Next, foo creates local storage for c and d. They are each 4 bytes long, so it subtracts 8 from the esp register to make space for them. Finally, it saves the register ebxthe compiler has decided that it will need this register in this function.
subl $8,%esp pushl %ebx create two words on stack and save ebx register

At this point, our stack is now complete and looks like the diagram on page 377:

saved frame pointer local var x local var y parameter a parameter b return to main Frame pointer saved frame pointer local var c local var d Stack pointer saved ebx contents

main stack frame foo stack frame

Figure 215. Complete stack frame after entering called function

The frame pointer isnt absolutely necessary: you can get by without it and refer to the stack pointer instead. The problem is that during the execution of the function, the compiler may save further temporary information on the stack, so its difcult to keep track of the value of the stack pointerthats why most architectures use a frame pointer, which does stay constant during the execution of the function. Some optimizers, including newer versions of gcc, give you the option of compiling without a stack frame. This makes debugging almost impossible. On return from the function, the sequence is reversed:
movl -12(%ebp),%ebx leave ret and restore register ebx reload ebp and esp and return

The rst instruction reloads the saved register ebx, which could be stored anywhere in the stack. This instruction does not modify the stack. The leave instruction loads the stack pointer esp from the frame pointer ebp, which effectively discards the part stack below the saved ebp value. Then it loads ebp with the contents of the

6 February 2005 00:57

Chapter 21: Object les

383

word to which it points, the saved ebp, effectively reversing the stack linkage. The stack now looks like it did on entry. Next, the ret instruction pops the return address into the instruction pointer, causing the next instruction to be fetched from the address following the call instruction in the calling function. The function parameters x and y are still on the stack, so the next instruction in the calling function removes them by adding to the stack pointer:
addl $8,%esp and remove parameters

Object Archive formats

As we have seen, object les are frequently collected into libraries, which from our current point of view are archives maintained by ar. One of the biggest problems with ar archives is that there are so many different forms. Youre likely to come across the following ones: The so-called common archive format. This format starts with the magic string !<arch>\n. It is used both in System V.4 and in BSD versions since 4BSD. The PORT5AR format, which starts with the magic string <ar>. System V.3 denes it, but doesnt use it. The Seventh Edition archive, which starts with the magic number 0177545. It is also used by XENIX and System V.2 systems. In System V.3 systems, le reports this format as x.out randomized archive. 4.4BSD le refers to it as old PDP-11 archive, while many System V.4 les dont recognize it at all. UnixWare is one that does, and it calls it a pdp11/pre System V ar archive.

As long as you stick to modern systems, the only archive type youre likely to come across is the common archive format. If you do nd one of the others, you should remember that its not the archive that interests you, its the contents. If you have another way to get the contents, like the original object or source les, you dont need to worry about the archive.

6 February 2005 00:57

Comparative reference to UNIX data types

Table A-1 lists a number of typedefs that are dened in System V.4 and 4.4BSD. The list is not intended to be exhaustive, but it might be of assistance if you run into trouble with a new typedef. Youll note a couple of things about the table: There is a signicant deviation even between 4.4BSD and System V.4, although both are systems that have gone to some trouble to be portable. 4.4BSD uses a construct long long to describe 64 bit integers. This is not ofcially part of the C language, but is supported by the GNU C compiler, which is the standard compiler for 4.4BSD. The System V.4 compilers do not support long long, so they have to dene quad as a structure containing two longs.

Table A1: system type denitions

Type
addr_t (SVR4) amq_mount_tree (4.4BSD) amq_mount_tree_p (4.4BSD) ansistat_t (SVR4) audio_info_t (4.4BSD) auto_tree (4.4BSD) bitstr_t (4.4BSD) bool_t (4.4BSD) boolean_t (4.4BSD) boolean_t (SVR4) caddr_t (4.4BSD) cat_t (4.4BSD) cc_t cc_t1 charstat_t (SVR4) chr_merge_t (SVR4)

Denition
char * struct amq_mount_tree amq_mount_tree * struct ansi_state struct audio_info struct auto_tree unsigned char int int enum boolean char * unsigned char unsigned char char struct char_stat struct chr_merge

Description core address type

truth value

core address

385

5 February 2005 02:09

386 Table A1: system type denitions (continued)

Type
clock_t (4.4BSD) clock_t (SVR4) cnt_t (SVR4) comp_t (4.4BSD) comp_t (SVR4) create_t (SVR4) daddr_t dblk_t (SVR4) dev_t dirent_t (SVR4) dmask_t (SVR4) dpt_dma_t (SVR4) dpt_sblk_t (SVR4) emask_t (SVR4) emcp_t (SVR4) emip_t (SVR4) emop_t (SVR4) emp_t (SVR4) entryno_t (SVR4) ether_addr_t (SVR4) eucioc_t (SVR4) faddr_t (SVR4) fhandle_t (4.4BSD) fhandle_t (SVR4) fixpt_t (4.4BSD) fpos_t (4.4BSD) fpos_t (SVR4) frtn_t (SVR4) fsid_t (4.4BSD) gdp_t (SVR4) gdpmisc_t (SVR4) gid_t (4.4BSD) gid_t (SVR4) gid_t2 greg_t (SVR4) gregset_t (SVR4) hostid_t (SVR4) id_t (SVR4) idata_t (SVR4)

Denition
unsigned long_ long short u_short ushort enum create long struct datab unsigned long struct dirent unsigned short struct ScatterGather struct dpt_srb unsigned short unsigned char * struct emind * struct emout * struct emtab * int u_char [] struct eucioc char * struct fhandle struct svcfh unsigned long off_t long struct free_rtn struct { long val[2]; } struct gdp struct gdpmisc unsigned long uid_t unsigned short int greg_t [] long long struct idata

Description

count type

disk address device number

far address (XENIX Compatibility)

xed point number

le system id type

group id GID type group ID

process id, group id, etc.

5 February 2005 02:09

Appendix A: UNIX data types Table A1: system type denitions (continued)

387

Type
index_t (SVR4) indx_t (4.4BSD) ino_t inode_t (SVR4) instr_t (SVR4) int8_t (4.4BSD) int16_t (4.4BSD) int32_t (4.4BSD) int64_t (4.4BSD) ioctl_t (4.4BSD) jdata_t (SVR4) k_fltset_t (SVR4) k_sigset_t (SVR4) key_t (4.4BSD) key_t (SVR4) klm_testargs (SVR4) klm_testrply (SVR4) kmabuf_t (SVR4) kmasym_t (SVR4) kvm_t (4.4BSD) label_t (SVR4) level_t (SVR4) lid_t (SVR4) lock_data_t (4.4BSD) lock_t (SVR4) major_t (SVR4) mblk_t (SVR4) minor_t (SVR4) mode_t (4.4BSD) mode_t (SVR4) n_time nfsv2fh_t (4.4BSD) nl_catd (SVR4) nlink_t (4.4BSD) nlink_t (SVR4)

Denition
short u_int16_t unsigned long struct inode char char short int long long void * struct jdata unsigned long unsigned long long int struct klm_testargs struct klm_testrply struct kmabuf struct kmasym struct __kvm struct {int val [6];} lid_t unsigned long struct lock short unsigned long struct msgb unsigned long unsigned short unsigned long u_long union nfsv2fh nl_catd_t * unsigned short unsigned long

Description index into bitmaps inode number

8 bit signed integer 16 bit integer 32 bit integer 64 bit integer Third arg of ioctl kernel fault set type kernel signal set type IPC key type IPC key type

setjmp/longjmp save area users view of security level internal representation of security level lock work for busy wait major part of device number minor part of device number permissions le attribute type ms since 00:00 GMT

link count le link type

5 February 2005 02:09

388 Table A1: system type denitions (continued)

Type
nmcp_t (SVR4) nmp_t (SVR4) nmsp_t (SVR4) o_dev_t (SVR4) o_gid_t (SVR4) o_ino_t (SVR4) o_mode_t (SVR4) o_nlink_t (SVR4) o_pid_t (SVR4) o_uid_t (SVR4) off_t (4.4BSD) off_t (SVR4) paddr_t (4.4BSD) paddr_t (SVR4) pgno_t (4.4BSD) pid_t (4.4BSD) pid_t (SVR4) priv_t (SVR4) ptr_t (4.4BSD) ptrdiff_t pvec_t (SVR4) qaddr_t (4.4BSD) qband_t (SVR4) qshift_t (4.4BSD) quad_t (4.4BSD) queue_t (SVR4) recno_t (4.4BSD) regoff_t (4.4BSD) rf_token_t (SVR4) rlim_t (SVR4) rm_t (SVR4) rune_t (4.4BSD) rval_t (SVR4) s_token (SVR4) scrnmap_t (SVR4) scrnmapp_t (SVR4) segsz_t (4.4BSD) sel_t (SVR4)

Denition
unsigned char * struct nmtab * struct nmseq * short o_uid_t unsigned short unsigned short short short unsigned short quad_t long long unsigned long u_int32_t long long unsigned long void * int unsigned long quad_t * struct qband u_quad_t long long struct queue u_int32_t off_t struct rf_token unsigned long enum rm int union rval u_long unsigned char [] unsigned char * long unsigned short

Description

old device type old GID type old inode type old le attribute type old le link type old process id type old UID type le offset type le offset type physical address type physical address type process id process id type pointer type Difference between two pointers kernel privilege vector

"rune" type: extended character

Screen map type Pointer to screen map type segment size selector type

5 February 2005 02:09

Appendix A: UNIX data types Table A1: system type denitions (continued)

389

Type
sema_t (SVR4) sig_atomic_t sig_t (4.4BSD) sigset_t (4.4BSD) size_t (4.4BSD) size_t (SVR4) speed_t (4.4BSD) speed_t (SVR4) spl_t (SVR4) srqtab_t (SVR4) ssize_t (4.4BSD) stack_t (SVR4) stridx_t (SVR4) strmap_t (SVR4) sv_t (SVR4) swblk_t (4.4BSD) symfollow_t (SVR4) sysid_t (SVR4) tcflag_t tcl_addr_t (SVR4) tcl_data_t (SVR4) tcl_endpt_t (SVR4) tco_addr_t (SVR4) tco_endpt_t (SVR4) tcoo_addr_t (SVR4) tcoo_endpt_t (SVR4) time_t tpproto_t (SVR4) tpr_t (4.4BSD) ttychar_t (4.4BSD) u_char u_int u_int16_t (4.4BSD) u_int32_t (4.4BSD) u_int64_t (4.4BSD) u_int8_t (4.4BSD) u_long u_quad_t (4.4BSD) u_short

Denition
int int void (*) (int) unsigned int int unsigned long unsigned long int unsigned char [] int struct sigaltstack ushort [] unchar [] char long enum symfollow short unsigned long struct tcl_addr union tcl_data struct tcl_endpt struct tco_addr struct tco_endpt struct tcoo_addr struct tcoo_endpt long struct tpproto struct session * unsigned char [] [] unsigned char unsigned int unsigned short unsigned int unsigned long long unsigned char unsigned long unsigned long long unsigned short

Description

return type of signal function

to return byte count or indicate error String map index type String map table type swap offset system id

time of day in seconds

16 bit unsigned int 32 bit unsigned int 64 bit unsigned int 64 bit unsigned int Abbreviation quads Abbreviation

5 February 2005 02:09

390 Table A1: system type denitions (continued)

Type
uchar_t (SVR4) uid_t (4.4BSD) uid_t (SVR4) uid_t2 uinfo_t (SVR4) uint uint_t (SVR4) uio_rw_t (SVR4) uio_seg_t (SVR4) ulong (SVR4) ulong_t (SVR4) unchar (SVR4) use_t (SVR4) ushort ushort_t (SVR4) vcexcl_t (SVR4) vfs_namemap_t (4.4BSD) vifbitmap_t (4.4BSD) vifi_t (4.4BSD) vpix_page_t (SVR4) wchar_t (4.4BSD) wchar_t (SVR4) whymountroot_t (SVR4) xdrproc_t
1 2

Denition
unsigned char unsigned long long unsigned short struct master * unsigned int unsigned int enum uio_rw enum uio_seg unsigned long unsigned long unsigned char unsigned char unsigned short unsigned short enum vcexcl struct vfs_namemap u_long u_short struct vpix_page int long enum whymountroot bool_t (*)()

Description User ID User ID User ID Abbreviation

Abbreviation Abbreviation use count for swap Abbreviation

type of a vif index Wide character type Wide character type

Only 4.4BSD telnet Only 4.4BSD Kerberos

5 February 2005 02:09

There is little standardization in the choice of compiler options from one compiler to another, though a couple (-o and -c, for example) are the same across all platforms. Even the -o option differs slightly in meaning from one system to another, however. If there is any reliable documentation, its what was supplied with your compiler. This doesnt help you, of course, if you have a Makele from some unfamiliar machine and youre trying to gure out what the options there mean. This table should ll that gap: if you nd an option, chances are this table will help you guess what it means. If youre looking for a way to tell your compiler what to do, read its documentation. This appendix provides the following comparative references between the GNU, SGI IRIX, SCO UNIX, Solaris, SunOS, System V.3, System V.4 and XENIX versions of the C compiler control program and the C preprocessor. Flags used by the C compiler control program (cc or gcc), starting after this section. gcc-specic options specifying dialect, starting on page 405. gcc-specic debugging options, starting on page 406. gcc-specic warning options, starting on page 407. We discuss the more important warnings in , starting on page . Flags used by the C preprocessor cpp, starting on page 410.

C compiler options
-a
Generate extra code to write prole information for tcov.

(gcc, SunOS) (some MS-DOS compilers) (SVR3)

Linker output should be an absolute le (i.e. the opposite of the -r option).

-a align
Align in structs to align boundary.

-A -A
processor conditional #if #question(answer).

(gcc, SVR4) -Aquestion(answer) asserts that the answer to question is answer. This can used with the pre-

391

5 February 2005 04:34

392

-A-

(gcc, SVR4)
Disable standard assertions. In addition, SVR4 cc undenes all standard macros except those beginning with __.

-acpp
Use alternative cpp based on GNU cpp.

(SGI) (SunOS) (gcc, SGI) (SGI) (gcc)

-align block
Force the global bss symbol block to be aligned to the beginning of a page.

-ansi
Enforce strict ANSI compatibility.

-ansiposix
Enforce strict ANSI compatibility and dene _POSIX_SOURCE.

-B

Specify the path of the directory from which the compiler control program gcc should start the individual passes of the compiler.

-B dynamic

(SunOS, SVR4)

Dynamic linking: tell the linker to search for library les named libfoo.so and then libfoo.a when passed the option -lfoo.

-B static -b target
Cross-compile for machine target.

(SunOS, SVR4) (gcc) (gcc, SGI ANSI C, SCO UNIX, SunOS, SVR4)

Static linking: tell the linker to just search for libfoo.a when passed the option -lfoo.

-C -c

Tell the preprocessor not to discard comments. Used with the -E option.

(all)
Stop compiler after producing the object le, do not link.

-call_shared

(older MIPS)

Produce an executable that uses sharable objects (default). On more modern SGI machines, this is called -KPIC.

-cckr(SGI)
Dene K&R-style preprocessor variables.

-common

(SGI)

Cause multiple denitions of global data to be considered to be one denition, and do not produce error messages.

-compat

(SCO UNIX)

Create an exectauble which is binary compatible across a number of Intel-based systems. Use XENIX libraries to link.

-cord

(SGI)

5 February 2005 04:34

Chapter 0: Compiler options Rearrange functions in the object le to reduce cache conicts.

393

-CSON
Enable common subexpression optimization. Used in conjunction with the -Ooption.

(SCO UNIX) (SCO UNIX) (all)

-CSOFF -D

Disable common subexpression optimization. Used in conjunction with the -Ooption. Dene a preprocessor macro. The form -Dfoo denes foo, but does not give it a value. This can be tested with #ifdef and friends. The form -Dfoo=3 denes foo to have the value 3.

This can be tested with #if. -d

Report the compiler passes and arguments as they are executed.

(XENIX, SCO UNIX) (gcc)

-d when

Make dumps during compilation for debugging the compiler. when species when the dump should be made. Most of these should not be needed by normal users, however the forms -dD (leave all macro denitions in the preprocessor output), -dM (dump only the macro denitions in effect at the end of preprocessing) and -dN (like -dD except that only the macro names are output) can be used with the -E option in order to debug preprocessor macros.

-dalign
Generate double load/store instructions for better performance

(SunOS on Sun-4 systems) (gcc)

-dD

Special case of the -d option: leave all macro denitions in the preprocessor output. The resulting output will probably not compile, but its useful for debugging preprocessor macros.

-dl
Dont generate line number information for the symbolic debugger.

(SVR3) (gcc)
Special case of the -d option: dump only the macro denitions in effect at the end of preprocessing.

-dM

-dn
Dont use dynamic linking. This cannot be used with the -G option.

(SVR4) (SGI) (SCO UNIX, XENIX) (SunOS) (SVR3)

Dont generate symbol attribute information for the symbolic debugger. This option and -dl can be combined as -dsl. Together they are the opposite of the -g option.

-dollar
Allow the symbol $ in C identiers.

-dos
Create an executable for MS-DOS systems.

-dryrun
Display the commands that the compiler would execute, but do not execute them

-ds

5 February 2005 04:34

394

-dy
Use dynamic linking where possible. This is the default.

(SVR4) (all)
Write preprocessor output to standard output, then stop. Some compilers interpret the -o option and write the output there instead if specied.

-E

-EP

(SCO UNIX, XENIX)

Use this instead of the -E option to generate preprocessor output without #line directives. The output is written to standard output. In addition, SCO UNIX copies the output to a le with the sufx .i.

-F num
Set the size of the program stack to num (hexadecimal) bytes.

(SCO UNIX, XENIX) (gcc)

-f

A family of options specifying details of C dialect to be compiled. See page 405 for more details.

-f type
Specify the kind of oating-point code to generate on Sun-2, Sun-3 and Sun-4 systems.

(SunOS) (SCO UNIX, XENIX) (SCO UNIX, XENIX) (SGI) (SCO UNIX, XENIX) (SCO UNIX, XENIX) (SGI) (SCO UNIX, XENIX) (SCO UNIX, XENIX) (SCO UNIX, XENIX)

-Fa name
Write an assembler source listing to name (default le.s).

-Fc name
Write a merged assembler and C source listing to name (default le.L).

-feedback name
Specify the name of the feedback le used in conjunction with the -cord option.

-Fe name
Specify the name of the executable le.

-Fl name -float

Cause the compiler not to promote float to double.

Write an assembler listing with assembler source and object code to name (default le.L).

-Fm name
Write a load map to name (default a.map).

-Fo name
Specify the name of the object le.

-Fp
Specify oating point arithmetic options for MS-DOS cross-compilation.

-framepointer

(SGI)

Use a register other than the stack pointer (sp) for the frame pointers (see Chapter 21, Object les and friends, page 377).

5 February 2005 04:34

Chapter 0: Compiler options

395

-fullwarn(SGI)
Produce all possible warnings.

-Fs name
Write a C source listing to name (default le.S).

(SCO UNIX, XENIX) (SVR4)

-G

Instruct the linker to create a shared object rather than a dynamically linked executable. This is incompatible with the -dn option.

-G size
Limit items to be placed in the global pointer area to size bytes.

(SGI) (all)

-g

Create additional symbolic information in order to support symbolic debuggers. gcc has a number of suboptions to specify the amount and the nature of the debugging informationsee page 406 for more details. SGI C species a numeric level for the amount of debug information to produce.

-Gc

(SCO UNIX)
Generate code with the alternate calling sequence and naming conventions used in System V 386 Pascal and System V 386 FORTRAN.

-go
Produce additional symbol table information for adb.

(SunOS) (SCO UNIX)

Removes stack probe routines. Effective only in non-protected environments.

-Gs -H
Print the names of header les to the standard output as they are #included.

(gcc, System V) (SCO UNIX, XENIX) (SCO UNIX, SunOS) (all) (SGI)
Remove /usr/include from the list of paths to search for header les.

-H num
Set the maximum length of external symbols to num.

-help
Display help for cc.

-I dir
Add dir to a list of pathnames to search for header les included by the #include directive.

-I -I-

(gcc)
Search the list of include pathnames only when the #include directive is of the form #include header". Do not search these directories if the directive is #include <header>. In addition, do not search the current directory for header les. If -I dir options are specied after -I-,

they apply for all forms of the #include directive. -i (SCO UNIX, XENIX)

5 February 2005 04:34

396 Create separate instruction and data spaces for small model programs.

-J
Change the default mode for the char type to unsigned.

(SCO UNIX) (SunOS, Sun-2 and Sun-3)

Generate 32-bit offsets in switch statements.

-J -J sfm

(SVR4)

Specify the pathname of the assembly language source math library libsfm.sa. The positioning of this option is important, since the library is searched when the name is encountered.

-j

(SGI)
Create a le le.u containing intermediate code. Does not create an object le unless used in conjunction with -c.

-KPIC
Generate position-independent code.

(SGI) (gcc)

-imacros le

Process le before reading the regular input. Do not produce any output for leonly the macro denitions will be of use.

-include le

(gcc)

Process le as input before processing the regular input le. The text of the le will be handled exactly like the regular les.

-K
Specify various code generation options.

(SVR4) (SCO UNIX, XENIX)

Remove stack probes from a program. Useful only in non-protected environments.

-K -k options
Pass options to the ucode loader.

(SGI) (SGI) (SCO UNIX, XENIX)

-ko name
Cause the output of the intermediate code loader to be called name.

-L

Create an assembler listing with assembled code and assembler source instructions with the name le.L.

-L dir

(All but SCO UNIX, XENIX)

Add dir to the list of directories to search to resolve library references. See Chapter 18, Function libraries, page 369 for further details.

-l

(all but XENIX)

Specify a library. The option -lbaz will search the library paths specied via -L options (see above) for a le typically called libbaz.a. See Chapter 18, Function libraries, page 369 for more details.

5 February 2005 04:34

Chapter 0: Compiler options

397

-LARGE

(SCO UNIX, XENIX)

Invoke the large model compiler to run. Used if heap space problems occur during compilation.

-link specs

(SCO UNIX, XENIX)

Pass specs to the linker. All text following up to the end of the command line is passed to the linker, so this has to be the last command on the line.

-M
Instruct the linker to output a message for each multiply dened external symbol.

(SVR3) (gcc, SGI, SunOS, Solaris)

Instruct the preprocessor to write a list of Makele dependencies to stdout. Suppress normal preprocessor output.

-M

-MM
Like the -M option, but only process #include le" directivesignore #include <le>.

(gcc) (gcc)
Like the -M directive, but output to a le whose name is made by replacing the nal .c with .d. This option does not suppress preprocessor output.

-MD

-MDupdate le

(SGI)

While compiling, update le to contain header, library and runtime dependency information for the output le.

-MMD
Combination of -MD and -MM. Does not suppress preprocessor output.

(gcc) (SCO UNIX, XENIX)

-Ma
Compile strict ANSI.

-M model

(SCO UNIX, XENIX)

Select model (only 16-bit modes). model may be c (compact), s (small), m (medium), l (large) or h (huge).

-M num

(SCO UNIX, XENIX)

Specify processor model for which code should be generated. 0 species 8086, 1 species 80186, 2 species 80286 and 3 species 80386 or later. 16-bit models (0 to 2) may be followed by models s, m or l.

-Mb
Reverse the word order for long types.

(SCO UNIX, XENIX) (SCO UNIX, XENIX)

Generate code for separate stack and data segments.

-Md -Me
Enable the keywords far, near, huge, pascal and fortran.

(SCO UNIX, XENIX) (SCO UNIX, XENIX)

-Mf

5 February 2005 04:34

398 Enable software oating point.

-Mt num
Set the maximum size of data items to num. Only valid for large model.

(SCO UNIX, XENIX) (SVR3)

-m
Write a load map to standard output.

-m le
Write a load map to le.

(SCO UNIX, XENIX) (SGI)

-mipsnum

Specify the target machine. num 1 (default) generates code for R2000/R3000, and 2 generates code for R4000.

-misalign
Generate code to allow loading and storing misaligned data.

(SunOS on Sun-4)

-mp(SGI)
Enable multiprocessing directives.

-n
Select pure text model (separated text and data).

(SCO UNIX, XENIX) (SCO UNIX, XENIX) (SCO UNIX, XENIX) (SCO UNIX, XENIX) (SGI) (SCO UNIX) (SGI) (SGI) (gcc, SGI)

-ND name
Set the names of each data segment to name.

-nl num
Set the maximum length of external symbols to num.

-NM name
Set the names of each module to name.

-nocpp
Do not run the preprocessor when compiling.

-nointl
Create a binary without international functionality.

-non_shared
Produce an executable that does not use shared objects.

-noprototypes
Remove prototype error and warning messages when run in -cckr mode.

-nostdinc

Do not search the standard include le locations (like /usr/include) for header les. Only search the directories specied with the -I option. gcc also has a version -nostdinc++ for C++ programs.

-nostdlib

(gcc)

Dont include the standard startup les and library paths when linking. Only les explicitly mentioned on the command line will be included.

5 February 2005 04:34

Chapter 0: Compiler options

399

-NT name
Set the names of each text segment to name.

(SCO UNIX, XENIX) (all)

-O
Perform optimizations. In some, it may be followed by a level number (-O1 normal optimizations, -O2 additional optimizations, etc.). -O means the same thing as -O1. Others, such as the SCO compiler, use letters to specify specic optimizations.

-o le

(all)

Name the output le le. System V compilers only use this option to specify the name of the nal executable, whereas other compilers use it to specify the name of the output of the nal compiler pass. This can give rise to compatibility problemssee Chapter 20, Compilers, page 351 for further details.

-oldcpp
Run with old-style cpp.

(SGI) (SGI) (SCO UNIX) (gcc)

-Olimit size -os2

Create an executable program for OS/2.

Set the maximum size of a routine to be optimized by the global optimizer to size basic blocks.

-P
Instruct the preprocessor not to generate #line commands. Used with the -E option.

-P

(SunOS, SGI, SVR4, SCO UNIX, XENIX)

Use instead of the -E option to generate preprocessor output without #line directives. The output will be stored in le.i.

-p
Generate extra code to aid proling using the proling program prof.

(all) (SCO UNIX, XENIX) (SGI) (gcc, SGI)

-pack
Ignore alignment considerations in structs and pack as tightly as possible.

-pca
Run the pca processor to discover parallelism in the source code.

-pedantic errors treats them as errors instead of warnings. -pg

Like -p, except that the output is suitable for processing by the gprof proler.

Be pedantic about syntax checking, issue all required warnings. The variety -pedantic-

(gcc, SunOS) (SunOS) (gcc, SunOS)

-pic, -PIC
Generate position-independent code. The form -PIC allows a larger global offset table.

-pipe

Specify that output from one pass should be piped to the next pass, rather than the more traditional technique of storing it in a temporary le.

5 February 2005 04:34

400

-prototypes -qp
A synonym for -p.

(SGI) (System V) (gcc (System V versions), SVR4)

Output ANSI function prototypes for all functions in the source le when run in -cckr mode.

-Qn

Do not output .ident directives to the assembler output to identify the versions of each tool used in the output le.

-Qy

(gcc (System V versions), SVR4)

Output .ident directives to the assembler output to identify the versions of each tool used in the output le.

-Qprog opt

(SunOS)

Pass option opt to program prog. prog may be as (the assembler), cpp (the preprocessor), inline (the assembly code reorganizer) or ld (the loader). -Qpath (SunOS) Specify search paths for compiler passes and other internal les, such as *crt*.o.

-Qproduce type

(SunOS)

Produce source code output of type type. type species the lename extension and may be one of .c (C source), .i (preprocessor output), .o (object output from the assembler) or .s (assembler output from the compiler).

-R
Merge the data segment into text. This creates read-only data.

(SunOS) (SCO UNIX, XENIX)

Invoke the incremental linker /lib/ldr for the link step.

-r -r
Instruct the linker to retain relocation information in the nal executable.

(SVR3) (gcc, SGI, SunOS, System V)

Stop after compiling the output assembler code, and do not assemble it. Save the results in a le le.s.

-S

-S

(SCO UNIX, XENIX)

Create a human-readable assembler source listing in le.s. This listing is not suitable for assembly.

-s
Strip the nal executable.

(SCO UNIX, XENIX, SVR3) (gcc) (SunOS)

Generate additional symbol table information for the Sun Source Code Browser.

-save-temps
Keep intermediate les even when they are no longer needed.

-sb

5 February 2005 04:34

Chapter 0: Compiler options

401

-SEG num
Set the maximum number of segments that the linker can handle to num.

(SCO UNIX, XENIX) (gcc) (SGI) (SGI) (SGI)

-shared
Produce a shared object which can be linked with other objects to form an executable.

-show
Print the names of the passes and their arguments during compilation.

-signed
Use signed characters instead of the default unsigned characters.

-sopt

Invoke the C source-to-source optimizer. There is nothing corresponding to this on other platforms.

-Ss subtitle
Sets subtitle of the source listing. This also causes the linker pass to be omitted.

(SCO UNIX) (SCO UNIX) (gcc)

-St title
Sets title of the source listing. This also causes the linker pass to be omitted.

-static

Produce a statically linked object. This is only of interest on systems which have shared libraries.

-systype
Specify the name of the compilation environment. Valid names are bsd4, svr3 and svr4.

(MIPS) (SVR3)

-t

Instruct the linker to suppress warnings about multiply dened symbols that are not the same size.

-target arch
Specify the target machine. arch can be one of sun2, sun3 or sun4.

(SunOS) (SCO UNIX)

-Tc

Specify that the input le is a C source le. This can be used if the le does not have a standard .c le name extension.

-temp=dir
Store compiler temporary les in dir.

(SunOS) (SunOS) (gcc)

-time
Print time information for each compiler pass.

-traditional

Treat the input sources as pre-ANSI-C. There is also an option -traditional-cpp which only affects the preprocessor.

-trigraphs

(gcc)

5 February 2005 04:34

402 Enable trigraph processing. By default, trigraphs are disabled unless the -ansi option is specied.

-U macro
Undene macro.

(all) (gcc, SVR3)

-u symbol

Force the linker to resolve the symbol symbol by searching additional libraries where specied.

-u
Undene all predened macros.

(SCO UNIX) (gcc)

-undef
Do not predene standard macros. This includes the macros which dene the architecture.

-use-readonly-const(SGI)
Do not allow writing to strings and aggregate constants.

-use-readwrite-const(SGI)
Allow writing to strings and aggregate constants.

-V
Print version numbers of the compiler passes as they are invoked.

(System V) (gcc 2.X) (SCO UNIX) (XENIX)

-V version
Tell gcc to run version version of gcc.

-V"string" -V version

Place string in the object le, typically for use as a copyright notice or version information. Compile a program compatible with specic versions of UNIX. version may be 2 (Seventh Edition compatible), 3 (System III compatible) or 5 (System V compatible).

-v

(gcc, SGI)
Produce verbose output. gcc output includes the complete invocation parameters of each pass and the version numbers of the passes.

-v
Perform more and stricter semantic checks.

(SVR4) (SGI) (gcc)

Without print a number of additional warning messages. With an argument, add a specic kind of warning message checksee page 407 for more details.

-varargs
Print warnings for lines that may requires the varargs.h macros.

-W

-W num

(SCO UNIX, XENIX)

Specify the level of warning messages. If num is 0, no warnings are produced. A maximum number of warnings is produced by -W3.

5 February 2005 04:34

Chapter 0: Compiler options

403

-W0,option
Pass option to the compiler.

(System V) (System V) (gcc, System V) (System V) (gcc, System V) (System V) (gcc, SCO UNIX, SunOS, XENIX)

-W2,option
Pass option to the optimizer.

-Wa,option
Pass option to the assembler.

-Wb,option
Pass option to the basic block analyzer.

-Wl,option
Pass option to the linker.

-Wp,option
Pass option to the preprocessor.

-w
Inhibit warning messages.

-w num
If num is 0 or 1, suppress warning messages. If num is 2, treat warnings as errors.

(SGI) (SGI) (SGI) (SCO UNIX, XENIX)

-wline
Produce lint-like warning messages.

-woff numbers
Suppress warning messages corresponding to numbers.

-X -Xa
Compile full ANSI C. Extensions are enabled.

Remove the standard directories from the list of directories to searched for #include les.

(SVR4) (SVR4)
Compile strictly conforming ANSI C. Extensions are disabled.

-Xc -Xcpluscomm
Allow the C++ comment delimiter // when processing C code.

(SGI) (SGI) (SCO UNIX) (SGI)

-xansi
Process ANSI C, but accept the extensions allowed by -cckr.

-xenix
Produce XENIX programs using XENIX libraries and include les.

-xgot

Compile using a 32 bit offset in the Global Symbol Table. This can be ignored for other systems.

5 February 2005 04:34

404

-x2.3

(SCO UNIX)

Produce XENIX programs using XENIX libraries and include les. The programs are compatible with release 2.3 of XENIX (the last release, with 80386 capabilities).

-Xlinker,option
Pass option to the linker.

(gcc) (SVR3)

-Xp
Compile for a POSIX.1 environment.

-Xs
Compile for a System V.3 environment (i.e. not POSIX.1).

(SVR3) (SVR4)
Compile pre-ANSI C, but with compatibility warnings.

-Xt -x
Instruct the linker to save space by not preserving local symbols in the nal executable.

(SVR3) (gcc)

-x lang

Specify the language to be compiled. lang may be one of c, objective-c, c-header, c++, cpp-output, assembler or assembler-with-cpp. This overrides the lename extensions.

-Y0,dir
Search for compiler in directory dir.

(SVR3) (SVR3) (SVR3) (SVR3) (SVR3) (SVR3) (SVR3) (gcc (System V versions)) (SVR3, gcc (System V versions))

-Y2,dir
Search for optimizer in directory dir.

-Ya,dir
Search for assembler in directory dir.

-Yb,dir
Search for basic block analyzer in directory dir.

-YI,dir
Search for Default include directory in directory dir.

-Yl,dir
Search for link editor in directory dir.

-YL,dir
Search for rst default library directory in directory dir.

-Ym,dir
Search for m4 in directory dir.

-YP,dirs

Tell the compiler to search the directories dirs (a colon-separated list, like the PATH environment variable) for libraries specied via the -l option. This is an alternative to -L. It is not additive: only the directories specied in the last -YP option are searched.

5 February 2005 04:34

Chapter 0: Compiler options

405

-Yp,dir
Search for compiler in directory dir.

(SVR3) (SVR3) (SVR3) (SCO UNIX, XENIX)

-YS,dir
Search for startup les crt1.o and crtend.o in directory dir.

-YU,dir
Search for second default library directory in directory dir.

-z
Display the passes and arguments, but do not execute them.

-z -Za
Restrict the language to ANSI specications.

(SVR3)
Instruct the linker not to bind anything at address 0 to aid run-time detection of null pointers.

(SCO UNIX, XENIX) (SCO UNIX, XENIX)

Include line number information in the object le.

-Zd -Ze the -Me option. -Zi

Include symbolic information in the object le.

(SCO UNIX)
Enables the keywords far, near, huge, pascal and fortran keywords. The same as

(SCO UNIX, XENIX) (SCO UNIX)

Do not include default library information in the object le.

-Zl -Zpalign -Zs

Perform syntax check only, do not compile.

(SCO UNIX, XENIX, SVR3) (SCO UNIX, XENIX)

Force structs to align to the an align boundaries. align may be 0, 2 or 4, and defaults to 1.

gcc dialect options

gcc supplies a large number of options to specify what dialect of C should be compiled. In addition, it supplies a further large number of options for C++ dialect. Well only look at the C dialect options herecheck the gcc release for the complete documentation. -ansi
Compile ANSI C. Flag any non-standard extension as warnings, but do not treat them as errors. This option implies the options -fn-asm and -trigraphs.

-fno-asm
Do not recognize the keywords asm, inline or typeof, so that they can be used as

5 February 2005 04:34

406 identiers. The keywords __asm__, __inline__ and __typeof__ can be used instead.

-fno-builtin
Dont recognize builtin function names that do not begin with two leading underscores.

-trigraphs
Support ANSI C trigraphs.

-traditional
Support pre-ansi dialects. This also implies -funsigned-bitfields and -fwritable-

strings. -traditional-cpp
Provide pre-ANSI style preprocessing. This is implied by -traditional.

-fcond-mismatch
Allow conditional expressions (such as a: b? c) where the second and third arguments have different types.

-funsigned-char
By default, characterss are unsigned. This effectively makes the declaration char the same thing as unsigned char.

-fsigned-char
By default, characterss are signed. This effectively makes the declaration char the same thing as signed char.

-fsigned-bitfields
Make bit elds signed by default. This is the default action.

-funsigned-bitfields
Make bit elds unsigned by default.

-fno-signed-bitfields
Make bit elds unsigned by default.

-fno-unsigned-bitfields
Make bit elds signed by default. This is the default action.

-fwritable-strings
Allocate strings in the data segment, so that the program can write to them. See Chapter 20, Compilers, page 338 for a discussion of this misfeature.

-fallow-single-precision
Do not perform operations on single precision oating point values with double precision arithmetic. This is only needed if you specify -traditional.

5 February 2005 04:34

Chapter 0: Compiler options

407

gcc debugging options

-g mods
Produce standard debugging information. This can be used in conjunction with gdb. It sometimes includes information that can confuse other debuggers.

-ggdb mods
Produce debugging information in the native format (if that is supported), including GDB extensions if at all possible.

-gstabs mods
Produce debugging information in stabs format without GDB extensions.

-gcoff mods
Produce debugging information in the COFF format used by sdb on older System V systems.

-gxcoff mods
Produce debugging information in the XCOFF format used dbs on IBM RS/6000 systems.

-gdwarf mods
Produce debugging information in the DWARF format used by sdb on most SVR4 systems.

mods are optional and may take the values + or the digits 1 to 3: + species that additional information for gdb should be included in the output. This may cause other debuggers to reject the object. 1 species that only minimal debugging information: include information about function names and external variables, but not about local variables or line numbers. 2 (the default): include function names, all variables and line numbers. In addition, 3 includes macro denitions. Not all systems support this feature.

gcc warning options

-W
Print an number of standard extra warning messages. See , starting on page , for a discussion of the individual situations.

-Wimplicit
Warn if functions or parameters are declared implicitly (in other words, if the explicit declaration is missing).

-Wreturn-type
Warn if a function is dened without a return type (in other words, one that defaults to int). Also warn if return is used without an argument in a non-void function.

-Wunused
Warn when local or static variables are not used, and if a statement computes a value which is

5 February 2005 04:34

408 not used.

-Wswitch
Warn if a switch statement has an index of an enumeral type and does not cater for all the possible values of the enum, or if a case value is specied which does not occur in the enum.

-Wcomment
Warn if the sequence /* is found within a comment. This might mean that a comment end is missing.

-Wtrigraphs
Warn if trigraphs are encountered. Only effective if -ftrigraphs is also specied.

-Wformat
Check the parameters supplied to printf, scanf and friends to ensure that they agree with the format string.

-Wchar-subscripts
Warn if an array subscript has type char.

-Wuninitialized
Warn if an automatic variable is used before it is initialized. This requires the optimizer to be enabled.

-Wparentheses
Warn if parentheses are omitted in assignments in contexts where truth values are expected (for example, if (a = foo ()), or when unusual and possibly confusing sequences of nested operators occur without parentheses.

-Wenum-clash
Warn if enum types are mixed. This is only issued for C++ programs. See Chapter 20, Compilers, page 339 for further details.

-Wtemplate-debugging
Warn if debugging is not fully available for the platform when using templates in a C++ program.

-Wall
Specify all of the warning options above. The FSF considers this a good compromise between accuracy and completeness.

-fsyntax-only
Check for syntax errors, but dont compile.

-pedantic
Issue all warnings specied by ANSI C. Reject programs which use extensions not dened in the Standard. The Free Software Foundation does not consider this to be a useful option, since ANSI C does not specify warnings for all possible situations. It is included because it is required by the ANSI Standard.

5 February 2005 04:34

Chapter 0: Compiler options

409

-pedantic-errors
The same thing as -pedantic, but the warnings are treated as errors.

-w
Inhibit all warning messages.

-Wno-import
Inhibit warning messages about the use of #import.

-Wtraditional
Warn about: Macro parameters in strings, functions declared external within a block and then referenced outside the block and switch statements with long indexes. These are treated differently in ANSI and traditional C.

-Wshadow
Warn if a local variable shadows another local variable.

-Wid-clash-len
Warn whenever two different identiers match in the rst len characters. To quote the FSF documentation: This may help you prepare a program that will compile with certain obsolete, brain-damaged compilers.

-Wpointer-arith
Warn about anything that depends on the size of a function type or of void. GNU C assigns these types a size of 1, for convenience in calculations with void * pointers and pointers to functions.

-Wcast-qual
Warn when a cast removes a type qualier from a pointer, for example if a const char * is cast to a char *.

-Wcast-align
Warn if a pointer is cast to a type which has an increased alignment requirement. For example, warn if a char * is cast to an int * on machines where integers require specic alignments.

-Wwrite-strings
Give string constants the type const char []. This will cause a warning to be generated if a string address is copied into a non-const char * pointer.

-Wconversion
Warn if the existence of a prototype causes a different type conversion from the default, or if a negative integer constant expression is implicitly converted to an unsigned type.

-Waggregate-return
Warn when functions that return structures, unions or arrays are dened or called.

-Wstrict-prototypes

5 February 2005 04:34

410 Warn if a function is declared or dened without specifying the argument types.

-Wmissing-prototypes
Warn if a global function is dened without a previous prototype declaration, even if the denition itself provides the prototype. This warning is intended to help detect missing declarations of global functions in header les.

-Wredundant-decls
Warn if anything is declared more than once in the same scope, even in cases where multiple declaration is valid and changes nothing.

-Wnested-externs
Warn if an extern declaration is encountered within an function.

-Winline
Warn if a function was declared as inline, or the C++ option -finline-functions was specied, and the function cannot be inlined.

-Woverloaded-virtual
C++ only: warn when a derived class function declaration may be an error in dening a virtual function.

-Werror
Treat all warnings as errors.

cpp options
-$ (gcc)
Disable the use of the character $ in identifers. This is passed by gcc when the -ansi option is specied.

-A
processor conditional #if #question (answer).

(gcc) -Aquestion(answer) asserts that the answer to question is answer. This can used with the pre-

-A-

(gcc)
Disable standard assertions. In addition, SVR4 cc undenes all standard macros except those beginning with __.

-B
Recognize the C++ comment string //.

(SunOS, Solaris) (gcc, SVR3, SunOS, Solaris, XENIX)

Do not strip comments from the preprocessor output.

-C -Dname -Dname.

(gcc, SVR3, SunOS, Solaris, XENIX)

Dene name as 1. This is the equivalent to specifying -Dname=1 to cc and not the same as

5 February 2005 04:34

Chapter 0: Compiler options

411

-Dname=def

(gcc, SVR3, SunOS, Solaris, XENIX)

Dene name. This is the same as the corresponding cc option. This will be overridden by

-Uname even if the -U option appears earlier on the command line. -dM (gcc)
Suppress normal preprocessor output and output #dene commands for all macros instead. This can also be used with an empty le to show the values of predened macros.

-dD

(gcc)
Do not strip #dene commands from the preprocessor output. This can be useful for debugging preprocessor macros.

-H
Print the pathnames of included les on stderr.

(gcc, SVR3, SunOS, Solaris) (gcc, SVR3, SunOS, Solaris) (gcc)

Search the list of include pathnames only when the #include directive is of the form #include header". Do not search these directories if the directive is #include<header>. In addition, do not automatically search the current directory for header les. If -I dir options are speci-

-Idir
Add dir to the path to search for #include directives.

-I-

ed after -I-, they apply for all forms of the #include directive. -imacros le

(gcc)

Process le before reading the regular input. Do not produce any output for leonly the macro denitions will be of use.

-include le

(gcc)

Process le as input before processing the regular input le. The text of the le will be handled exactly like the regular les.

-idirafter dir

(gcc)

Add dir to the second include path. The second include path is an include path which is searched when a le isnt found in the standard include path (the one built by the -I option).

-iprefix prex
Specify a prex for the -iwithprefix option (see next entry).

(gcc) (gcc)

-iwithprefix dir

Add a the directory prex/dir to the second include path. prex must previously have been set with the iprex command.

-lang-language

(gcc)

Specify the source language. -lang-c++ enables the comment sequence //, -lang-objc enables the #import command, -lang-objc++ enables both, -lang-c disables both.

-lint

(gcc)

Replace lint commands such as /* NOTREACHED */ with the corresponding pragma, e.g. #pragma lint NOTREACHED.

5 February 2005 04:34

412

-M -MM

(gcc, SunOS, Solaris)

Write a list of Makele dependencies to stdout. Suppress normal preprocessor output.

(gcc)
Like the -M option, but only process #include le"directivesignore #include <le>.

-MD

(gcc)
Like the -M directive, but output to a le whose name is made by replacing the nal .c with .d. This option does not suppress preprocessor output.

-MMD
Combination of -MD and -MM. Does not suppress preprocessor output.

(gcc) (gcc)

-nostdinc

Do not search the standard include le locations (like /usr/include) for header les. Only search the directories specied with the -I option. A version -nostdinc++ exists for C++ programs.

-P
Do not output #line directives.

(gcc, SVR3, SunOS, Solaris, XENIX) (SunOS, Solaris)

Limit the length of preprocessor directives to 8 characters.

-p -pedantic -pedantic-errors

(gcc) (gcc)

Issue the warnings that ANSI C species for specic situations. See Page 399 for more details. If the situations specied in the ANSI Standard occur, option them as errors rather than warnings.

-R
Allow recursive macros.

(SunOS, Solaris) (SVR3, SunOS, Solaris)

Limit the length of preprocessor directives to 8 characters. For backward compatibility only.

-T -traditional
Preprocess in the traditional (pre-ANSI) manner.

(gcc) (gcc) (SunOS, Solaris) (SVR3, SunOS, Solaris, XENIX)

-trigraphs
Recognize and convert trigraphs.

-undef
Undene all predened symbols.

-Uname

Remove denition of name. This will also override -D options placed later on the command line.

-undef

(gcc)

5 February 2005 04:34

Chapter 0: Compiler options Do not predene standard macros.

413

-Ydir
Search only directory dir for #include les.

(SunOS, Solaris) (gcc) (gcc)

-Wall
Set both -Wcomment and -Wtrigraphs.

-Wcomment

Warn if the sequence /* is found within a comment. This could imply that a comment end is missing.

-Wtraditional

(gcc)

Warn about macro parameters in strings. These are treated differently in ANSI and traditional C.

-Wtrigraphs
Warn if trigraphs are encountered. Only effective if -ftrigraphs is also specied.

(gcc)

5 February 2005 04:34

Assembler directives and options

as options
Its particularly evident that as seldom sees the light of day when you look at the options, which differ greatly from one system to the next. GNU as doesnt even maintain compatibility between versions 1 and 2, as you can see in the following table: -a (GNU 2.x)
List high-level language, assembly output, and symbols. This is the generic form of the -a option; the following variants modify this in some manner. Combinations are possible: for example, -alh lists the high-level input and assembly output, but not the symbol table. In order to get the high-level language input, you also need to specify the -g option.

-ad

(GNU 2.x)
List high-level language, assembly output, and symbols, but omit debugging pseudo-ops from listing.

-ah
List high-level language source.

(GNU 2.x) (GNU 2.x)

List assembly output.

-al -an

(GNU 2.x)
Disable forms processing of the listing. This only works in combination with other -a options.

-as
List symbols.

(GNU 2.x) (GNU 1.x)

Turn on assembler debugging (if available).

-D -D
No effect just for compatibility.

(GNU 2.x) (SVR3)

415

-dl

5 February 2005 02:09

416 Dont put line number information in object le.

-f
skip preprocessing (for compiler output)

(GNU 2.x) (GNU 1.x)

Generate debugging symbols for source language debugging of assembly programs.

-g -I path
Add path to the search list for .include directives

(GNU 2.x) (GNU 2.x)

-K
Issue warnings when difference tables altered for long displacements.

-k
Warn about problems with calculating symbol differences.

(GNU 1.x) (GNU)

Keep local symbols starting with L in the symbol table output to object le.

-L -m
preprocess with m4

(System V) (System V)
Turn off long/short address optimization.

-n -o
Specify output le name.

(GNU 2.x, System V) (System V)

Put assembler version number in object le.

-Qy -R
Merge the data segment into the text segment, making it read-only.

(GNU 1.x) (System V)

Remove the input le after assembly.

-R -W
Suppress warnings.

(GNU 1.x) (GNU 1.x)

Suppress the preprocessor pass which removes comments and redundant white space from the input. This can also be done with the #NO_APP directive.

-f

-T
Accept (and ignore) obsolete directives without complaining.

(System V) (System V)
Print the current version number.

-V -v
Print the current version number.

(GNU)

5 February 2005 02:09

Appendix C: Assembler directives and options

417

-W
Suppress warning messages

(GNU 2.x)

-Y
Specify directory for m4 processor and predened macros (Y,dir).

(System V) (System V)
Specify directory for predened macros (Yd,dir).

-Yd -Ym
Specify directory for m4 processor (Ym,dir).

(System V)

as directives
Assembler directives are mainly provided for the convenience of the compiler, and are seldom documented. Here is a list of the directives provided by GNU as, one of the few which is documented. Many of these directives are provided only on certain platformsread Using as, by Dean Elsner and Jay Fenlason, for specic information. .abort
Abort the assembly. This is obsolescent. It was intended to be used by a compiler piping its output into the assembler when it discovered a fatal error.

.ABORT
A synonym for .abort.

.align boundary [, content]

Increment the assembler location counter, (the pointer to the location where the next byte will be emitted), to a boundary which has zeros in the last boundary binary positions. If content is specied, any bytes skipped will be lled with this value.

.app-file string
Specify the start of a new logical le string. This is obsolescent.

.ascii string . . .
Emit each string into consecutive addresses. Do not append a trailing \0 character.

.asciz string
Emit each string into consecutive addresses. Append a trailing \0 character.

.byte expressions
Emit zero or more expressions into the next output byte.

.comm symbol , length

Declare symbol a named common area in the bss section. length is the minimum lengththe actual length will be determined by the linker as the maximum of the length elds of all object

5 February 2005 02:09

418 les which dene the symbol.

.data subsection
Switch to data section subsection (default zero). All assembled data will go to this section.

.def name
Begin dening COFF debugging information for a symbol name. The denition is completed by a .endef directive.

.desc symbol, abs-expression

Set the symbol descriptor to the low 16 bits of abs-expression. This is ignored if the assembler is outputting in COFF format.

.double onums
Emit double oating point number onums.

.eject
Force a page break in the assembly listing at this point.

.else
else in conditional assemblysee the .if directive.

.endef
End a symbol denition begun with .def.

.endif
End a conditional assembly block. See the .if directive.

.equ symbol, expression

Set the value of symbol to expression. This is the same thing as .set.

.extern
In some assemblers, dene a symbol external to the program. This is ignored by GNU as, which treats all undened symbols as external.

.file string
Specify the start of a new le. This directive is obsolescent, and may not be available.

.fill repeat , size , value

Create repeat repeated data blocks consisting of the low-order size bytes of value.

.float onums
Emit oating point numbers onums.

.global symbol

5 February 2005 02:09

Appendix C: Assembler directives and options Dene symbol as an external symbol.

419

.globlsymbol
A synonym for .global.

.hword expressions
Emit the values of each expression, truncated to 16 bits if necessary.

.ident
This directive is used by some assemblers to place tags in object les. GNU as ignores it.

.if expression
If expression evaluates to non zero, assemble the following code down to the corresponding

.else or .endif directive. If the next directive is .else, do not assemble the code between the .else and the .endif. If expression evaluates to 0, do not assemble the code down to the corresponding .else or .endif directive. .ifdef symbol
Like .if, but the condition is fullled if symbol is dened.

.ifndef symbol
Like .if, but the condition is fullled if symbol is not dened.

.ifnotdef symbol
Like .if, but the condition is fullled if symbol is not dened.

.include le"
Process the source le le before continuing this le.

.int expressions
Emit 32 bit values of each expression.

.lcomm symbol , length

Reserve length bytes of local common in bss, and give it the name symbol.

.ln line-number
Change the logical line number of the next line to line-number. This corresponds to the C preprocessor line directive.

.ln line-number
A synonym for .line.

.list
Increment the listing counter (initially 0). If the listing counter is > 0, the following lines will be listed in the assembly listing, otherwise they will not. .nolist decrements the counter.

5 February 2005 02:09

420

.long expressions
A synonym for .int.

.nolist
Decrement the listing countersee .list.

.octa bignums
Evaluate each bignum as a 16 byte integer and emit its value.

.org new-lc, ll
Set the location counter of the current section to new-lc. new-lc must be either absolute or an expression in the current subsection: you cant use .org to cross sections. .org may not decrement the location counter. The intervening bytes are lled with the value ll (default 0).

.psize lines, columns

Set the page size for assembly listings to lines lines (default 60) and columns columns (default 200). If lines is set to 0, no automatic pagination will occur.

.quad bignums
Evaluate each bignum as an 8 byte integer and emit its value.

.sbttl subheading
Set the subtitle of assembly listings to subheading.

.section name, subsection

Switch to section called name (default .text), subsection (default zero). All emitted data goes to this section.

.set symbol, expression

Dene the value of symbol to be expression. This may be used more than once to change the value of symbol after it is dened. The value of an external symbol will be the value of the last .set directive.

.short expressions
Emit the values of each expression, truncated to 16 bits if necessary.

.single onums
Emit oating point numbers onums. This is the same as .float.

.space size, ll
Emit size bytes of value ll. ll defaults to 0.

.space
Usually a synonym for .block, but on some hardware platforms GNU as uses it differently.

5 February 2005 02:09

Appendix C: Assembler directives and options

421

.stabd
Emit debug information. See page for more information.

.stabn
Emit debug information. See page for more information.

.stabs
Emit debug information. See page for more information.

.text subsection
Switch to text section subsection (default zero). All assembled data will go to this section.

.title heading
Set the title of the assembly listing to heading.

.word expressions
Emit 32 bit values of each expression.

Debug information
Debug information is very dependent on the kind of object le format in use: In a.out format, it is dened by the directives .stabd, .stabn and .stabs. They can take up to ve parameters: desc is the symbol descriptor, and is 16 bits wide. other is the symbols other attribute. This is normally not used. string is the name of the symbol. type is the symbol type, and is 8 bits wide. value is the value of the symbol, and must be absolute.

These symbols are used as follows: .stabd type, other, desc

Dene a debugging entry without a name. The value of the symbol is set to the current value of the location counter. This is commonly used for line number information, which is type 68 for line number references in the text segment. For example .stabd 68, 0, 27 species that the current location is the beginning of line 27.

.stabn type, other, desc, value

Dene a debugging entry without a name. The value of the symbol is set to value.

.stabs string, type, other, desc, value

Dene a debugging entry with the name string. The value of the symbol is set to value.

5 February 2005 02:09

422

For further information about stabs formats and types, see the header le stab.h and the man page stab(5). In COFF format, it is dened by the directives .dim, .scl, .size, .tag, .type and .val. They are enclosed in a .def/.endef pair. For example, to dene a symbol foo, you would write
.def foo .value bar .size 4 .endef

.dim
Set dimension information.

.scl class
Set the storage class value of the symbol to class.

.size size
Set the size of the symbol to size.

.tag structname
Specify the struct denition of the current symbol.

.type int
Set the type of the symbol to type.

.val addr
Set the value of the symbol to addr.

In ELF format, debug information is output to a special section called .debug, so no specic directives are needed.

5 February 2005 02:09

Linker options
Like the assembler, the linker seldom sees the light of day: you normally start both programs via the C compiler control program cc. As with the assembler, this gives rise to a surprising diversity of options. The following list compares the linker options for the GNU linkers (two of them, with conicting options), SCO UNIX, Solaris 2, SunOS 4, System V.3, System V.4, and SCO XENIX. Currently available BSD systems use one of the GNU linkers: for example, BSD/386 up to version 1.1 uses the old linker, and BSD/OS 2.0 uses the new linker. The Solaris 2 linker is basically the System V.4 linker, but it has a few extra ags. Unless otherwise noted, all SVR4 options also apply to Solaris 2. -Aarchitecture (GNU)
For the Intel 960 family only: architecture is a two-letter abbreviation specifying a member of the processor family.

-A le

(old GNU)

Dont incorporate the text and data from le into the output le, just use the symbols. This can be used to implement crude dynamic loading.

-A le

(SunOS 4)

Perform an incremental load: the resultant output le is to be read in to a process executing from the program le, which will be used to resolve symbolic references.

-A address
Produce a standalone program to be loaded at address.

(XENIX) (SCO, SVR3, SVR4)

-a -align datum

Produce an executable le. This is the default behaviour, and is the opposite of the -r option.

(SunOS 4) (SunOS 4) (SunOS 4, Solaris 2)

Force datum to be page-aligned. This is typically used for FORTRAN common blocks.

-assert assertion
Check an assertion. If the assertion fails, print a diagnostic and abort the link.

-Bbinding

Specify the kind of binding to perform. binding may be dynamic (perform dynamic binding at run time), nosymbolic (do not perform symbolic relocation), static (perform static 423

5 February 2005 04:34

424 binding at link time), or symbolic (force symbolic relocation). Solaris 2 does not support the keyword nosymbolic.

-Bstatic
Specify static libraries only. GNU ld accepts this option, but ignores it.

(SunOS 4, GNU) (XENIX) (SVR4)

-B number
Set the text selector bias to number

-b

When performing dynamic linking, do not perform special processing for relocations to symbols in shared objects.

-b format

(new GNU)

Specify the binary format of the les whose names follow. This is only needed when linking les with multiple formats.

-C
Ignore the case of the symbols.

(XENIX) (new GNU) (XENIX) (old GNU, SunOS 4)

-c le
Read commands from le. These commands override the standard link format.

-c x
Specify the target CPU type 80x86. x defaults to 3.

-D size

Pad the data segment to size. The padding may overlap with the bss segment. The SunOS 4 linker interprets size in hexadecimal.

-D number
Set the data selector bias to number.

(XENIX) (SVR4) (GNU, SunOS 4)

-dyn
Specify dynamic (yn is y) or static (yn is n) linking.

-d -dc

When creating a relocatable output le with the -r option, convert common symbols to bss.

(SunOS 4)
Perform the -d option, but also copy initialized data referenced by this program from shared objects.

-dp -defsym symbol = expression -e symbol

Set the entry address in the output le to symbol.

(SunOS 4)
Force an alias denition of undened procedure entry points. Used with dynamic binding.

(new GNU) (all)

Create the global symbol symbol in the output le and assign the value expression to it.

5 February 2005 04:34

Appendix D: Linker options

425

-Fformat

(new GNU)

This is an obsolete option which some older linkers used to specify object le formats. GNU ld accepts it, but ignores it.

-F name

(Solaris 2)

Used when building shared objects. The symbol table of the shared object being built is used as a lter on the symbol table of the shared object name.

-F size
Reserve size bytes for the run-time stack.

(XENIX) (SCO, SVR3)

-f ll

Fill unassigned memory (gaps in text and data segments, and also the bss segment) with the 16-bit pattern ll.

-format format -G size

(new GNU) (new GNU)

Specify the binary format of the les whose names follow. This is the same as the -b option. Only for MIPS ECOFF format: set the minimum size of objects to be optimized using the GP register.

-G
Produce a shared object in dynamic mode.

(Solaris 2) (new GNU, XENIX)

Include symbolic information in the output le. The GNU linker accepts this option, but ignores it, since this is the default behaviour.

-g

-h name
When building a dynamic object, record name as the name of the le to link at run time.

(SVR4) (Solaris 2)

-I name

Use name as the path name of the interpreter to be written into the program header. In static mode, name defaults to no interpreter, and in dynamic mode it defaults to /usr/lib/ld.so.1.

-i
Create a relocatable output le. Same as the -r option.

(new GNU) (Solaris 2)

Ignore the LD_LIBRARY_PATH setting.

-i -i
Create separate instruction and data space for small model programs.

(XENIX) (all)

-L dir
Search the given directory for library archives in addition to the default directories. ld searches directories supplied with the -L option in order of appearance in the argument list and before the default directories.

-l lib

(all)

5 February 2005 04:34

426 Search the specied libraries for a library called liblib.a. This is the same as the C compiler -l option. SunOS4 allows you to write -l lib.version to indicate a specific library version number.

-La
Set advisory le locking

(XENIX) (XENIX)
Set mandatory le locking.

-Lm -LI[NENUMBERS]
Create a map le including line number information.

(SCO) (GNU, SunOS 4)

-M
Print a load map on the standard output.

-M maple
Read directives to ld from maple.

(Solaris 2) (SCO, SVR3)

-M
Print warning messages for multiply dened external denitions.

-m
Print a load map on the standard output.

(SCO, SVR3, SVR4) (XENIX)

Specify the memory model. x can be s (small), m (middle), l (large), h (huge), or e (mixed).

-Mx -m emulation
Emulate the emulation linker.

(new GNU) (XENIX) (SCO) (new GNU) (GNU, SunOS 4)

-m le
Write a map listing to le.

-M[AP]:number
Create a map listing with up to number symbols. number defaults to 2048.

-Map le
Print a load map to le.

-N

Create an OMAGIC format binary. This is the default format for relocatable object les. OMAGIC format binaries have writable text segments. Where appropriate, this option implies -Bstatic.

-N

(SVR3)
Place the text section at the beginning of the text segment, and the data segment immediately after the text segment.

-N num
Set the page size to num bytes.

(XENIX)

5 February 2005 04:34

Appendix D: Linker options

427

-n

(GNU, SunOS 4)
Create an NMAGIC format shared executable binary. The text segment is read-only. Where appropriate, this option implies -Bstatic.

-n num
Truncate symbol names to num characters.

(XENIX) (new GNU) (all) (new GNU) (XENIX)

-noinhibit-exec
Create an output le even if errors are encountered during linking.

-o le
Write output to le instead of the default a.out.

-oformat format
Write the output le in format format.

-P
Disable packing of segments.

-p
Start the data segment on a page boundary, even if the text segment is not shared.

(SunOS 4) (Solaris 2)

-Qyn

If yn is y, add an ident string to the .comment section of the output le identifying the version of the linker used. cc does this by default. -Qn suppresses this header.

-q
Create a QMAGIC format demand loaded executable binary.

(old GNU on BSD) (new GNU) (XENIX)

Ensure a relocation table of non-zero size.

-R le
Read symbol information from le, but do not include it in the output.

-R -Rd offset
Set the data segment relocation offset to offset.

(XENIX) (XENIX) (Solaris 2)

-Rt offset
Set the text segment relocation offset to offset.

-R paths

Specify paths as a colon-separated list of directories to be searched for libraries by the runtime linker.

-r
Generate a relocatable output le.

(all) (GNU, SunOS 4)

Strip only stab symbols from a.out les.

-S -s

(all)

5 February 2005 04:34

428 Strip all symbols from the output le. This overrides other strip options.

-SE[GMENTS]:number
Allow the program to have number segments. The default value is 128.

(SCO) (new GNU) (SCO) (new GNU)

-sort-common
Disable sorting of common blocks by size.

-ST[ACK]:size
Specify that the stack should be size bytes long.

-T le

Read commands from le. These commands override the standard link format. This is the same as the -c option.

-T address
Start the text segment at address.

(old GNU, SunOS 4) (new GNU) (GNU, SunOS 4) (GNU, SunOS 4) (GNU)

-Tbss address
Start the bss segment at address.

-Tdata address
Start the data segment at address.

-Ttext address
Start the text segment at address. The same as -T.

-t
Print the names of input les to stderr as they are processed.

-t
Do not warn about multiply dened symbols of different size.

(SCO, SVR3, SVR4) (all)

-u symbol

Consider symbol to be undened. This can be used to force the extraction of certain les from a library.

-Ur

(new GNU)
Generate relocatable output, like the -r option. For C++ programs only, resolve references to constructors.

-V
Print full version number information, including supported emulations.

(new GNU) (SCO, SVR3, Solaris 2)

Print version number information for ld.

-V -VS number
Store version number in the optional header of the output le.

(SCO, SVR3) (new GNU)

-v
Print version number information for ld only.

5 February 2005 04:34

Appendix D: Linker options

429

version
Print version number information for ld only, then exit.

(new GNU) (new GNU)

-warn-common

Warn when a common symbol is combined with another common symbol or with a symbol denition.

-X

(GNU, SunOS 4)
Strip local symbols which start with the letter L. This is the default behaviour of the assembler. The new GNU linker will only perform this operation if the -s or -S options are also specied.

-x

(GNU, SCO, SunOS 4, SVR3)

Strip all local symbols. The new GNU linker will only perform this operation if the -s or -S options are also specied.

-Y [L][U], dir

(SCO, SVR3, SVR4 in BSD mode)

Change the default directory used for nding libraries. If L is specied, the standard library directory (LLIBDIR, normally /usr/lib) is replaced with dir. If U is specied and the linker was built with a second library directory (LLIBDIR), it is replaced with dir.

-YP, dir
Change the default directory used for nding libraries to dir.

(Solaris 2) (old GNU, SunOS 4) (old GNU, SunOS 4)

-y symbol
Trace symbol on stderr during linking.

-z

Create a ZMAGIC format demand loaded executable binary. On SunOS 4, this implies the -Bdynamic option.

-z -z defs

(SCO, SVR3)
Do not bind anything at address 0, in order to allow run-time detection of null pointers.

(Solaris 2)

Force a fatal error if any undened symbols remain at the end of a link. This is the default for executables, but not for relocatable output.

-z nodefs
Allow undened symbols in an executable.

(Solaris 2) (Solaris 2)

-z text

Force a fatal error if any relocations against non-writable, allocatable sections remain when performing a dynamic link.

5 February 2005 04:34

Where to get sources

In this book, Ive mentioned a large quantity of freely available software. Heres some information on how to get it. All the software is available on the Internet, but a large quantity is also available on CD-ROM, sometimes ported to specic platforms. Your choice of where to get it from depends a lot on how often you need the software, how fast your connection to the Internet is and how much you pay to transfer data on the Internet. I personally prefer to keep as many packages as possible on CD-ROM.

CD-ROM producers
A large number of companies produce CD-ROMs, but the following are of particular interest: The Free Software Foundation 675 Massachusetts Avenue Cambridge MA, 02139 Phone: +1 617 876 3296 Mail: gnu@prep.ai.mit.edu
The producers of GNU software. They sell a CD-ROM with all the GNU software. If you buy your CD-ROM here, you also help support the Free Software Foundation, which is dependent on such income to survive. The primary Internet site for the Free Software Foundation is prep@ai.mit.edu, and you can nd the software in the directory /pub/gnu. This site is frequently overloaded, so please use more local mirrors where possible.

InfoMagic, Inc. P.O. Box 30370 Flagstaff AZ 86003-0370 Phone: +1-800-800-6613 +1-602-526-9565
431

5 February 2005 04:34

432

Fax: +1-602-526-9573 Mail: info@infomagic.com

A number of UNIX-oriented CDs, including Internet tools, Linux, and standards.

OReilly & Associates 103A Morris Street Sebastopol CA 95472 Phone: +1-800-998-9938 +1-707-829-0515 Fax: +1-707-829-0104 Mail: order@ora.com
Our favourite source. High-quality, well-thought out books on UNIX, many with CD-ROMs.

Prime Time Freeware 370 Altair Way, Suite 150 Sunnyvale CA 94086 Phone: +1-408-433-9662 Fax: +1-408-433-0727 Mail: ptf@cfcl.com
A small supplier of software of special interest to programmers: well-organized, six-monthly distributions of the latest software packages, source only, including separate editions for T X E and articial intelligence. In addition, ported software for System V.4 (Intel) and Sun platforms.

Walnut Creek CD-ROM 4041 Pike Lane, Suite D-893 Concord CA 94520 Phone: +1-800-786-9907 +1-510-674-0783 Fax: +1-510-674-0821 Mail: orders@cdrom.com
By far the largest choice of CDs, including nearly everything the other companies have to offer. Mainly ported software, including denitive FreeBSD distribution and ported software for System V.4 (Intel), Linux, and Sun.

5 February 2005 04:34

Appendix E: Where to get sources Table E1: Software sources

433

Package GNU software 4.4BSD Lite ghostscript jargon le ncurses patch RCS tcpdump TeX dvips dviware ghostview POSIX.2 regex SeeTeX t1ascii X11 System V shared memory
1

Internet site prep.ai.mit.edu:/pub/gnu in GNU distribution In GNU distribution netcom.com:pub/zmbenhal/ncurses in GNU distribution in GNU distribution ftp.ee.lbl.gov ftp.shsu.edu:/tex-archive in TEX distribution in TEX distribution in TEX distribution zoo.toronto.edu:/pub in TEX distribution in TEX distribution ftp.x.org:/pub/R6 ftp.ora.com ll in this space XXX

CD suppliers FSF, PTF, WC IM, ORA, WC FSF, PTF, WC PTF, FSF PTF, WC, FSF PTF, WC, FSF WC1 PTF, WC PTF PTF PTF PTF PTF PTF, ORA, WC

tcpdump is included in the FreeBSD distribution.

The initials of the CD-ROM publishers are self-evident. The initials in bold print are, in my personal opinion, the best choice for the package in question. Your mileage may vary.

5 February 2005 04:34

Bibliography
4.4 Berkeley Software Distribution System Managers Manual. OReilly & Associates, Inc., 1994. A Fast File System for UNIX. Marshall Kirk McKusick, William N. Joy, Samuel J. Lefer and Robert S. Fabry. A description of the Berkeley Fast File System, now known as ufs (Unix File System). Chapter 5 of the Berkeley Software Distribution System Managers Manual. A Stream Input-Output System. Dennis M. Ritchie, in AT&T Bell Laboratories Technical Journal, Volume 63, No. 8, Part 2, page 1897. A description of the original Eighth Edition Streams concept. Advanced Programming in the UNIX environment. W. Richard Stevens, Addison Wesley 1992. An excellent treatise on systems programming under modern UNIX variants. CVS II: Parallelizing Software Development. Brian Berliner, in RCS distribution. Checking C Programs with lint. Ian F. Darwin, OReilly & Associates Inc., 1988. A description of the lint program checker. Connecting to the Internet. Susan Estrada, OReilly & Associates Inc., 1993. A Buyers Guide to the Internet. Encyclopedia of graphics le formats. James D. Murray and William vanRyper, OReilly and Associates Inc., 1994. Includes CD-ROM. A complete description of graphics le formats. Learning the UNIX Operating System. Grace Todino, John Strang and Jerry Peek, OReilly and Associates Inc., 1993. A straightforward introduction to UNIX. Making TEX work. Norman Walsh, OReilly and Associates Inc., 1994. A usable book on T X. E Managing projects with make, Second Edition. Andrew Oram and Steve Talbott, OReilly and Associates Inc., 1991. Managing uucp and Usenet, tenth edition. Tim OReilly and Grace Todino, OReilly and Associates Inc., 1992. The denitive guide to uucp management. POSIX Programmers Guide. Donald A. Lewine, OReilly & Associates Inc., 1991. Portability of C Programs and the UNIX System. S. C. Johnson, D. M. Ritchie, published in the Bell System Techinical Journal July/August 1978, Volume 57, No. 6, Part 2, pages
435

5 February 2005 04:34

436

2021-2048. An early description of the portable C compiler. Practical C Programming, second edition. Steve Oualline, OReilly and Associates Inc., 1992. A mid-level book stressing robust programming techniques rather then clever programming. Programming with curses. John Strang, OReilly and Associates Inc., 1986. A description of the BSD version of curses. Programming with GNU Software. Mike Loukides, OReilly & Associates, Inc., 1995 RCS A System for Version Control. Walter F. Tichy, 1991. Part of the RCS distribution. RS-232 made easy, second edition. Martin D. Seyer, Prentice-Hall 1991. A discussion of the RS-232 standard. SCO UNIX in a Nutshell. Ellie Cutler and the staff of OReilly & Associates, Inc. OReilly and Associates Inc., 1994. Software Portability with imake. Paul DuBois, OReilly & Associates Inc., 1993. A complete manual for imake and associated topics. TCP/IP Illustrated, Volume 1. W. Richard Stevens, Addison Wesley 1994. A description of the IP protocol suite from the viewpoint of the tcpdump program. Termcap and Terminfo. John Strang, Tim OReilly and Linda Mui. OReilly and Associates Inc., 1989. A description of Termcap and Terminfo. The TEXbook , Donald E. Knuth, Addison Wesley, 1989. A mystery story about TEX, also the main reference. The Annotated C++ Reference Manual, Margaret A. Ellis and Bjarne Stroustrup. Addison Wesley, 1990. The denitive reference manual for C++. The C Programming Language, rst edition. Brian W. Kernighan, Dennis M. Ritchie, Prentice-Hall, 1978. The rst complete description of the C programming language. The Design and the Implementation of the 4.3BSD UNIX Operating System. Samuel J. Lefer, Marshall Kirk McKusick, Michael J. Karels, John S. Quarterman. Addison-Wesley, 1990. The denitive description of the 4.3BSD kernel and communications. The Design of the UNIX System. Maurice J. Bach, Prentice-Hall, 1986. An in-depth description of an early version of System V. The Magic Garden explained: The Internals of UNIX System V Release 4. Berny Goodheart and James Cox, Prentice-Hall 1994. The denitive guide to the internals of System V Release 4. The New Hackers Dictionary. Eric Raymond (ed.), MIT Press 1991. The Standard C Library. P. J. Plauger, Prentice-Hall 1992. An in-depth description of the Standard (i.e. ANSI) C library. The Whole Internet Users Guide and Catalog, second edition. Ed Krol, OReilly & Associates Inc., 1994. The denitive guide to the Internet. The evolution of CPast and Future. L. Rosler, in AT&T Bell Laboratories Technical Journal, Volume 63, No. 8, Part 2, page 1685. A description of the state of the C language in 1984.

5 February 2005 04:34

Appendix F: Bibliography

437

Travels into several remote nations of the world, by Lemuel Gulliver. Jonathan Swift, Bejamin Motte, London, 1726. A satirical treatment of early 18th century English politics. Typesetting tables with tbl. Henry McGilton, Mary McNabb, Trilithon Press, 1990. A tutorial introduction to tbl. UNIX Curses Explained. Berny Goodheart, Prentice-Hall 1991. A description of BSD and System V versions of Curses. UNIX in a Nutshell, for System V and Solaris 2.0. Daniel Gilly and the staff of OReilly & Associates, Inc. OReilly and Associates Inc., 1992. UNIX in a Nutshell, for 4.3BSD. c the staff of OReilly & Associates, Inc. OReilly and Associates Inc., 1990. UNIX Network Programming. W. Richard Stevens, Prentice-Hall 1990. Includes a comparison of sockets and STREAMS. UNIX Power Tools. Jerry Peek, Tim OReilly, Mike Loukides, OReilly and Associates Inc., 1993. Includes CD-ROM. An indispensible collection of tips and tricks. UNIX System V Application Binary Interface, Revised Edition, UNIX Press, 1992. UNIX Time-sharing system UNIX Programmers Manual, Seventh Edition. Holt, Rinehart and Winston, 1979. The original Seventh Edition UNIX documentation (two volumes). Understanding and using COFF. Gintaras R. Gircys, OReilly & Associates Inc., 1988. A description of the Common Object File Format. Using as. A detailed description of GNU as, Dean Elsner and Jay Fenlason, source only from the Free Software Foundation. Part of the GNU binutils distribution. Using uucp and Usenet. Grace Todino and Dale Dougherty, OReilly and Associates Inc., 1987. The denitive guide to using uucp. X Window System Administrators Guide (Volume 8 of the X Window System documentation). Linda Mui, Eric Pearce, OReilly & Associates Inc., 1993. Available with companion CD-ROM. lex and yacc, Second Edition. John R. Levine, Tony Mason and Doug Brown, OReilly & Associates Inc., 1992. sed and awk. Dale Dougherty, OReilly & Associates Inc., 1992. An in-depth treatment of both utilities. sendmail. Bryan Costales with Eric Allman and Neil Rickert, OReilly & Associates Inc., 1993. Everything you never wanted to know about sendmail.

5 February 2005 04:34