Mixed Effects Models in S and S-Plus PDF

To Elisa and Laura
To Mary Ellen, Barbara, and Michael
Preface
Mixed-eects models provide a exible and powerful tool for the analysis of
grouped data, which arise in many areas as diverse as agriculture, biology,
economics, manufacturing, and geophysics. Examples of grouped data include longitudinal data, repeated measures, blocked designs, and multilevel
data. The increasing popularity of mixed-eects models is explained by the
exibility they oer in modeling the within-group correlation often present
in grouped data, by the handling of balanced and unbalanced data in a
unied framework, and by the availability of reliable and ecient software
for tting them.
This book provides an overview of the theory and application of linear and nonlinear mixed-eects models in the analysis of grouped data.
A unied model-building strategy for both linear and nonlinear models is
presented and applied to the analysis of over 20 real datasets from a wide variety of areas, including pharmacokinetics, agriculture, and manufacturing.
A strong emphasis is placed on the use of graphical displays at the various
phases of the model-building process, starting with exploratory plots of the
data and concluding with diagnostic plots to assess the adequacy of a tted
model. Over 170 gures are included in the book.
The class of mixed-eects models considered in this book assumes that
both the random eects and the errors follow Gaussian distributions. These
models are intended for grouped data in which the response variable is (at
least approximately) continuous. This covers a large number of practical
applications of mixed-eects models, but does not include, for example,
generalized linear mixed-eects models (Diggle, Liang and Zeger, 1994).
viii
Preface
The balanced mix of real data examples, modeling software, and theory
makes this book a useful reference for practitioners who use, or intend to
use, mixed-eects models in their data analyses. It can also be used as a text
for a one-semester graduate-level applied course in mixed-eects models.
Researchers in statistical computing will also nd this book appealing for
its presentation of novel and ecient computational methods for tting
linear and nonlinear mixed-eects models.
The nlme library we developed for analyzing mixed-eects models in implementations of the S language, including S-PLUS and R, provides the
underlying software for implementing the methods presented in the text,
being described and illustrated in detail throughout the book. All analyses
included in the book were produced using version 3.1 of nlme with S-PLUS
3.4 running on an Iris 5.4 Unix platform. Because of platform dependencies, the analysis results may be expected to vary slightly with dierent
computers or operating systems and with dierent implementations of S.
Furthermore, the current version of the nlme library for R does not support
the same range of graphics presentations as does the S-PLUS version. The
latest version of nlme and further information on the NLME project can
be obtained at
http://nlme.stat.wisc.edu or
http://cm.bell-labs.com/stat/NLME.
Errata and updates of the material in the book will be made available
on-line at the same sites.
The book is divided into parts. Part I, comprising ve chapters, is dedicated to the linear mixed-eects (LME) model and Part II, comprising
three chapters, covers the nonlinear mixed-eects (NLME) model. Chapter 1 gives an overview of LME models, introducing some examples of
grouped data and the type of analyses that applies to them. The theory
and computational methods for LME models are the topics of Chapter
2. Chapter 3 describes the structure of grouped data and the many facilities available in the nlme library to display and summarize such data.
The model-building approach we propose is described and illustrated in
detail in the context of LME models in Chapter 4. Extensions of the basic LME model to include variance functions and correlation structures for
the within-group errors are considered in Chapter 5. The second part of
the book follows an organization similar to the rst. Chapter 6 provides
an overview of NLME models and some of the analysis tools available for
them in nlme. The theory and computational methods for NLME models
are described in Chapter 7. The nal chapter is dedicated to model building
in the context of NLME models and to illustrating in detail the nonlinear
modeling facilities available in the nlme library.
Even though the material covered in the book is, for the most part,
self-contained, we assume that the reader has some familiarity with linear
regression models, say at the level of Draper and Smith (1998). Although
enough theory is covered in the text to understand the strengths and weak-
Preface
ix
nesses of mixed-eects models, we emphasize the applied aspects of these.

Readers who desire to learn in more detail the theory of mixed-eects
are referred to the excellent book by Davidian and Giltinan (1995). Some
knowledge of the S language is denitely desireable, but not a pre-requisite
for following the material in the book. For those who are new to, or less
familiar with S, we suggest using in conjunction with this book the, by now,
classic reference Venables and Ripley (1999), which provides an overview
of S and an introduction to a wide variety of statistical models to be used
with S.
The authors may be contacted via electronic mail at
jcp@research.bell-labs.com
bates@stat.wisc.edu
and would appreciate being informed of typos, errors, and improvements
to the contents of this book.
Typographical Conventions:
The S language objects and commands referenced throughout the book are
printed in a monospaced typewriter font like this, while the S classes are
printed in sans-serif font like this. The standard prompt > is used for S
commands and the prompt + is used to indicate continuation lines.
To save space, some of the S output has been edited. Omission of complete lines are usually indicated by
. . .
but some blank lines have been removed without indication. The S output
was generated using the options settings
> options( width = 68, digits = 5 )
The default settings are for 80 and 7, respectively.

Acknowledgements:
This book would not exist without the help and encouragement of many
people with whom we have interacted over the years. We are grateful to
the people who have read and commented on earlier versions of the book;
their many useful suggestions have had a direct impact on the current
organization of the book. Our thanks also go to the many people who have
tested and made suggestions on the nlme library, in particular the beta
testers for the current version of the software. It would not be possible to
name all these people here, but in particular we would like to thank John
Chambers, Yonghua Chen, Bill Cleveland, Saikat DebRoy, Ram
on DasUriarte, David James, Diane Lambert, Renaud Lancelot, David Lansky,
Brian Ripley, Elisa Santos, Duncan Temple Lang, Silvia Vega, and Bill
Preface
Venables. Finally, we would like to thank our editor, John Kimmel, for his
continuous encouragement and support.
Jose C. Pinheiro
Douglas M. Bates
March 2000
Contents
Preface
Linear Mixed-Eects Models
1 Linear Mixed-Eects Models

1.1 A Simple Example of Random Eects . . . . . . . . . . . .
1.1.1 Fitting the Random-Eects Model With lme . . . .
1.1.2 Assessing the Fitted Model . . . . . . . . . . . . . .
1.2 A Randomized Block Design . . . . . . . . . . . . . . . . .
1.2.1 Choosing Contrasts for Fixed-Eects Terms . . . . .
1.2.2 Examining the Model . . . . . . . . . . . . . . . . .
1.3 Mixed-Eects Models for Replicated, Blocked Designs . . .
1.3.1 Fitting Random Interaction Terms . . . . . . . . . .
1.3.2 Unbalanced Data . . . . . . . . . . . . . . . . . . . .
1.3.3 More General Models for the Random Interaction
Eects . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4 An Analysis of Covariance Model . . . . . . . . . . . . . . .
1.4.1 Modeling Simple Linear Growth Curves . . . . . . .
1.4.2 Predictions of the Response and the Random Eects
1.5 Models for Nested Classication Factors . . . . . . . . . . .
1.5.1 Model Building for Multilevel Models . . . . . . . .
1.6 A Split-Plot Experiment . . . . . . . . . . . . . . . . . . . .
1.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
vii
1
3
4
8
11
12
14
19
21
23
25
27
30
30
37
40
44
45
52
52
xii
Contents
2 Theory and Computational Methods for LME Models

2.1 The LME Model Formulation . . . . . . . . . . . . . . . . .
2.1.1 Single Level of Grouping . . . . . . . . . . . . . . . .
2.1.2 A Multilevel LME Model . . . . . . . . . . . . . . .
2.2 Likelihood Estimation for LME Models . . . . . . . . . . .
2.2.1 The Single-Level LME Likelihood Function . . . . .
2.2.2 Orthogonal-Triangular Decompositions . . . . . . . .
2.2.3 Evaluating the Likelihood Through Decompositions
2.2.4 Components of the Proled Log-Likelihood . . . . .
2.2.5 Restricted Likelihood Estimation . . . . . . . . . . .
2.2.6 Multiple Levels of Random Eects . . . . . . . . . .
2.2.7 Parameterizing Relative Precision Factors . . . . . .
2.2.8 Optimization Algorithms . . . . . . . . . . . . . . .
2.3 Approximate Distributions . . . . . . . . . . . . . . . . . . .
2.4 Hypothesis Tests and Condence Intervals . . . . . . . . . .
2.4.1 Likelihood Ratio Tests . . . . . . . . . . . . . . . . .
2.4.2 Hypothesis Tests for Fixed-Eects Terms . . . . . .
2.4.3 Condence Intervals . . . . . . . . . . . . . . . . . .
2.5 Fitted Values and Predictions . . . . . . . . . . . . . . . . .
2.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
57
58
58
60
62
62
66
68
71
75
77
78
79
81
82
83
87
92
94
94
96
3 Describing the Structure of Grouped Data

3.1 The Display Formula and Its Components . . . . . . . . . .
3.2 Constructing groupedData Objects . . . . . . . . . . . . . .
3.2.1 Roles of Other Experimental or Blocking Factors . .
3.2.2 Constructors for Balanced Data . . . . . . . . . . . .
3.3 Controlling Trellis Graphics Presentations of Grouped Data
3.3.1 Layout of the Trellis Plot . . . . . . . . . . . . . . .
3.3.2 Modifying the Vertical and Horizontal Scales . . . .
3.3.3 Modifying the Panel Function . . . . . . . . . . . . .
3.3.4 Plots of Multiply-Nested Data . . . . . . . . . . . .
3.4 Summaries . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
97
97
101
104
108
110
110
113
114
116
120
130
130
4 Fitting Linear Mixed-Eects Models

4.1 Fitting Linear Models in S with lm and lmList
4.1.1 The lmList Function . . . . . . . . . .
4.2 Fitting Linear Mixed-Eects Models with lme .
4.2.1 Fitting Single-Level Models . . . . . . .
4.2.2 Patterned VarianceCovariance Matrices
Random Eects: The pdMat Classes . .
4.2.3 Fitting Multilevel Models . . . . . . . .
4.3 Examining a Fitted Model . . . . . . . . . . . .
133
134
139
146
146
. .
. .
. .
. .
for
. .
. .
. .
. . .
. . .
. . .
. . .
the
. . .
. . .
. . .
.
.
.
.
.
.
.
.
. . 157
. . 167
. . 174
Contents
4.3.1 Assessing Assumptions

4.3.2 Assessing Assumptions
4.4 Chapter Summary . . . . . .
Exercises . . . . . . . . . . . . . .
on
on
. .
. .
the
the
. .
. .
Within-Group Error
Random Eects . .
. . . . . . . . . . . .
. . . . . . . . . . . .
xiii
.
.
.
.
174
187
196
197
5 Extending the Basic Linear Mixed-Eects Model

5.1 General Formulation of the Extended Model . . . . . . . . .
5.1.1 Estimation and Computational Methods . . . . . . .
5.1.2 The GLS model . . . . . . . . . . . . . . . . . . . . .
5.1.3 Decomposing the Within-Group VarianceCovariance
Structure . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Variance Functions for Modeling Heteroscedasticity . . . . .
5.2.1 varFunc classes in nlme . . . . . . . . . . . . . . . .
5.2.2 Using varFunc classes with lme . . . . . . . . . . . .
5.3 Correlation Structures for Modeling Dependence . . . . . .
5.3.1 Serial Correlation Structures . . . . . . . . . . . . .
5.3.2 Spatial Correlation Structures . . . . . . . . . . . . .
5.3.3 corStruct classes in nlme . . . . . . . . . . . . . . .
5.3.4 Using corStruct Classes with lme . . . . . . . . . .
5.4 Fitting Extended Linear Models with gls . . . . . . . . . .
5.5 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
201
202
202
203
II
271
Nonlinear Mixed-Eects Models
205
206
208
214
226
226
230
232
239
249
266
267
6 NLME Models: Basic Concepts and Motivating

Examples
6.1 LME Models vs. NLME Models . . . . . . . . . .
6.2 Indomethicin Kinetics . . . . . . . . . . . . . . .
6.3 Growth of Soybean Plants . . . . . . . . . . . . .
6.4 Clinical Study of Phenobarbital Kinetics . . . . .
6.5 Chapter Summary . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
273
273
277
287
294
300
301
7 Theory and Computational Methods for NLME

7.1 The NLME Model Formulation . . . . . . . . . .
7.1.1 Single-Level of Grouping . . . . . . . . . .
7.1.2 Multilevel NLME Models . . . . . . . . .
7.1.3 Other NLME Models . . . . . . . . . . . .
7.2 Estimation and Inference in NLME Models . . .
7.2.1 Likelihood Estimation . . . . . . . . . . .
7.2.2 Inference and Predictions . . . . . . . . .
7.3 Computational Methods . . . . . . . . . . . . . .
7.4 Extending the Basic NLME Model . . . . . . . .
Models
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
. . . . . .
305
306
306
309
310
312
312
322
324
328
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
xiv
Contents
7.5
7.6
7.4.1 General model formulation . . . . . . .

7.4.2 Estimation and Computational Methods
An Extended Nonlinear Regression Model . . .
7.5.1 General Model Formulation . . . . . . .
Chapter Summary . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
8 Fitting Nonlinear Mixed-Eects Models

8.1 Fitting Nonlinear Models in S with nls and nlsList . .
8.1.1 Using the nls Function . . . . . . . . . . . . . .
8.1.2 Self-Starting Nonlinear Model Functions . . . . .
8.1.3 Separate Nonlinear Fits by Group: The nlsList
Function . . . . . . . . . . . . . . . . . . . . . . .
8.2 Fitting Nonlinear Mixed-Eects Models with nlme . . .
8.2.1 Fitting Single-Level nlme Models . . . . . . . . .
8.2.2 Using Covariates with nlme . . . . . . . . . . . .
8.2.3 Fitting Multilevel nlme Models . . . . . . . . . .
8.3 Extending the Basic nlme Model . . . . . . . . . . . . .
8.3.1 Variance Functions in nlme . . . . . . . . . . . .
8.3.2 Correlation Structures in nlme . . . . . . . . . .
8.3.3 Fitting Extended Nonlinear Regression Models
with gnls . . . . . . . . . . . . . . . . . . . . . .
8.4 Chapter Summary . . . . . . . . . . . . . . . . . . . . .
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
328
329
332
333
334
336
337
. . 338
. . 338
. . 342
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
347
354
354
365
385
391
391
395
. . 401
. . 409
. . 410
References
A Data Used in Examples and Exercises
A.1 AlfalfaSplit-Plot Experiment on Varieties of Alfalfa . . .
A.2 AssayBioassay on Cell Culture Plate . . . . . . . . . . .
A.3 BodyWeightBody Weight Growth in Rats . . . . . . . .
A.4 CefamandolePharmacokinetics of Cefamandole . . . . .
A.5 CO2Carbon Dioxide Uptake . . . . . . . . . . . . . . . .
A.6 DialyzerHigh-Flux Hemodialyzer . . . . . . . . . . . . .
A.7 DNaseAssay Data for the Protein DNase . . . . . . . .
A.8 EarthquakeEarthquake Intensity . . . . . . . . . . . . .
A.9 ergoStoolErgometrics Experiment with Stool Types . .
A.10 Glucose2Glucose Levels Following Alcohol Ingestion . .
A.11 IGFRadioimmunoassay of IGF-I Protein . . . . . . . . .
A.12 IndomethIndomethicin Kinetics . . . . . . . . . . . . . .
A.13 LoblollyGrowth of Loblolly Pine Trees . . . . . . . . . .
A.14 MachinesProductivity Scores for Machines and Workers
A.15 OatsSplit-plot Experiment on Varieties of Oats . . . . .
A.16 OrangeGrowth of Orange Trees . . . . . . . . . . . . . .
A.17 OrthodontOrthodontic Growth Data . . . . . . . . . . .
415
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
423
425
425
427
427
428
429
429
430
431
432
433
433
434
435
435
436
436
Contents
A.18 OvaryCounts of Ovarian Follicles . . . . . . . . . . .

A.19 OxboysHeights of Boys in Oxford . . . . . . . . . . .
A.20 OxideVariability in Semiconductor Manufacturing .
A.21 PBGEect of Phenylbiguanide on Blood Pressure . .
A.22 PBIBA Partially Balanced Incomplete Block Design
A.23 PhenobarbPhenobarbitol Kinetics . . . . . . . . . . .
A.24 PixelPixel Intensity in Lymphnodes . . . . . . . . .
A.25 QuinidineQuinidine Kinetics . . . . . . . . . . . . . .
A.26 RailEvaluation of Stress in Rails . . . . . . . . . . .
A.27 SoybeanSoybean Leaf Weight over Time . . . . . . .
A.28 SpruceGrowth of Spruce Trees . . . . . . . . . . . .
A.29 TheophTheophylline Kinetics . . . . . . . . . . . . .
A.30 WaferModeling of Analog MOS Circuits . . . . . . .
A.31 Wheat2Wheat Yield Trials . . . . . . . . . . . . . .
B S Functions and Classes
ACF . . . . . . . . . . .
ACF.lme . . . . . . . .
anova.lme . . . . . . .
coef.lme . . . . . . .
coef.lmList . . . . .
fitted.lme . . . . . .
fixef . . . . . . . . .
gapply . . . . . . . . .
getGroups . . . . . . .
gls . . . . . . . . . . .
gnls . . . . . . . . . .
groupedData . . . . .
gsummary . . . . . . .
intervals . . . . . . .
intervals.lme . . . .
intervals.lmList . .
lme . . . . . . . . . . .
lmeControl . . . . . .
lmList . . . . . . . . .
logLik . . . . . . . . .
nlme . . . . . . . . . .
nlmeControl . . . . .
nlsList . . . . . . . .
pairs.lme . . . . . . .
plot.lme . . . . . . .
plot.nfnGroupedData
plot.nmGroupedData
plot.Variogram . . .
predict.lme . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
xv
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
437
437
437
438
439
440
440
441
443
443
444
444
448
448
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
451
451
452
453
455
457
458
459
460
461
462
464
466
469
471
471
473
474
476
478
479
479
483
485
486
488
490
492
494
495
xvi
Contents
qqnorm.lme . . . . .
ranef . . . . . . . .
ranef.lme . . . . . .
ranef.lmList . . . .
residuals.lme . . .
selfStart . . . . . .
selfStart.default
selfStart.formula
Variogram . . . . . .
Variogram.lme . . .
.
.
.
.
.
.
.
.
.
.
497
498
499
501
503
504
505
506
507
508
C A Collection of Self-Starting Nonlinear Regression

Models
C.1 SSasympThe Asymptotic Regression Model . . . . . . . .
C.1.1 Starting Estimates for SSasymp . . . . . . . . . . . .
C.2 SSasympOffAsymptotic Regression with an Oset . . . .
C.2.1 Starting Estimates for SSasympOff . . . . . . . . . .
C.3 SSasympOrigAsymptotic Regression Through the Origin
C.3.1 Starting Estimates for SSasympOrig . . . . . . . . .
C.4 SSbiexpBiexponential Model . . . . . . . . . . . . . . . .
C.4.1 Starting Estimates for SSbiexp . . . . . . . . . . . .
C.5 SSfolFirst-Order Compartment Model . . . . . . . . . .
C.5.1 Starting Estimates for SSfol . . . . . . . . . . . . .
C.6 SSfplFour-Parameter Logistic Model . . . . . . . . . . .
C.6.1 Starting Estimates for SSfpl . . . . . . . . . . . . .
C.7 SSlogisSimple Logistic Model . . . . . . . . . . . . . . .
C.7.1 Starting Estimates for SSlogis . . . . . . . . . . . .
C.8 SSmicmenMichaelisMenten Model . . . . . . . . . . . . .
C.8.1 Starting Estimates for SSmicmen . . . . . . . . . . .
511
511
511
512
512
513
513
514
515
516
516
517
518
519
519
520
521
Index
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
523
Part I
Linear Mixed-Eects
Models
1
Linear Mixed-Eects Models:
Basic Concepts and Examples
Many common statistical models can be expressed as linear models that

incorporate both xed eects, which are parameters associated with an
entire population or with certain repeatable levels of experimental factors,
and random eects, which are associated with individual experimental units
drawn at random from a population. A model with both xed eects and
random eects is called a mixed-eects model.
Mixed-eects models are primarily used to describe relationships between
a response variable and some covariates in data that are grouped according
to one or more classication factors. Examples of such grouped data include
longitudinal data, repeated measures data, multilevel data, and block designs.
By associating common random eects to observations sharing the same
level of a classication factor, mixed-eects models exibly represent the
covariance structure induced by the grouping of the data.
In this chapter we present an overview of linear mixed-eects (LME)
models, introducing their basic concepts through the analysis of several
real-data examples, starting from simple models and gradually moving to
more complex models. Although the S code to t these models is shown,
the purpose here is to present the motivation for using LME models to
analyze grouped data and not to concentrate on the software for tting and
displaying the models. This chapter serves as an appetizer for the material
covered in later chapters: the theoretical and computational methods for
LME models described in Chapter 2 and the linear mixed-eects modeling
facilities available in the nlme library, covered in detail in Chapter 4.
The examples described in this chapter also serve to illustrate the breadth
of applications of linear mixed-eects models.
1. Linear Mixed-Eects Models
1.1 A Simple Example of Random Eects

The data shown in Figure 1.1 are from an experiment in nondestructive
testing for longitudinal stress in railway rails cited in Devore (2000, Example 10.10, p. 427). Six rails were chosen at random and tested three times
each by measuring the time it took for a certain type of ultrasonic wave
to travel the length of the rail. The only experimental setting that changes
between the observations is the rail. We say these observations are arranged
in a one-way classication because they are classied according to a single
characteristicthe rail on which the observation was made. These data are
described in greater detail in Appendix A.26.
The quantities the engineers were interested in estimating from this experiment are the average travel time for a typical rail (the expected travel
time), the variation in average travel times among rails (the between-rail
variability), and the variation in the observed travel times for a single rail
(the within-rail variability). We can see from Figure 1.1 that there is considerable variability in the mean travel time for the dierent rails. Overall
the between-rail variability is much greater than the within-rail variability.
The data on the rails experiment are given in an object called Rail that
is available with the nlme library. Giving the name Rail by itself to the S
interpreter will result in the data being displayed.
> Rail
Grouped Data: travel ~ 1 | Rail
Rail travel
1
1
55
2
1
53
3
1
54
. . .
17
6
85
18
6
83
4
Rail
3
6
1
5
2
40
60
80
100
travel
FIGURE 1.1. Travel time in nanoseconds for ultrasonic head-waves in a sample of six railroad rails. The times shown are the result of subtracting 36,100
nanoseconds from the original observation.
As would be expected, the structure of the data is quite simpleeach row

corresponds to one observation for which the rail and the travel time are
recorded. The names of the variables in the data frame are Rail and travel.
There is also a formula, travel ~ 1 | Rail, associated with the data. This
formula is discussed in Chapter 3, where we describe structures available
in the nlme library for representing grouped data.
Data from a one-way classication like the rails example can be analyzed
either with a xed-eects model or with a random-eects model. The distinction between the two models is according to whether we wish to make
inferences about those particular levels of the classication factor that were
used in the experiment or to make inferences about the population from
which these levels were drawn. In the latter case the levels usually correspond to dierent subjects or dierent plots or dierent experimental units
of some sort.
To illustrate the importance of accounting for the classication factor
when modeling grouped data such as the rails example, we initially ignore
the grouping structure of the data and assume the simple model
yij = + ij ,
i = 1, . . . , M,
j = 1, . . . , ni ,
(1.1)
where yij is the observed travel time for observation j on rail i, is the
mean travel time across the population of rails being sampled, and the ij
are independent N (0, 2 ) error terms. The number of rails is M and the
number of observations on rail i is ni . In this case M =6 and n1 = n2 =
M
= n6 = 3. The total number of observations is N = i=1 ni = 18.
The lm function is used to t the single-mean model (1.1) in S. Its rst
argument is a formula describing the model and its second argument is a
data frame containing the variables named in the model formula.
> fm1Rail.lm <- lm( travel ~ 1, data = Rail )
> fm1Rail.lm
Call:
lm(formula = travel ~ 1, data = Rail)
Coefficients:
(Intercept)
66.5
Degrees of freedom: 18 total; 17 residual
Residual standard error: 23.645
As is typical with S, we do not produce output directly from the tting

process. Instead we store the tted model as an object called fm1Rail.lm
then cause this object to be displayed. It contains the parameter estimates
= 66.5 and
= 23.645.
The boxplots of the residuals from the fm1Rail.lm t by rail number,
displayed in Figure 1.2, illustrate the fundamental problem with ignoring
Rail
3
6
1
5
2
-40
-20
20
Residuals
FIGURE 1.2. Boxplots of residuals by rail number for the lm t of the single-mean
model (1.1) to the data from the rail experiment.
the classication factor when modeling grouped data: the group eects
are incorporated into the residuals (which, in this case, have identical signs
for each rail), leading to an inated estimate of the within-rail variability.
The rail eects indicated in Figure 1.2 may be incorporated into the
model for the travel times by allowing the mean of each rail to be represented by a separate parameter. This xed-eects model for the one-way
classication is written
yij = i + ij ,
i = 1, . . . , M,
j = 1, . . . , ni ,
(1.2)
where the i represents the mean travel time of rail i and, as in (1.1), the
errors ij are assumed to be independently distributed as N (0, 2 ). We can
again use lm to t (1.2).
> fm2Rail.lm <- lm( travel ~ Rail - 1, data = Rail )
> fm2Rail.lm
Call:
lm(formula = travel ~ Rail - 1, data = Rail)
Coefficients:
Rail2 Rail5 Rail1 Rail6 Rail3 Rail4
31.667
50
54 82.667 84.667
96
A -1 is used in the model formula to prevent the default inclusion of an

intercept term in the model. As expected, there is considerable variation
in the estimated mean travel times per rail. The residual standard error
obtained for the xed-eects model (1.2),
= 4.0208, is about one-sixth
of the corresponding estimate obtained for the single-mean model (1.1),
indicating that the fm2Rail.lm model has successfully accounted for the rail
eects. This is better illustrated by the boxplots of the fm2Rail.lm residuals
Rail
3
6
1
5
2
-6
-4
-2
Residuals
FIGURE 1.3. Boxplots of residuals by rail number for the lm t of the xed-eects
model (1.2) to the data from the rail experiment.
by rail number, shown in Figure 1.3. The residuals are now centered around
zero and have considerably smaller magnitudes than those in Figure 1.2.
Even though the xed-eects model (1.2) accounts for the rail eects, it
does not provide a useful representation of the rails data. Its basic problem
is that it only models the specic sample of rails used in the experiment,
while the main interest is in the population of rails from which the sample
was drawn. In particular, fm2Rail.lm does not provide an estimate of the
between-rail variability, which is one of the central quantities of interest in
the rails experiment. Another drawback of this xed-eects model is that
the number of parameters in the model increases linearly with the number
of rails.
A random-eects model circumvents these problems by treating the rail
eects as random variations around a population mean. The following reparameterization of model (1.2) helps motivate the random-eects model
for the rails data. We write

(1.3)
yij = + i + ij ,

where = 6i=1 i /6 represents the average travel time for the rails in the
experiment. The random-eects model replaces by the mean travel time
over the population of rails and replaces the deviations i by random
variables whose distribution is to be estimated.
A random-eects model for the one-way classication used in the rails
experiment is written
yij = + bi + ij ,
(1.4)
where is the mean travel time across the population of rails being sampled, bi is a random variable representing the deviation from the population
mean of the mean travel time for the ith rail, and ij is a random variable
representing the deviation in travel time for observation j on rail i from
the mean travel time for rail i.
To complete the statistical model, we must specify the distribution of

the random variables bi , i = 1, . . . , M and ij , i = 1, . . . , M ; j = 1, . . . , ni .
We begin by modeling both of these as independent, constant variance,
normally distributed random variables with mean zero. The variances are
denoted b2 for the bi , or between-rail variability, and 2 for the ij , or
within-rail variability. That is,
bi N (0, b2 ),
ij N (0, 2 ).
(1.5)
This model may be modied if it does not seem appropriate. As described

in Chapter 4, we encourage using graphical and numerical diagnostic tools
to assess the validity of the model and to suggest ways in which it could
be modied. To start, however, we will use this simple model.
This model with two sources of random variation, bi and ij , is sometimes
called a hierarchical model (Lindley and Smith, 1972; Bryk and Raudenbush, 1992) or a multilevel model (Goldstein, 1995). The bi are called random eects because they are associated with the particular experimental
unitsrails in this casethat are selected at random from the population
of interest. They are eects because they represent a deviation from an
overall mean. That is, the eect of choosing rail i is to shift the mean
travel time from to + bi . Because observations made on the same rail
share the same random eect bi , they are correlated. The covariance between
on the same rail is b2 corresponding to a correlation of

2observations
2
2
b / b + .
The parameters of the statistical model created by combining (1.4) and
(1.5) are , b2 , and 2 . Note that the number of parameters will always
be three, irrespective of the number of rails in the experiment. Although
the random eects, bi , i = 1, . . . , M may behave like parameters, formally
they are just another level of random variation in the model so we do not
estimate them as such. We will, however, form predictions bi of the values
of these random variables, given the data we observed.
1.1.1 Fitting the Random-Eects Model With lme

The lme function from the nlme library for S can be used to t linear
mixed-eects models, using either maximum likelihood (ML) or restricted
maximum likelihood (REML). These estimation methods for the parameters in LME models are described in detail in 2.2.
A typical call to lme is similar to a call to lm. As in lm, the rst two
arguments to lme, fixed and data, give the model for the expected response
(the xed-eects part of the model) and the object containing the data
to which the model should be t. The third argument, random, is a onesided formula describing the random eects and the grouping structure
for the model. Another important argument is method. Specifying method =
"ML" produces maximum likelihood ts while method = "REML", the default,
produces restricted maximum likelihood ts.
Many variations in the specications of linear mixed-eects models for

lme are possible, as shown later in this and other chapters. Details of all
the possible arguments and their forms are given in Appendix B.

We obtain the restricted maximum likelihood t of the model given by
(1.4) and (1.5) to the Rail data with
> fm1Rail.lme <- lme(travel ~ 1, data = Rail, random = ~ 1 | Rail)
The rst argument indicates that the response is travel and that there is
a single xed eect, the intercept. The second argument indicates that the
data will be found in the object named Rail. The third argument indicates
that there is a single random eect for each group and that the grouping
is given by the variable Rail. Note that there is a variable or column Rail
within the data frame that is also named Rail. Because no estimation
method is specied, the default, "REML", is used.
We can query the tted lme object, fm1Rail.lme, using dierent accessor
functions, also described in detail in Appendix B. One of the most useful
of these is the summary function
> summary( fm1Rail.lme )
Linear mixed-effects model fit by REML
Data: Rail
AIC
BIC logLik
128.18 130.68 -61.089
Random effects:
Formula: ~ 1 | Rail
(Intercept) Residual
StdDev:
24.805
4.0208
Fixed effects: travel ~ 1
Value Std.Error DF t-value p-value
(Intercept) 66.5
10.171 12 6.5382 <.0001
Standardized Within-Group Residuals:
Min
Q1
Med
Q3
Max
-1.6188 -0.28218 0.035693 0.21956 1.6144
Number of Observations: 18
Number of Groups: 6
We see that the REML estimates for the parameters have been calculated
as
= 66.5,
b = 24.805,
= 4.0208,
corresponding to a log-restricted-likelihood of 61.089. The estimated mean

travel time is identical to the estimated intercept in the fm1Rail.lm t,
10
and the estimated within-rail standard deviation

is identical to the residual standard error from fm2Rail.lm. This will not occur in general; it is a
consequence of the Rail data being a balanced one-way classication that
has the same number of observations on each rail. We also note that the
estimated between-rail standard deviation
b is similar to the residual standard error from the fm1Rail.lm t.
The output of the summary function includes the values of the Akaike
Information Criterion (AIC ) (Sakamoto, Ishiguro and Kitagawa, 1986)
and the Bayesian Information Criterion (BIC ) (Schwarz, 1978), which is
also sometimes called Schwarzs Bayesian Criterion (SBC ). These are
model comparison criteria evaluated as
AIC = 2 log Lik + 2npar ,
BIC = 2 log Lik + npar log(N ),
where npar denotes the number of parameters in the model and N the
total number of observations used to t the model. Under these denitions, smaller is better. That is, if we are using AIC to compare two or
more models for the same data, we prefer the model with the lowest AIC.
Similarly, when using BIC we prefer the model with the lowest BIC.
To examine the maximum likelihood estimates we would call lme with the
same arguments as for fm1Rail.lme except for method = "ML". A convenient
way of tting such alternative models is to use the update function where
we only need to specify the arguments that are dierent from those in the
earlier t.
> fm1Rail.lmeML <- update( fm1Rail.lme, method = "ML" )
> summary( fm1Rail.lmeML )
Linear mixed-effects model fit by maximum likelihood
Data: Rail
AIC
BIC logLik
134.56 137.23 -64.28
Random effects:
Formula: ~ 1 | Rail
StdDev:
22.624
4.0208
Fixed effects: travel ~ 1
(Intercept) 66.5
9.554 12 6.9604 <.0001
Min
Q1
Med
Q3
Max
-1.611 -0.28887 0.034542 0.21373 1.6222
Number of Groups: 6
Standardized residuals
11
1.5
1.0
0.5
0.0
-0.5
-1.0
-1.5
30
40
50
60
70
80
90
Fitted values
FIGURE 1.4. Standardized residuals versus the tted values for the REML t of
a random-eects model to the data from the rail experiment.
Notice that the ML estimate of is 4.0208, the same as the REML

estimate. Equality of the ML and REML estimates of occurs for this
simple model, but will not occur in general. The ML estimate of b , 22.624,
is smaller than the REML estimate, 24.805. Finally the ML estimate of ,
66.5, is the same as the REML estimate. Again, exact equality of the ML
and REML estimates of the xed eects need not occur in more complex
models, but it is commonplace for them to be nearly identical.
1.1.2 Assessing the Fitted Model

The tted model can, and should, be examined using graphical and numerical summaries. One graphical summary that should be examined routinely
is a plot of the residuals versus the tted responses from the model. This
plot is used to assess the assumption of constant variance of the ij . Because this plot is a common diagnostic, it is the default plot method for a
tted lme model. That is, it is produced by the simple call
> plot( fm1Rail.lme )
# produces Figure 1.4
The standardized residuals, shown on the vertical axis in Figure 1.4, are
the raw residuals, eij = yij bi , divided by the estimated standard
deviation,
, of the ij .
In this plot we are looking for a systematic increase (or, less commonly,
a systematic decrease) in the variance of the ij as the level of the response
increases. If this is present, the residuals on the right-hand side of the plot
will have a greater vertical spread than those on the left, forming a horizontal wedge-shaped pattern. Such a pattern is not evident in Figure 1.4.
With more complicated models there are other diagnostic plots that we
may want to examine, as discussed in Chapter 4.
12
We should also examine numerical summaries of the model. A basic summary is a set of condence intervals on the parameters, , and b , as
produced by the intervals function.
> intervals( fm1Rail.lme )
Approximate 95% confidence intervals
Fixed effects:
lower est. upper
(Intercept) 44.339 66.5 88.661
Random Effects:
Level: Rail
lower
est. upper
sd((Intercept)) 13.274 24.805 46.354
Within-group standard error:
lower
est. upper
2.695 4.0208 5.9988
We can see that there is considerable imprecision in the estimates of all

three of these parameters.
Another numerical summary, used to assess the signicance of terms in
the xed-eects part of the model, is produced by the anova function
> anova( fm1Rail.lme )
numDF denDF F-value p-value
(Intercept)
1
12 42.748 <.0001
In this case, the xed-eects model is so simple that the analysis of variance
is trivial. The hypothesis being tested here is = 0. The p-value, which
is that probability of observing data as unusual as these or even more so
when actually is 0, is so small as to rule out this possibility. Regardless
of the p-value, the hypothesis = 0 is of no practical interest here because
the data have been shifted by subtracting 36,100 nanoseconds from each
measurement.
1.2 A Randomized Block Design

In the railway rails example of the last section, the observations were classied according to one characteristic onlythe rail on which the observation
was made. In other experiments we may have more than one classication
factor for each observation. A randomized block design is a type of experiment in which there are two classication factors: an experimental factor
for which we use xed eects and a blocking factor for which we use random
eects.
The data shown in Figure 1.5 and available as the object ergoStool in

+
Subject
1
2
1
7
3
6
9
4
5
8
>
2
s
s
s
>
+
>
s
s
s
s
>
>
>
>
>
10
4
+
+
>
s
s
13
+
>
+
+
+
+
12
14
Effort required to arise (Borg scale)
FIGURE 1.5. Eort required (Borg scale) to arise from a stool for nine dierent
subjects each using four dierent types of stools. Dierent symbols, shown in the
key at the top of the plot, are used for the dierent types of stools.
the nlme library are from an ergometrics experiment that has a randomized
block design. The experimenters recorded the eort required by each of
nine dierent subjects to arise from each of four types of stools. We want
to compare these four particular types of stools so we use xed eects
for the Type factor. The nine dierent subjects represent a sample from
the population about which we wish to make inferences so we use random
eects to model the Subject factor.
From Figure 1.5 it appears that there are systematic dierences between
stool types on this measurement. For example, the T2 stool type required
the greatest eort from each subject while the T1 stool type was consistently
one of the low eort types. The subjects also exhibited variability in their
scoring of the eort, but we would expect this. We say that Subject to be
a blocking factor because it represents a known source of variability in the
experiment. Type is said to be an experimental factor because the purpose
of the experiment is to determine if there are systematic dierences in the
level of eort to arise from the dierent types of stools.
We can visually compare the magnitude of the eects of the Type and
Subject factors using a design plot
> plot.design( ergoStool )
This plot is produced by averaging the responses at each level of each factor
and plotting these averages. We see that the variability associated with the
Type factor is comparable to the variability associated with the Subject
factor. We also see that the average eort according to stool type is in the
order T1 T4 T3 T2.
11
2
1
3
7
6
9
5
8
Type
Subject
10
9
mean of effort
12
14
Factors
FIGURE 1.6. Design plot for the data in the stool ergometric experiment. The
mean of the response (effort) is plotted for each level of each of the factors Type
and Subject.
1.2.1 Choosing Contrasts for Fixed-Eects Terms

A model with xed eects j for the Type factor and random eects bi for
the Subject factor could be written
yij = j + bi + ij ,
bi
i = 1, . . . , 9,
N (0, b2 ),
j = 1, . . . , 4,
ij N (0, 2 ),
(1.6)
or, equivalently,
y i = X i + Z i bi + i ,
bi
N (0, b2 ),
where, for i = 1, . . . , 9,

yi1
1
yi2
0
yi = , X i =
yi3
0
yi4
0
0
1
0
0
0
0
1
0
i = 1, . . . , 9,
i N (0, 2 I),
0
0
,
0
1

1
1
Zi = 1 =
,
1
1

i1
i2
.
i =
i3
i4
This form of xed-eects matrix X i is sometimes called the cell means

form because the jth component of represents what would be the mean
eort to arise from the jth type of stool if the whole population were tested.
These j have a simple interpretation, but are not convenient to use
when assessing dierences between stool types. To make it easier to assess
these dierences we use an alternative form of the X i matrices with one column representing some overall mean or reference level and three columns
representing changes between the types of stools. The three columns representing the changes are called the contrasts. There are several dierent
choices available for these contrasts (Venables and Ripley, 1999, 6.2). In
S-PLUS, the default choice for unordered factors, such as the Type factor,
is the Helmert contrasts
15
> contrasts( ergoStool$Type )

[,1] [,2] [,3]
1
-1
-1
-1
2
1
-1
-1
3
0
2
-1
4
0
0
3
(In R the default contrasts for an unordered factor are the treatment
contrasts, which are described below.)
The X i matrices for a given set of contrasts can be displayed with the
model.matrix function. To save space we show the X 1 matrix only.
> ergoStool1 <- ergoStool[ ergoStool$Subject == "1", ]
> model.matrix( effort ~ Type, ergoStool1 ) # X matrix for Subject 1
(Intercept) Type1 Type2 Type3
1
1
-1
-1
-1
2
1
1
-1
-1
3
1
0
2
-1
4
1
0
0
3
Using the Helmert contrasts shown above, the components of represent:

1 Mean level of eort for the four stool types.
2 Dierence between T2 and T1.
3 Twice the dierence between T3 and the average of T1 and T2.
4 Three times the dierence between T4 and the average of T1, T2,
and T3.
Fitting the model in this form with lme produces
> fm1Stool <+
lme(effort ~ Type, data = ergoStool, random = ~ 1 | Subject)
> summary( fm1Stool )
. . .
Random effects:
Formula: ~ 1 | Subject
StdDev:
1.3325
1.1003
Fixed effects: effort ~ Type
Value Std.Error
(Intercept) 10.250
0.48052
Type1
1.944
0.25934
Type2
0.093
0.14973
Type3 -0.343
0.10588
Correlation:
(Intr) T1
T2
T1 0
DF t-value p-value
24 21.331 <.0001
24
7.498 <.0001
24
0.618 0.5421
24 -3.236 0.0035
16
T2
T3

0
0
0
0

Min
Q1
Med
Q3
Max
-1.802 -0.64317 0.057831 0.701 1.6314
Number of Groups: 9
By convention, the coecient corresponding to the rst column in the

X i , which is the column of 1s, is called the intercept. The name originated with models like the analysis of covariance model of 1.4 where a
straight-line model for each group is written in terms of its slope and its
intercept with the y-axis. In those cases, this parameter is the y-intercept.
For the model considered here, the parameter labelled (Intercept) is the
estimate of mean eort for all four types of stools across the population.
The other three parameters, labelled Type1, Type2, and Type3, are described
above. Their individual interpretations are not as important as the collective variability among the stool types they represent. The signicance of
this variability, and hence the overall signicance of the Type term, is assessed with the anova function.
> anova( fm1Stool )
(Intercept)
1
24 455.01 <.0001
Type
3
24
22.36 <.0001
On some occasions we may want to switch to other contrasts that provide

more meaningful parameter estimates for the experiment. For example, if
stool type T1 was a standard stool and we wished to compare the other
types to this standard type, we could use the contrasts called the treatment
contrasts. These contrasts represent the change from the rst level of the
factor to each of the other levels.
One way to cause the treatment contrasts to be used is to reset the
contrasts option. Its value should be a vector of two character strings. The
rst string is the name of the function to use for factors, such as Type, and
the second is the function to use for ordered factors, which are described
in 1.6.
> options( contrasts = c( factor = "contr.treatment",
+
ordered = "contr.poly" ) )
> contrasts( ergoStool$Type )
2 3 4
1 0 0 0
2 1 0 0
3 0 1 0
4 0 0 1
17
> fm2Stool <+

lme(effort ~ Type, data = ergoStool, random = ~ 1 | Subject)
Data: ergoStool
AIC
BIC logLik
133.13 141.93 -60.565
Random effects:
StdDev:
1.3325
1.1003
Fixed effects: effort ~ Type
(Intercept) 8.5556
0.57601 24 14.853 <.0001
Type2 3.8889
0.51868 24
7.498 <.0001
Type3 2.2222
0.51868 24
4.284 0.0003
Type4 0.6667
0.51868 24
1.285 0.2110
Correlation:
(Intr) Type2 Type3
Type2 -0.45
Type3 -0.45
0.50
Type4 -0.45
0.50 0.50
...
> anova( fm2Stool )
(Intercept)
1
24 455.01 <.0001
Type
3
24
22.36 <.0001
Although the individual parameter estimates for the Type factor are different between the two ts, the anova results are the same. The dierence
in the parameter estimates simply reects the fact that dierent contrasts
are being estimated. The similarity of the anova results indicates that the
overall variability attributed to the Type factor does not change. In each
case, the row labelled Type in the analysis of variance table represents a
test of the hypothesis
H0 : 2 = 3 = 4 = 0,
which is equivalent to reducing model (1.6) to
y i = 1 + Z i bi + i ,
i = 1, . . . , 9,
bi N (0, b2 ),
i N (0, 2 I).
This reduced model is invariant under the change in contrasts.

There is a more subtle eect of changing from one form of X j matrix to
anotherthe value of the REML criterion changes. As described in 2.2.5,
when the model parameters are kept at a xed value, a change in the X j
matrices results in change in the value of the restricted likelihood function.
18
Even though we converge to the same variance component estimates

=
1.1003 and
b = 1.3325, the value of the estimation criterion itself changes.
Because the AIC and BIC criteria are based on the REML criterion, they
will also change.
As a consequence, when using REML estimation we can only use likelihood ratio tests or comparisons of AIC or BIC for models with the same
xed-eects structure and the same contrasts for any factors used in the
xed-eects structure.
We can t the cell means parameterization of the model if we add the
term -1 to the formula for the xed eects. This causes the column of 1s
to be removed from the model matrices X j ,
> model.matrix( effort ~ Type - 1, ergoStool1 )
Type1 Type2 Type3 Type4
1
1
0
0
0
2
0
1
0
0
3
0
0
1
0
4
0
0
0
1
and the tted model is now expressed in terms of the mean eort for each
stool type over the population
> fm3Stool <+ lme(effort ~ Type - 1, data = ergoStool, random = ~ 1 | Subject)
Data: ergoStool
AIC
BIC logLik
133.13 141.93 -60.565
Random effects:
StdDev:
1.3325
1.1003
Fixed effects: effort ~ Type - 1
Type1 8.556
0.57601 24 14.853 <.0001
Type2 12.444
0.57601 24 21.604 <.0001
Type3 10.778
0.57601 24 18.711 <.0001
Type4 9.222
0.57601 24 16.010 <.0001
Correlation:
Type1 Type2 Type3
Type2 0.595
Type3 0.595 0.595
Type4 0.595 0.595 0.595
19

Min
Q1
Med
Q3
Max
-1.802 -0.64317 0.057831 0.701 1.6314
Number of Groups: 9
This change in the xed-eects structure does change the anova results.
> anova( fm3Stool )
Type
4
24 130.52 <.0001
The hypothesis being tested by anova for this model is

H0 : 1 = 2 = 3 = 4 = 0,
which is equivalent to reducing model (1.6) to
y i = Z i bi + i ,
i = 1, . . . , 9,
bi N (0, b2 ),
i N (0, 2 I).
That is, the hypothesis H0 completely eliminates the xed-eects parameters from the model so the mean response across the population would be
zero. This hypothesis is not meaningful in the context of this experiment.
To reiterate, some general principles to keep in mind regarding xedeects terms for factors are:
The overall eect of the factor should be assessed with anova, not
by examining the t-values or p-values associated with the xedeects parameters. The anova output does not depend on the choice
of contrasts as long as the intercept term is retained in the model.
Interpretation of the parameter estimates for a xed-eects term depends on the contrasts being used.
For REML estimation, likelihood-ratio tests or comparisons of AIC
or BIC require the same xed-eects structure and the same choice
of contrasts in all models.
The cell means parameters can be estimated by adding -1 to a
model formula but this will usually make the results of anova meaningless.
1.2.2 Examining the Model

As in the rail example, we should examine the tted model both graphically and numerically. The intervals function provides an indication of the
precision of the estimates of the variance components.
20
1
0
-1
10
12
14
Fitted values (Borg scale)
FIGURE 1.7. Standardized residuals versus the tted values for the REML t
of a random-eects model to the data in the ergometric experiment on types of
stools.
> intervals( fm1Stool )

Fixed effects:
lower
(Intercept) 9.25825
Type1 1.40919
Type2 -0.21644
Type3 -0.56111
est.
10.250000
1.944444
0.092593
-0.342593
upper
11.24175
2.47970
0.40162
-0.12408
Random Effects:
Level: Subject
lower
est. upper
sd((Intercept)) 0.74923 1.3325 2.3697
lower
est. upper
0.82894 1.1003 1.4605
We see that is estimated relatively precisely, whereas b can vary by a

factor of about 5, which is a factor of 25 if we express the estimates as
variances.
The plot of the standardized residuals versus the tted values, shown
in Figure 1.7, does not indicate a violation of the assumption of constant
variance for the ij terms.
Figure 1.7 shows the overall behavior of the residuals relative to the tted
values. It may be more informative to examine this behavior according to
the Subject factor or according to the Type factor, which we can do by
providing an explicit formula to the plot method for the tted lme model.
The formula can use functions such as resid or fitted applied to the tted
model. As a shortcut, a "." appearing in the formula is interpreted as the
1.3 Mixed-Eects Models for Replicated, Blocked Designs

8
10
12
14
10
12
21
14
2
1
0
-1
1
0
-1
10
12
14
10
12
14
10
12
14
Fitted values (Borg scale)
FIGURE 1.8. Standardized residuals versus the tted values by Subject for the
REML t of a random-eects model to the data in the ergometric experiment on
types of stools.
tted model object itself. Thus, to plot the standardized, or Pearson,

residuals versus the tted values by Subject, we use
> plot( fm1Stool,
+
form = resid(., type = "p") ~ fitted(.) | Subject,
+
abline = 0 )
The argument abline = 0 adds a horizontal reference line at y = 0 to each

panel. In its more general form, the value of the abline argument should
be a numeric vector of length two giving the intercept and the slope of the
line to be drawn on each panel. Diagnostic plots for assessing the adequacy
of lme ts are discussed in detail in 4.3.
1.3 Mixed-Eects Models for Replicated, Blocked

Designs
In the ergometric experiment on the types of stools, each subject tried each
type of stool once. We say this design is unreplicated because only one observation is available at each combination of experimental conditions. In
other experiments like this it may be feasible to take replicate measurements. For example, the Machines data, described in Milliken and Johnson
(1992, Chapter 23) and shown in Figure 1.9, gives the productivity score
for each of six randomly chosen workers tested on each of three dierent
machine types. Each worker used each machine three times so we have
three replicates at each set of conditions.
In Figure 1.9 we can see that there are strong indications of dierences
between machines and also some indications of dierences between workers.
22
Worker
A
5
3
1
4
2
6
>
C
+++
>>
+
++ +
+ + >>>
>>>
+++ >>
>
+ ++ >>>
+ ++
45
> >>
50
55
60
65
70
Productivity score
FIGURE 1.9. Productivity scores for three types of machines as used by six dierent workers. Scores take into account the number and the quality of components
produced.
We note that there is very little variability in the productivity score for the
same worker using the same machine.
As we did for the experiment on the types of stools, we will model the
subject or Worker factor with random eects and the type or Machine factor
with xed eects. The replications in this experiment will allow us to assess
the presence of interactions between worker and machine. That is, we can
address the question of whether the eect of changing from one type of
machine to another is dierent for dierent workers.
The comparative dotplot in Figure 1.9 allows us to see patterns across
the workers and to see dierences between machines within each worker.
However, the possibility of interactions is not easy to assess in this plot. An
alternative plot, called an interaction plot, shows the potential interactions
more clearly. It is produced by averaging the scores for each worker on each
machine, plotting these averages versus the machine type, and joining the
points for each worker. The function interaction.plot in S creates such a
plot. It is most easily called after attaching the data frame with the data
so the variables in the data frame can be accessed by name.
> attach( Machines ) # make variables in Machines available by name
> interaction.plot( Machine, Worker, score, las = 1) # Figure 1.10
> detach()
# undo the effect of attach( Machines )
(The optional argument las = 1 to interaction.plot alters the label style

on the vertical axis to a more appealing form.)
If there were no interactions between machines and workers, the lines
in the interaction plot would be approximately parallel. The lines in Figure 1.10 do not seem to be parallel, especially relative to the variability
within the replicates that we can see in Figure 1.9. Worker 6 has an unusual pattern compared to the other workers.

Worker
70
mean of score
23
5
3
1
4
2
6
65
60
55
50
45
A
C
Machine
FIGURE 1.10. An interaction plot for the productivity scores for six dierent
workers using three dierent machine types.
1.3.1 Fitting Random Interaction Terms

A model without interactions has the same form as the model for the
ergometric experiment.
yijk = j + bi + ijk ,
bi
i = 1, . . . , 6,
N (0, b2 ),
j = 1, . . . , 3,
2
ijk N (0, ).
k = 1, . . . , 3,
(1.7)
There is a xed eect for each type of machine and a random eect for
each worker. As before, the xed eects for the machines will be re-coded
as an intercept and a set of contrasts when we t this model as
> fm1Machine <+
lme( score ~ Machine, data = Machines, random = ~ 1 | Worker )
> fm1Machine
Data: Machines
Log-restricted-likelihood: -145.23
Fixed: score ~ Machine
(Intercept) Machine1 Machine2
59.65
3.9833
3.3111
Random effects:
Formula: ~ 1 | Worker
StdDev:
5.1466
3.1616
Number of Groups: 6
Because the workers represent a random sample from the population

of interest, any interaction terms modeling dierences between workers in
changing from one machine to another will also be expressed as random
eects. The model incorporating the random interaction terms, bij , i =
24
1, . . . , 6, j = 1, . . . , 3, is
yijk = j + bi + bij + ijk ,
bi
N (0, 12 ),
i = 1, . . . , 6,
bij
N (0, 22 ),
j = 1, . . . , 3,
k = 1, . . . , 3,
2
ijk N (0, ).
This model has random eects at two levels: the eects bi for the worker
and the eects bij for the type of machine within each worker. In a call
to lme we can express this nesting as Worker/Machine in the formula for
the random eects. This expression is read as Worker and Machine within
Worker . We can update the previous model with a new specication for
the random eects.
> fm2Machine <- update( fm1Machine, random = ~ 1 | Worker/Machine )
> fm2Machine
Data: Machines
59.65
3.9833
3.3111
Random effects:
(Intercept)
StdDev:
4.7814
Formula: ~ 1 | Machine %in% Worker
StdDev:
3.7294 0.96158
Number of Groups:
Worker Machine %in% Worker
6
18
This model has produced a value of the REML criterion of 109.64,

which is considerably greater than that of fm1Machine, 145.23. The anova
function, when given two or more arguments representing tted models,
produces likelihood ratio tests comparing the models.
> anova( fm1Machine, fm2Machine )
Model df
AIC
BIC logLik
Test L.Ratio p-value
fm1Machine
1 5 300.46 310.12 -145.23
fm2Machine
2 6 231.27 242.86 -109.64 1 vs 2 71.191 <.0001
The likelihood ratio statistic comparing the more general model (fm2Machine)
to the more specic model (fm2Machine) is huge and the p-value for the test
is essentially zero, so we prefer fm2Machine.
25
The anova function with multiple arguments also reproduces the values
of the AIC and the BIC criteria for each model. As described in 1.1.1,
these criteria can be used to decide which model to prefer. Because the
preference is according to smaller is better, both these criteria show a
strong preference for fm2Machine over fm1Machine.
1.3.2 Unbalanced Data

The Machines data are balanced in that every Worker is tested on every
Machine exactly three times. Milliken and Johnson (1992) analyze this example both as balanced data and as unbalanced data. To obtain the unbalanced data, they randomly deleted ten observations, as indicated in their
Table 23.1. They observe that the software they used to estimate the random eects components (SAS PROC VARCOMP) did not produce sensible
maximum likelihood estimates (although the current version of this software does). The lme function does produce sensible maximum likelihood
estimates or restricted maximum likelihood estimates from the unbalanced
data.
>
## delete selected rows from the Machines data
> MachinesUnbal <- Machines[ -c(2,3,6,8,9,12,19,20,27,33), ]
>
## check that the result is indeed unbalanced
> table(MachinesUnbal$Machine, MachinesUnbal$Worker)
6 2 4 1 3 5
A 3 2 2 1 1 3
B 3 3 3 1 2 2
C 3 3 3 3 3 3
> fm1MachinesU <- lme( score ~ Machine, data = MachinesUnbal,
+
random = ~ 1 | Worker/Machine )
> fm1MachinesU
Data: MachinesUnbal
59.648
3.9812
3.3123
Random effects:
(Intercept)
StdDev:
4.7387
Formula: ~ 1 | Machine %in% Worker
StdDev:
3.7728
0.9332
26
Number of Groups:
Worker Machine %in% Worker
6
18
> intervals( fm1MachinesU )
Fixed effects:
lower
est.
upper
(Intercept) 55.2598 59.6476 64.0353
Machine1 1.5139 3.9812 6.4485
Machine2 1.8940 3.3123 4.7307
Random Effects:
Level: Worker
lower
est. upper
sd((Intercept)) 2.2162 4.7388 10.132
Level: Machine
lower
est. upper
sd((Intercept)) 2.4091 3.7728 5.9084
lower
est. upper
0.71202 0.9332 1.2231
The estimates of the standard deviations and the condence intervals

on these parameters look reasonable when compared to those from the
full data set. The techniques used in lme for parameter estimation do not
depend on the data being balanced.
However, for either balanced or unbalanced data we must have sucient
information in the data to be able to estimate the variance components and
the xed-eects parameters. We can t a model with random interaction
eects to the Machines data because there are replications. If we tried to
t a nested model to unreplicated data, such as the ergoStool data, it may
appear that we are successful until we examine the intervals on the variance
components.
> fm4Stool <- lme( effort ~ Type, ergoStool, ~ 1 | Subject/Type )
> intervals( fm4Stool )
Fixed effects:
lower
(Intercept) 9.25825
Type1 1.40919
Type2 -0.21644
Type3 -0.56111
est.
10.250000
1.944444
0.092593
-0.342593
upper
11.24175
2.47970
0.40162
-0.12408
27
Random Effects:
Level: Subject
lower
est. upper
sd((Intercept)) 0.74952 1.3325 2.3688
Level: Type
lower
est. upper
sd((Intercept)) 0.05386 0.99958 18.551
lower
est. upper
4.3603e-07 0.45988 485050
Apparently the standard deviations 2 and could vary over twelve orders
of magnitude!
If we write this model for these data, taking into account that each
subject only tries each type of stool once, we would have
yij = i + bj + bij + ij , i = 1, . . . , 3, j = 1, . . . , 6,
bj N (0, 12 ), bij N (0, 22 ), ij N (0, 2 ).
We can see that the bij are totally confounded with the ij so we cannot
estimate separate standard deviations for these two random terms. In fact,
the estimates reported for and 2 in this model give a combined variance
that corresponds to
2 from fm1Stool.
> (fm1Stool$sigma)^2
[1] 1.2106
> (fm4Stool$sigma)^2 + 0.79621^2
[1] 1.2107
The lesson here is that it is always a good idea to check the condence
intervals on the variance components after tting a model. Having abnormally wide intervals usually indicates problems with the model denition.
In particular, a model with nested interaction terms can only be t when
there are replications available in the data.
1.3.3 More General Models for the Random Interaction

Eects
In the model (1.3.1), the random interaction terms all have the same variance 22 . Furthermore, these random interactions are assumed to be independent of one another, even within the same subject. A more general
model could treat the random interactions for each subject as a vector and
allow the variancecovariance matrix for that vector to be estimated from
the set of all positive-denite matrices.
To express this model we return to the matrix/vector representation used
in (1.2.1). We dene y i to be the entire response vector for the ith subject,
28
to be the three-dimensional vector of xed-eects parameters for the

population, bi to be the three-dimensional vector of random eects for the
ith subject, X i to be the 9 3 xed-eects design matrix for subject i, and
Z i to be the 9 3 random-eects design matrix for subject i. The general
form of the model is then
y i = X i + Z i bi + i ,
bi N (0, ),
i = 1, . . . , 6,
i N (0, 2 I),
where is a positive-denite, symmetric 3 3 matrix.

To be more specic we must dene the matrices X i and Z i or, equivalently, dene the formulae that generate these matrices as model matrices.
As discussed in 1.2.1, in the xed-eects we generally use a formula that
creates X i with a single column of 1s and two columns of contrasts. We
could do the same for the Z i but, because the random eects are assumed
to have an expected value of 0 anyway, it is often more informative to use
a formula such as ~ Machine - 1 that removes the intercept column.
Sample model matrices, evaluated on the Worker1s data only, are
> Machine1 <- Machines[ Machines$Worker == "1", ]
> model.matrix( score ~ Machine, Machine1 )
# fixed-effects X_i
1
1
-1
-1
2
1
-1
-1
3
1
-1
-1
19
1
1
-1
20
1
1
-1
21
1
1
-1
37
1
0
2
38
1
0
2
39
1
0
2
> model.matrix( ~ Machine - 1, Machine1 )
# random-effects Z_i
MachineA MachineB MachineC
1
1
0
0
2
1
0
0
3
1
0
0
19
0
1
0
20
0
1
0
21
0
1
0
37
0
0
1
38
0
0
1
39
0
0
1
The tted model using this formulation is

> fm3Machine <- update( fm1Machine, random = ~Machine - 1 |Worker)
> summary( fm3Machine )
29
Data: Machines
AIC
BIC logLik
231.89 251.21 -105.95
Random effects:
Formula: ~ Machine - 1 | Worker
Structure: General positive-definite
StdDev
Corr
MachineA 4.07928 MachnA MachnB
MachineB 8.62529 0.803
MachineC 4.38948 0.623 0.771
Residual 0.96158
Fixed effects: score ~ Machine
(Intercept) 59.650
2.1447 46 27.813 <.0001
Machine1 3.983
1.2104 46
3.291 0.0019
Machine2 3.311
0.5491 46
6.030 <.0001
Correlation:
(Intr) Machn1
Machine1 0.811
Machine2 -0.540 -0.453
Min
Q1
Med
Q3
Max
-2.3935 -0.51378 0.026908 0.47245 2.5334
Number of Groups: 6
This model can be compared to the previous two models using the multiargument form of anova.
> anova( fm1Machine, fm2Machine, fm3Machine
Model df
AIC
BIC logLik
fm1Machine
1 5 300.46 310.12 -145.23
fm2Machine
2 6 231.27 242.86 -109.64 1
fm3Machine
3 10 231.89 251.21 -105.95 2
)
vs 2
vs 3
71.191
7.376
<.0001
0.1173
Because the p-value for the test comparing models 2 and 3 is about 12%,
we would conclude that the t fm3Machine is not signicantly better than
fm2Machine, taking into account the fact that fm3Maching requires four additional parameters in the model.
The AIC criterion is nearly the same for models 2 and 3, indicating that
there is no strong preference between these models. The BIC criterion does
indicate a strong preference for model 2 relative to model 3. In general BIC
puts a heavier penalty than does AIC on having more parameters in the
model. Because there are a total of ten parameters in model 3 compared
30
to six parameters in model 2, the BIC criterion will tend to prefer model 2
unless model 3 provides a substantially better t.
1.4 An Analysis of Covariance Model

Traditionally, the term analysis of variance has been applied to models for
a continuous response as it relates to various classication factors for the
observations. The model for the rails example described in 1.1
yij = + bi + ij ,
bi
i = 1, . . . , M,
N (0, b2 ),
j = 1, . . . , ni ,
ij N (0, 2 )
is an example of an analysis of variance model with random-eects terms.

A linear regression model, such as
yi = 1 + 2 xi + i ,
i = 1, . . . , N,
i N (0, 2 ),
relates a continuous response (the yi ) to one or more continuous covariates

(the xi ).
The term analysis of covariance designates a type of model that relates
a continuous response to both a classication factor and to a continuous
covariate. If yij is the jth observation in the ith group of data and xij is
the corresponding value of the covariate, an analysis of covariance model
with a random eect for the intercept would be
yij = 1 + bi + 2 xij + ij ,
bi
N (0, b2 ),
i = 1, . . . , M,
j = 1, . . . , ni ,
2
ij N (0, ).
(1.8)
This model combines a random-eects analysis of variance model with a

linear regression model.
1.4.1 Modeling Simple Linear Growth Curves

A common application of random-eects analysis of covariance models is in
modeling growth curve datathe results on dierent subjects of repeated
measurements of some characteristic over time. The terms repeated measures and longitudinal data are also applied to such data.
A classic example of such data, given in Pottho and Roy (1964), is a set
of measurements of the distance from the pituitary gland to the pterygomaxillary ssure taken every two years from 8 years of age until 14 years of
age on a sample of 27 children16 males and 11 females. The data, available as the S object Orthodont and shown in Figure 1.11, were collected by
orthodontists from x-rays of the childrens skulls. The pituitary gland and
the pterygomaxillary ssure are two easily located points on these x-rays.

8
F03
10
12
31
14
F04
F11
30
25
Distance from pituitary to pterygomaxillary fissure (mm)
20
F10
F09
F06
F01
F05
F07
F02
F08
M13
M14
M09
M15
M06
M04
M01
M10
30
25
20
30
25
20
M16
M05
M02
M11
M07
M08
M03
M12
30
25
20
10
12
14
10
12
14
10
12
14
10
12
14
Age (yr)
FIGURE 1.11. Distance from the pituitary to the pterygomaxillary ssure versus
age for a sample of 16 boys (subjects M01 to M16) and 11 girls (subjects F01 to
F11). The aspect ratio for the panels has been chosen to facilitate comparison of
the slope of the lines.
From Figure 1.11 it appears that there are qualitative dierences between
boys and girls in their growth patterns for this measurement. In Chapter 4
we will model some of these dierences, but for now it is easier to restrict
our modeling to the data from the female subjects only. To extract the
data for the females only we rst check on the names of the variables in
the Orthodont object, then check for the names of the levels of the variables
32
Sex, then extract only those rows for which the Sex variable has the value
"Female".
> names( Orthodont )
[1] "distance" "age"
"Subject" "Sex"
> levels( Orthodont$Sex )
[1] "Male"
"Female"
> OrthoFem <- Orthodont[ Orthodont$Sex == "Female", ]
Figure 1.11 indicates that, for most of the female subjects, the orthodontic measurement increases with age and that the growth is approximately
linear over this range of ages. It appears that the intercepts, and possibly
the slopes, of these growth curves may dier between girls. For example,
subject 10 has considerably smaller measurements than does subject 11,
and the growth rate for subjects 2 and 3 is considerably greater than that
for subjects 5 and 8.
To explore this potential linear relationship further, we t separate linear
regression models for each girl using the lmList function.
> fm1OrthF.lis <- lmList( distance ~ age, data = OrthoFem )
> coef( fm1OrthF.lis )
(Intercept)
age
F10
13.55 0.450
F09
18.10 0.275
F06
17.00 0.375
F01
17.25 0.375
F05
19.60 0.275
F08
21.45 0.175
F07
16.95 0.550
F02
14.20 0.800
F03
14.40 0.850
F04
19.65 0.475
F11
18.95 0.675
The function coef is a generic function (Chambers and Hastie, 1992,

Appendix A) that extracts the estimated coecients from a tted model
object. For an lmList object the coecients are returned as a matrix with
one row for each of the groups of observations.
We might wish to consider whether we need to allow dierent slopes for
each girl. There are formal statistical tests to assess this and we will discuss
them later. For now we can proceed informally and examine individual
condence intervals on the parameters. As we have seen, the intervals
function is used to create condence intervals on the parameters in an
object representing a tted model.
> intervals( fm1OrthF.lis )
, , (Intercept)
lower est. upper
F10 10.071 13.55 17.029
Subject
(Intercept)
F11
F04
F03
F08
F02
F07
F05
F01
F06
F09
F10
|
|
|
|
10
15
|
|
25
0.0
|
|
|
|
20
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
|
|
age
|
|
33
|
|
0.4
|
0.8
1.2
FIGURE 1.12. Comparison of 95% condence intervals on the coecients of

simple linear regression models tted to the orthodontic growth curve data for
the female subjects.
F09
F06
F01
F05
F07
F02
F08
F03
F04
F11
14.621
13.521
13.771
16.121
13.471
10.721
17.971
10.921
16.171
15.471
, , age
lower
F10 0.1401
F09 -0.0349
F06 0.0651
F01 0.0651
F05 -0.0349
F07 0.2401
F02 0.4901
F08 -0.1349
F03 0.5401
F04 0.1651
F11 0.3651
18.10
17.00
17.25
19.60
16.95
14.20
21.45
14.40
19.65
18.95
est.
0.450
0.275
0.375
0.375
0.275
0.550
0.800
0.175
0.850
0.475
0.675
21.579
20.479
20.729
23.079
20.429
17.679
24.929
17.879
23.129
22.429
upper
0.7599
0.5849
0.6849
0.6849
0.5849
0.8599
1.1099
0.4849
1.1599
0.7849
0.9849
As often happens, displaying the intervals as a table of numbers is not

very informative. We nd it much more eective to plot these intervals
using
> plot( intervals ( fm1OrthF.lis ) )
Figure 1.12 is of interest as much for what it does not show as for what
it does show. First, consider what the gure does show. We notice that
the intervals for the intercepts are all the same width, as are the intervals
34
Subject
(Intercept)
F11
F04
F03
F08
F02
F07
F05
F01
F06
F09
F10
| | |
18
I(age - 11)
| | |
|
| | |
|
| | |
| | |
|
|
| | |
| | |
|
|
| | |
|
|
| | |
|
|
| | |
|
|
| | |
|
|
20
22
24
26
0.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
0.4
|
0.8
1.2
FIGURE 1.13. Comparison of 95% condence intervals on the coecients of

simple linear regression models tted to the centered orthodontic growth curve
data for the female subjects.
for the slope with respect to age. This is a consequence of having balanced
data; that is, all the subjects were observed the same number of times and
at the same ages. We also notice that there is considerable overlap in the
set of intervals for the slope with respect to age. It may be feasible to use
a model with a common slope.
The surprising thing about Figure 1.12 is that it does not show the
substantial dierences in the intercepts that Figure 1.11 would lead us to
expect. Furthermore, even though we have ordered the groups from the
one with the smallest average distance (subject F10) to the one with the
largest average distance (subject F11), this ordering is not reected in the
intercepts. Finally, we see that the pattern across subjects in the intervals
for the intercepts is nearly a reection of the pattern in the intervals for
the slopes.
Those with experience analyzing regression models may already have
guessed why this reection of the pattern occurs. It occurs because all the
data were collected between age 8 and age 14, but the intercept represents
a distance at age 0. The extrapolation back to age 0 will result in a high
negative correlation (about 0.98) between the estimates of the slopes and
their corresponding intercept estimate.
We will remove this correlation if we center the data. In this case, we
would t the distance as a linear function of age - 11 so the two coecients
being estimated are the distance at 11 years of age and the slope or growth
rate. If we t this revised model and plot the condence intervals
> fm2OrthF.lis <- update( fm1OrthF.lis, distance ~ I( age - 11 ) )
> plot( intervals( fm2OrthF.lis ) )
then these intervals (Figure 1.13) show the expected trend in the (Intercept)
term, which now represents the tted distance at 11 years.
To continue with the analysis of these data we could t a regression model
to the centered data with a common growth rate but separate intercepts for
each girl. Before doing that we should consider what we could infer from
35
such a model. We could use such a model to make inferences about the
growth rate for this sample of girls. Also, we could make inferences about
the expected distance for each girl at 11 years of age. Using combinations
of the parameters we could make inferences about the expected distance
for each of these girls at other ages. The key point is that we are in some
ways restricting ourselves to the distances that we have or could observe
on these particular girls
By tting a mixed-eects model to these data we allow ourselves to make
inferences about the xed eects, which represent average characteristics of
the population represented by these subjects, and the variability amongst
subjects. A call to lme to t linear growth curves with common slopes but
randomly distributed shifts to the girls orthodontic data is
> fm1OrthF <+
lme( distance ~ age, data = OrthoFem, random = ~ 1 | Subject )
> summary( fm1OrthF )
Data: OrthoFem
AIC
BIC logLik
149.22 156.17 -70.609
Random effects:
StdDev:
2.0685 0.78003
Fixed effects: distance ~ age
(Intercept) 17.373
0.85874 32 20.230 <.0001
age 0.480
0.05259 32
9.119 <.0001
Correlation:
(Intr)
age -0.674
Min
Q1
Med
Q3
Max
-2.2736 -0.70902 0.17282 0.41221 1.6325
Number of Groups: 11
We could also t a model with the formula distance ~ I(age - 11) but,
because of the requirement of a common slope, for model building purposes
the properties of the centered model are essentially equivalent to the uncentered model. Using the uncentered model makes it easier to compare
with other models described below.
36
The model being t would be expressed in matrix notation as

y i = X i + Z i bi + i ,
bi N (0, ),
i = 1, . . . , 11,
i N (0, 2 I),
with matrices
X 1 = = X 11
1 8
1 10
=
1 12 ,
1 14
Z 1 = = Z 11

1
1
=
1 .
1
The two-dimensional xed-eects vector consists of the mean intercept,

1 , for the population and the common slope or growth rate, 2 . The onedimensional random-eects vectors, bi , i = 1, . . . , 11, describe a shift in the
intercept for each subject. Because there is a common growth rate, these
shifts are preserved for all values of age. The matrix = b2 will be a 1 1
matrix in this case. It represents the variance of the measurements in the
population at a xed value of age.
The REML estimates for the parameters are
b = 2.0685,
= 0.78003,
1 = 17.373,
2 = 0.480.
To obtain the maximum likelihood estimates we use method = "ML".

> fm1OrthFM <- update( fm1OrthF, method = "ML" )
> summary( fm1OrthFM )
Data: OrthoFem
AIC
BIC logLik
146.03 153.17 -69.015
Random effects:
StdDev:
1.9699 0.76812
Fixed effects: distance ~ age
(Intercept) 17.373
0.85063 32 20.423 <.0001
age 0.480
0.05301 32
9.047 <.0001
Correlation:
(Intr)
age -0.685
Min
Q1
Med
Q3
Max
-2.3056 -0.71924 0.17636 0.4258 1.6689
37
Notice that, to the accuracy printed here, the estimates of the xed-eects
parameters are the same for ML and REML. The ML estimates of the
= 0.76812 are smaller than the
standard deviations,
b = 1.9699 and
corresponding REML estimates. This is to be expectedthe REML criterion was created to compensate for the downward bias of the maximum
likelihood estimates of variance components, so it should produce larger
estimates.
We have made the assumption of a common slope or growth rate for all
the subjects. To test this we can t a model with random eects for both
the intercept and the slope.
> fm2OrthF <- update( fm1OrthF, random = ~ age | Subject )
The predictions from this model are shown in Figure 1.14. We compare the
two models with the anova function.
> anova( fm1OrthF, fm2OrthF )
Model df
AIC
BIC logLik
fm1OrthF
1 4 149.22 156.17 -70.609
fm2OrthF
2 6 149.43 159.85 -68.714 1 vs 2 3.7896 0.1503
Because the p-value for the second model versus the rst is about 15%, we
conclude that the simpler model, fm1OrthF, is adequate.
1.4.2 Predictions of the Response and the Random Eects

The derivation of predicted values for the response and for the random
eects in the linear mixed-eects model is described in 2.5. We can extract
the best linear unbiased predictions (BLUPs) of the random eects from the
tted model with the random.effects function.
> random.effects( fm1OrthF )
(Intercept)
F10
-4.005329
F09
-1.470449
F06
-1.470449
F01
-1.229032
F05
-0.021947
F07
0.340179
F02
0.340179
F08
0.702304
F03
1.064430
F04
2.150807
F11
3.599309
38

8
F02
10
12
14
F08
10
F03
12
14
F04
F11
28
26
24
22
20
18
F10
F09
F06
F01
F05
F07
28
26
24
22
20
18
10
12
14
10
12
14
10
12
14
Age (yr)
FIGURE 1.14. Original data and tted linear relationships from a mixed-eects
model for the girls orthodontic data. This model incorporates random eects for
both the slope and the intercept.
The shorter name ranef is a synonym for random.effects.

> ranef( fm1OrthFM )
(Intercept)
F10
-3.995835
F09
-1.466964
F06
-1.466964
F01
-1.226119
F05
-0.021895
F07
0.339372
F02
0.339372
F08
0.700640
F03
1.061907

F04
F11
39
2.145709
3.590778
The coefficients function (or its shorter form coef) is used to extract
the coecients of the tted lines for each subject. For the tted model
1 + bi and the slope
fm1OrthF the intercept of the tted line for subject i is

is 2 .
> coef( fm1OrthF )
(Intercept)
age
F10
13.367 0.47955
F09
15.902 0.47955
F06
15.902 0.47955
F01
16.144 0.47955
F05
17.351 0.47955
F07
17.713 0.47955
F02
17.713 0.47955
F08
18.075 0.47955
F03
18.437 0.47955
F04
19.524 0.47955
F11
20.972 0.47955
Looking back at the BLUPs of the random eects for the ML and REML
ts, we can see that they are very similar. The same will be true of the
coecients for each subject and hence for the tted lines themselves. To
show this we can plot either the estimated BLUPs or the estimated coecients. The compareFits function is helpful here because it allows us to put
both sets of coecients on the same panels.
> plot(compareFits(coef(fm1OrthF), coef(fm1OrthFM)))
# Figure 1.15
In Figure 1.15 each line corresponds to one subject. In the left panel the
estimated intercepts from the REML t are shown as open circles while
those from the ML t are shown as +s. The two estimates for each subject
are essentially identical. In the right panel the estimates for the coecient
with respect to age are shown. Because there is no random eect associated
with this coecient, the estimates do not vary between subjects. Again, the
ML estimates and the REML estimates are essentially identical.
We may also want to examine the predictions for each subject from the
tted model. The augPred function produces predictions of the response for
each of the groups over the observed range of the covariate (i.e. the range
814 for age). These predictions are augmented with the original data to
produce a plot of the predicted lines for each subject superposed on the
original data as in Figure 1.16.
> plot( augPred(fm1OrthF), aspect = "xy", grid = T )
# Fig. 1.16
Further diagnostic plots, such as plots of residuals versus the tted values
by subject (not shown), did not indicate any serious deciencies in this
40

+
coef(fm1OrthF)
coef(fm1OrthFM)
(Intercept)
F11
F04
F03
F08
F02
F07
F05
F01
F06
F09
F10
age
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
14
16
18
20
0.40
0.45
0.50
0.55
FIGURE 1.15. A comparison of the coecients of the tted lines for each female
subject in the orthodontic example. The two sets of coecients are from the
restricted maximum likelihood t (fm1OrthF) and the maximum likelihood t
(fm1OrthFM).
model. We will return to this data set in Chapter 4 where we will t a

combined model to the data for both the males and the females. This
allows us to check for sex-related dierences in the growth patterns.
1.5 Models for Nested Classication Factors

In the Machines example of 1.3 we introduced the concept of nested random
eects to model an interaction between a xed-eects factor and a randomeects factor. Nested random-eects terms are also used when we have
nested classication factors.
Data from an experiment on the pixel intensity in computerized tomography (CT) scans, available as the object Pixel, are shown in Figure 1.17
and are described in Appendix A.24. The experimenters injected each of
ten dogs with a dye contrast then recorded the mean pixel intensities from
CT scans of the right and left lymph nodes in the axillary region on several
occasions up to 21 days post injection.
Each observation is classied according to the Dog and the Side of the
Dog on which it was made. The nature of the experiment is such that the
left and right sides are expected to be dierent, but the dierence is not
expected to be systematic in terms of left and right. That is, for one dog
the left side may have greater pixel intensities than the right, while for
another dog the opposite may be true. Thus Dog and Side are considered
to be nested classication factors. We will associate random-eects terms
with the Dog factor, and with the Side factor nested within Dog.
Figure 1.17 indicates that the intensities generally increase then decrease
over time, reaching a peak after about 10 days. There is, however, consid-

8
F02
10
12
14
F08
10
F03
12
41
14
F04
F11
28
26
24
22
20
18
F10
F09
F06
F01
F05
F07
28
26
24
22
20
18
10
12
14
10
12
14
10
12
14
Age (yr)
FIGURE 1.16. Original data and tted growth curves for each female subject
in the orthodontic example. The tted curves are from a restricted maximum
likelihood t of the analysis of covariance model.
erable variability between dogs in this pattern. Within the same dog, the
left and the right side generally follow the same pattern over time but often
with a vertical shift between sides.
We will start with a quadratic model with respect to the day covariate
so we can model the pattern of reaching a peak. We use random-eects for
both the intercept and the linear term at the Dog level and a single random
eect for the intercept at the Side within Dog level. This allows the overall
pattern to vary between dogs in terms of the location of the peak, but not
in terms of the curvature at the peak. The only dierence between sides
for the same dog will be a shift in the intercept.
42

0
Pixel intensity
10
15
20
10
15
20
10
1160
1140
1120
1100
1080
1060
1040
1160
1140
1120
1100
1080
1060
1040
0
10
15
20
10
15
20
10
15
20
Time post injection (days)

FIGURE 1.17. Pixel intensity on CT scans over time for lymph nodes on the left
and the right sides of 10 dogs.
> fm1Pixel <- lme( pixel ~ day + day^2, data = Pixel,

+
random = list( Dog = ~ day, Side = ~ 1 ) )
> intervals( fm1Pixel )
Fixed effects:
(Intercept)
day
I(day^2)
lower
1053.0968
4.3797
-0.4349
est.
1073.33914
6.12960
-0.36735
upper
1093.5814
7.8795
-0.2998
Random Effects:
Level: Dog
lower
est.
upper
sd((Intercept)) 15.92849 28.36994 50.52918
sd(day)
1.08085
1.84375 3.14514
cor((Intercept),day) -0.89452 -0.55472 0.19138
Level: Side
lower
est. upper
sd((Intercept)) 10.417 16.824 27.173
lower
est. upper
7.6345 8.9896 10.585
> plot( augPred( fm1Pixel ) )
If we write the pixel intensity of the jth sides on the ith dog at the kth
occasion as yijk i = 1, . . . , 10; j = 1, 2; k = 1, . . . , nij , and the time of the

0
8/R
10
15
20
9/L
9/R
10
15
43
20
10/L
10/R
1150
1100
1050
1000
6/L
6/R
7/L
7/R
8/L
3/R
4/L
4/R
5/L
5/R
Pixel intensity
1150
1100
1050
1000
1150
1100
1050
1000
1/L
1/R
2/L
2/R
3/L
1150
1100
1050
1000
0
10
15
20
10
15
20
10
15
20

FIGURE 1.18. The tted curves from a quadratic model t to the pixel intensity
data. The model includes random eects in the intercept and the linear term for
each dog, and a random eect in the intercept for each side of each dog.
kth scan on the ith dog as dik , the model being t can be expressed as
yijk = 1 + 2 dik + 3 d2ik + bi,1 + bi,2 dik + bij + ijk ,
i = 1, . . . , 10, j = 1, 2, k = 1, . . . , nij .
(1.9)
To describe the variance and covariance terms in the model, we consider

the nij -vector y ij of intensities measured on side j = 1, 2 within dog i =
1, . . . , 10. In this experiment nij , the number of observations on side j of
dog i, does not depend on j but does depend on i. For example, dog 9
was scanned on only two occasions but dogs 3 and 4 were each scanned on
seven occasions. The model can be represented in terms of design matrices
X ij for the xed eects, Z ij for the random eects for side j within dog
i, and Z i,j for the random eects for dog i on the measurements for side j
within dog i.
Because both sides are scanned at the same times these matrices depend
on i but not on j. For example, because dog 8 was scanned on days 4, 6,
10, and 14,
1 4 16
1 4
1
1 6 36
1 6
1
X8 1 = X8 2 =
1 10 100 , Z 8,1 = Z 8,2 = 1 10 , Z 8 1 = Z 8 2 = 1 .
1 14 196
1 14
1
44
As before, is the vector of xed eects. In this case, is three-dimensional.

The two-dimensional vector of random eects for dog i is written bi and
the one-dimension vector of random eects for side j within dog i is written
bij . Model (1.9) can then be expressed as
y ij = X ij + Z i,j bi + Z ij bij + ij ,
bi N (0, 1 ),
bij N (0, 22 ),
ij N (0, 2 I).
The parameters in this model are , 1 , 22 , and 2 . The summary and

intervals functions express the estimates of the variance components as
standard deviations and correlations, not as variances. We can use the
VarCorr function to examine them on both the standard deviation scale
and the variance scale.
> VarCorr( fm1Pixel )
Variance StdDev
Corr
Dog = pdSymm(~ day)
(Intercept) 804.8535
28.3699 (Intr)
day
3.3994
1.8437 -0.555
Side = pdSymm(~ 1)
(Intercept) 283.0551
16.8242
Residual 80.8131
8.9896
1.5.1 Model Building for Multilevel Models

As when modeling data with a single level of random eects, we should
evaluate whether the xed-eects structure and the random-eects structure are adequate to describe the observed data. We should also check if
we have incorporated unnecessary terms in the model.
The summary table for the xed-eects terms
> summary( fm1Pixel )
...
Fixed effects: pixel ~ day + day^2
(Intercept) 1073.3
10.172 80 105.52 <.0001
day
6.1
0.879 80
6.97 <.0001
I(day^2)
-0.4
0.034 80 -10.82 <.0001
...
indicates that the quadratic term is highly signicant. In a polynomial

model like this we would generally retain the linear term and the intercept
term if we retain the quadratic term. Thus, we will accept the xed-eects
model as it is and go on to examine the random-eects terms.
The rst question to address is whether there is a need to have a random
eect for each Side within each Dog. We can t the previous model without
the random eect for Side and compare the two ts with anova.
1.6 A Split-Plot Experiment
45
> fm2Pixel <- update( fm1Pixel, random = ~ day | Dog)

> anova( fm1Pixel, fm2Pixel )
Model df
AIC
BIC logLik
fm1Pixel
1 8 841.21 861.97 -412.61
fm2Pixel
2 7 884.52 902.69 -435.26 1 vs 2 45.309 <.0001
The p-value is extremely small indicating that the more general model,
fm1Pixel, is denitely superior. The AIC and BIC values conrm this.
We can also check if the random eect for day at the Dog level is warranted. If we eliminate this term the only random eects will be a random
eect for the intercept for each dog and for each side of each dog. We t
this model and compare it to fm1Pixel with
> fm3Pixel <- update( fm1Pixel, random = ~ 1 | Dog/Side )
> anova( fm1Pixel, fm3Pixel )
Model df
AIC
BIC logLik
fm1Pixel
1 8 841.21 861.97 -412.61
fm3Pixel
2 6 876.84 892.41 -432.42 1 vs 2 39.629 <.0001
Again, the likelihood-ratio test, and the AIC and BIC criteria, all strongly
favor the more general model, fm1Pixel.
Earlier we stated that there does not appear to be a systematic dierence
between the left and the right sides of the dogs. For some dogs the left side
produces higher pixel densities while for other dogs the right side does. We
can check that this indeed is the case by adding a term for Side to the xed
eects.
> fm4Pixel <- update( fm1Pixel, pixel ~ day + day^2 + Side )
> summary( fm4Pixel )
...
Fixed effects: pixel ~ day + I(day^2) + Side
(Intercept) 1073.3
10.171 80 105.53 <.0001
day
6.1
0.879 80
6.97 <.0001
I(day^2)
-0.4
0.034 80 -10.83 <.0001
Side
-4.6
3.813 9
-1.21 0.2576
...
With a p-value of over 25% the xed-eects term for Side would not be
considered signicant.
Finally, we would examine residual plots such as Figure 1.19 for deciencies in the model. There are no alarming patterns in this gure.

Multiple nested levels of random eects are also used in the analysis of
split-plot experiments such as that represented by the Oats data, shown in
46
1040
5
1080
1120
1040
1080
1120
9
2
1
0
-1
-2
10
-3
2
1
0
-1
-2
-3
1040
1080
1120
1040
1080
1120
1040
1080
1120
Fitted values
FIGURE 1.19. Standardized residuals versus tted values by dog for a multilevel
mixed-eects model t to the pixel data..
Figure 1.20. As described in Appendix A.15, the treatment structure in this

experiment was a 3 4 full factorial, with three varieties of oats and four
nitrogen concentrations. The term full factorial means that every variety
was used with every nitrogen concentration.
The agricultural plots for this experiment were grouped into six blocks,
each with three plots. Each plot was subdivided into four subplots. The varieties were randomly assigned to the plots within each block. The nitrogen
concentrations were randomly assigned to the subplots within each plot.
Physically, there are three levels of grouping of the experimental units:
block, plot, and subplot. Because the treatments are randomly assigned
at each level of grouping, we may be tempted to associate random eects
with each level. However, because there is only one yield recorded for each
subplot we cannot do this as we would saturate the model with random
eects. We use a random intercept at each of the block and the whole plot
levels.
Generally, we begin modeling a split-plot experiment using xed eects
for each of the experimental factors and for their interaction. For this experiment the nitro factor is recorded as a numeric variable. If we wish to
allow general patterns in the dependencies of yield on nitro we should
coerce it to a factor using, say, factor(nitro). In this particular example, there is a natural ordering of the levels of nitrogen applied so it makes
sense to coerce nitro to an ordered factor using ordered(nitro). As the
name implies, an ordered factor is a factor for which there is a natural
ordering of the levels. One consequence of using an ordered factor instead
of a factor is that the default contrasts for an ordered factor are orthogonal polynomial contrasts. The rst contrast estimates the linear trend, the
second estimates the quadratic eect orthogonal to the linear term, and so
on.

Golden Rain
Marvellous
0.0
IV
0.2
0.4
47
Victory
0.6
II
160
140
120
Yield (bushels/acre)
100
80
60
VI
III
160
140
120
100
80
60
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
Nitrogen concentration (cwt/acre)
FIGURE 1.20. Yield in bushels/acre of three dierent varieties of oats at four different concentrations of nitrogen (hundred weight/acre). The experimental units
were arranged into six blocks, each with three whole-plots subdivided into four
subplots. One variety of oats was used in each whole-plot with all four concentrations of nitrogen, one concentration in each of the four subplots. The panels
correspond to the blocks.
The model with xed eects for both experimental factors and for their
interaction and with random eects for both the Block factor and the
Variety (whole-plot) factor is t with
> fm1Oats <- lme( yield ~ ordered(nitro) * Variety, data = Oats,
+
random = ~ 1 | Block/Variety )
> anova( fm1Oats )
(Intercept)
1
45 245.15 <.0001
ordered(nitro)
3
45
37.69 <.0001
Variety
2
10
1.49 0.2724
ordered(nitro):Variety
6
45
0.30 0.9322
The anova results indicate that nitro is a signicant factor, but that neither
Variety nor the interaction between Variety and nitro are signicant.
48
If we drop the interaction term and ret, we obtain essentially the same
results for the two main eects, Variety and nitro.
> fm2Oats <- update( fm1Oats, yield ~ ordered(nitro) + Variety )
> anova( fm2Oats )
(Intercept)
1
51 245.14 <.0001
ordered(nitro)
3
Variety
2
> summary( fm2Oats )
...
Random effects:
Formula: ~ 1 | Block
(Intercept)
StdDev:
14.645
51
10
41.05
1.49
<.0001
0.2724
Formula: ~ 1 | Variety %in% Block

StdDev:
10.473
12.75
Fixed effects: yield ~ ordered(nitro) + Variety
(Intercept) 103.97
6.6406 51 15.657 <.0001
ordered(nitro).L
32.94
3.0052 51 10.963 <.0001
ordered(nitro).Q
-5.17
3.0052 51 -1.719 0.0916
ordered(nitro).C
-0.45
3.0052 51 -0.149 0.8823
Variety1
2.65
3.5395 10
0.748 0.4720
Variety2
-3.17
2.0435 10 -1.553 0.1515
...
In this model there is a random eect for Variety %in% Block as well as
a xed eect for Variety. These terms model dierent characteristics of the
response. The random eects term, as a nested random eect, is allowing
for dierent intercepts at the level of plots within blocks. The fact that
each plot is planted with one variety means that we can use the Variety
factor to indicate the plot as long as we have Variety nested within Block.
As seen in Figure 1.20 the yields in one of the plots within a block may be
greater than those on another plot in the same block for all levels of nitro.
For example, in block III the plot that was planted with the Marvellous
variety had greater yields than the other two plots at each level of nitro.
The random eect at the level of Variety %in% Block allows shifts like this
that may be related to the fertility of the soil in that plot, for example.
On the other hand, the xed-eects term for Variety is used to model a
systematic dierence in the yields that would be due to the variety of oats
planted in the plot. There do not appear to be such systematic dierences.
For example, even though the plot planted with the Marvellous variety is
49
the highest yielding plot in block III, the Marvellous plot is one of the
lowest yielding in block V.
Because the xed eect for Variety and the random eect for Variety
%in% Block are modeling dierent types of behavior, it makes sense to remove the xed eect while retaining the random eect
> fm3Oats <- update( fm1Oats, yield ~ ordered( nitro ) )
...
Random effects:
(Intercept)
StdDev:
14.506
StdDev:
11.039
12.75
Fixed effects: yield ~ ordered(nitro)
(Intercept) 103.97
6.6406 51 15.657 <.0001
ordered(nitro).L
32.94
3.0052 51 10.963 <.0001
ordered(nitro).Q
-5.17
3.0052 51 -1.719 0.0916
ordered(nitro).C
-0.45
3.0052 51 -0.149 0.8823
We see that the estimates for the random-eects variances and the xedeects for nitro have changed very little, if at all.
We can now examine the eect of nitrogen in more detail. We notice that
the linear term, ordered(nitro).L, is highly signicant, but the quadratic
and cubic terms (.Q and .C extensions) are not. To remove the cubic and
quadratic terms in the model, we simply revert to using nitro as a numeric
variable.
> fm4Oats <+
lme( yield ~ nitro, data = Oats, random = ~ 1 | Block/Variety )
. . .
Random effects:
(Intercept)
StdDev:
14.506
StdDev:
11.005
12.867
Fixed effects: yield ~ nitro
(Intercept) 81.872
6.9453 53 11.788 <.0001
nitro 73.667
6.7815 53 10.863 <.0001
50
Correlation:
(Intrc
nitro -0.293
Min
Q1
Med
Q3
Max
-1.7438 -0.66475 0.017104 0.54299 1.803
Number of Groups:
Block Variety %in% Block
6
18
With VarCorr and intervals we can examine the variance components

and their condence intervals for this model
> VarCorr( fm4Oats )
Variance StdDev
Block = pdSymm(~ 1)
(Intercept) 210.42
14.506
Variety = pdSymm(~ 1)
(Intercept) 121.10
11.005
Residual 165.56
12.867
> intervals( fm4Oats )
Fixed effects:
lower
est. upper
(Intercept) 67.942 81.872 95.803
nitro 60.065 73.667 87.269
Random Effects:
Level: Block
lower
est. upper
sd((Intercept)) 6.6086 14.506 31.841
Level: Variety
lower
est. upper
sd((Intercept)) 6.408 11.005 18.899
lower
est. upper
10.637 12.867 15.565
We can see that the random eects at the Block and plot levels account
for a substantial amount of the variability in the response. Although the
standard deviations of these random eects are not estimated very precisely,
it does not appear reasonable that they could be zero. To check this we
would t models without these random eects and use likelihood ratio tests
to compare them to fm4Oats. We do not show that here.

0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.4
V/Victory
V/Golden Rain
V/Marvellous
VI/Victory
VI/Golden Rain
VI/Marvellous
III/Victory
III/Golden Rain
III/Marvellous
IV/Victory
IV/Golden Rain
IV/Marvellous
51
0.6
160
140
120
100
80
Yield (bushels/acre)
60
160
140
120
100
80
60
I/Victory
I/Golden Rain
I/Marvellous
II/Victory
II/Golden Rain
II/Marvellous
160
140
120
100
80
60
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
Nitrogen concentration (cwt/acre)
FIGURE 1.21. Observed and predicted yields in bushels/acre for three dierent
varieties of oats at four dierent concentrations of nitrogen (hundred weight/acre)
by block and variety. Although the model has a random eect for variety, the
whole-plot factor, there is no xed eect for variety.
The modeling of the dependence on nitrogen level by a simple linear term

appears adequate. Plots of the original data and the tted curves, obtained
with
> plot( augPred( fm4Oats ), aspect = 2.5, layout = c(6, 3),
+
between = list( x = c(0, 0, 0.5) ) ) # produces Figure 1.21
do not show any systematic lack of t. (The extra arguments in the plot
call are used to enhance the appearance of the plot. They are described in
3.3.)
We could (and did) examine other common diagnostic plots to check for
inadequacies in this model, but did not nd any. We now have a simple,
adequate model to explain the dependence of the response on both the levels
52
of nitrogen applied, using xed-eects terms, and the random variability

in blocks and plots, using random-eects terms.
1.7 Chapter Summary

In this chapter we have presented some motivation for the use of linear
mixed-eects models and an overview of their application. We showed several examples of the types of data to which these models can be applied.
The dening characteristics of mixed-eects models are that they are applied to data where the observations are grouped according to one or more
levels of experimental units and that they incorporate both xed-eects
terms and random-eects terms. A xed-eects term in a model describes
the behavior of the entire population or of those units associated with repeatable levels of experimental factors. A random-eects term describes
the distribution within the population of a coecient. The eects in a
random-eects term are associated with the individual experimental units
sampled from the population.
A linear mixed-eects model is t with the lme function. A preliminary
list of tted linear models by experimental unit can be obtained with the
lmList function. Both lmList and lme ts can be examined with the accessor
functions coef, ranef, fixef, residuals, and fitted. A summary of the t is
obtained with the summary function. Diagnostic plots are generated with the
plot function or with specialized constructors such as augPred, compareFits,
and comparePred.
The signicance of xed-eects terms is assessed with the single-argument
form of the anova function or directly from the the summary function. Different forms of random-eects terms can be compared by tting dierent
models and comparing them with the multiple-argument form of anova.
The purpose of this chapter is to present the motivation for using LME
models with grouped data and to set the stage for later chapters in the
book, dealing with the theory and computational methods for LME models
(Chapter 2) and the linear mixed-eects modeling facilities in the nlme
library (Chapter 4).
Exercises
1. The PBIB data (Appendix A.22) are from an agricultural experiment
that was laid out as a partially balanced incomplete block design. This
is described in more detail in 2.4.2. The roles of the variables in these
data are indicated by the names: response, Treatment, and Block. The
structure is similar to that of the ergoStool data but, because there
Exercises
53
are only four observations in each block and there are 15 levels of
Treatment, each block receives only a subset of the treatments.
(a) Plot the PBIB data with plot(PBIB). Do there appear to be systematic dierences between blocks?
(b) Create a design plot, like Figure 1.6 (p. 14), for the PBIB data.
(c) Fit a linear mixed-eects model, with xed eects for Treatment
and random eects for the intercept by Block, to the PBIB data.
The call to lme would be like that used to t fm1Stool in 1.2.
(d) Apply anova to the tted model. Is the Treatment term signicant? Describe the hypothesis being tested.
(e) Create a plot of the residuals versus the tted values for this
tted model. This plot is like Figure 1.4. Does this plot indicate
greater variance in the residuals at higher levels of the response?
(f) Create a plot of the standardized residuals versus the tted values by Block. This plot is like Figure 1.8 (p. 21). Does this plot
indicate systematic patterns in the residuals by Block?
We will discuss the anova results for this tted model in more detail
in 2.4.2.
2. The Oxboys data described in 3.1 and Appendix A.19 consist of the
heights of 26 boys from Oxford, England, each measured on nine
dierent occasions. The structure is similar to that of the OrthoFem
data of 1.4.
(a) Plot the data (using plot(Oxboys)) and verify that a simple linear regression model gives a suitable representation of the boys
growth patterns. Do there appear to be signicant dierences in
the individual growth patterns?
(b) Fit a simple linear regression model to height versus age using
the lm function, ignoring the Subject eects. Obtain the boxplots
of the residuals by Subject with bwplot(Subject ~ resid(object),
Oxboys), where object should be replaced with the name of the
tted lm object. Explain the observed pattern.
(c) Use the lmList function to t separate simple linear regression
models for each Subject, using a call similar to the one used to
produce fm1OrthF.lis in 1.4. Compare the boxplots of the residuals by Subject for the lmList t (obtained with plot(object,
Subject ~ resid(.)), with object replaced with the name of the
lmList object) to those obtained for the lm t. Compare also the
residual standard errors from the two ts and comment.
(d) Plot the individual condence intervals on the parameters estimated in the lmList t and verify that both the intercept and
the slope vary signicantly with Subject.
54
(e) Use the lme function to t an LME model to the data with
random eects for both the intercept and the slope, using a call
similar to the one used to obtain fm1OrthF in 1.4. Examine the
boxplots of the residuals by Subject, comparing them to those
obtained for the lm and lmList ts.
(f) Produce the plot of the standardized residuals versus tted values (plot(object)) and the normal plot of the standardized
residuals (qqnorm(object)). (In both cases object should be replaced with the name of the lme object.) Can you identify any
departures from the models assumptions?
(g) Plot the augmented predictions for the lme t (obtained with
plot(augPred(object))). Do the linear models for each subject
appear adequate?
(h) Another way of assessing the linear models for each subject is
to plot the residuals versus age by Subject (use plot(object,
resid(.) ~ age | Subject), replacing object with the name of
the lme object). Several subjects have a noticeable scooping
pattern in their residuals, indicating the need for a model with
curvature.
(i) Use the lmList function to t separate quadratic models for each
subject. A quadratic model in age, as shown in fm1Pixel of 1.5,
would be t with lmList(height ~ age + age2, Oxboys).
(j) Examine a plot of the condence intervals on coecients from
this second lmList t. Are there indications that the coecients
dier between subjects? Are the quadratic coecients signicantly dierent from zero for some subjects?
(k) Fit the full mixed-eects model corresponding to the last lmList
t. The model will have linear and quadratic terms for age in the
xed-eects and the random eects. A simple way to describe
this model is lme(object) replacing object with the name of the
lmList t.
(l) Check residual plots and numerical summaries for this lme model.
Do there appear to be deciencies in the t? Do there appear to
be terms in the model that could be eliminated?
3. The LME model used for the Pixel data in 1.5 uses random eects for
the intercept and the slope at the Dog level and a single random eect
for the intercept at the Side within Dog level. We did not discuss there
how that random-eects model was chosen. The lmList function can
be used with multilevel data to investigate which terms in an LME
model require random eects.
(a) Use lmList to t a separate quadratic model in day for each Dog.
Print the tted object and examine the estimated coecients.
Exercises
55
Can you explain the error message printed in the lmList t? Notice that lmList was able to recover from the error and proceed
to normal completion.
(b) Plot the individual condence intervals for the coecients in the
lmList t. Verify that only the intercept and the linear coecient
seem to vary signicantly with Dog.
(c) Use the level argument to lmList to t separate quadratic models in day for each Side within Dog (use Dog/Side as the grouping
expression and set level=2). Print the tted object using summary
and explain the missing values (NA) for the standard errors of Dog
10.
(d) Plot the individual condence intervals for the coecients in
the lmList t by Side within Dog and verify that there is more
variation among the intercepts and the linear coecients than
among the quadratic coecients.
(e) Fit an LME model with random eects for the intercept and the
linear term at both levels of grouping. Compare the resulting
lme t to the fm1Pixel object in 1.5 using anova. Which model
should be the preferred?
4. The Alfalfa data described in Appendix A.1 is another example of a
split-plot experiment. The structure is similar to that of the Oats data
of 1.6: a 3 4 full factorial on varieties of alfalfa and date of third
cutting is used with 6 blocks each subdivided into 4 plots according
to a split-plot arrangement. The whole-plot treatments are given by
the varieties and the subplot treatments by the date of third cutting.
(a) Plot the data (using plot(Alfalfa)). Do there appear to be cutting dates that are consistently worse/better than the others?
What can you say about the block-to-block variation in the
yields?
(b) Use lme to t a two-level LME model with grouping structure Block/Variety, including a single random intercept for each
level of grouping (i.e., random = ~1 | Block/Variety). Assume
a full factorial structure with main eects and interactions for
the xed eects (i.e., fixed = Yield ~ Date * Variety). Use the
treatment contrasts (options(contrasts = c("contr.treatment",
"contr.poly"))) to get more interpretable coecients for the
xed eects.
(c) Examine the signicance of the terms in the model using anova,
verifying that there are no signicant dierences between varieties and no signicant interactions between varieties and cutting dates.
56
(d) Because the data are balanced, a similar ANOVA model can
be t using aov and the Error function (use aov(Yield ~ Date
* Variety + Error(Block/Variety), Alfalfa)). Compare the results from the aov and lme ts, in particular the F-values and
p-values for testing the terms in the xed-eects model (these are
obtained for the aov object using the summary function). In this
case, because of the balanced structure of the data, the REML
t (obtained with lme) and the ANOVA t (obtained with aov)
are identical.
(e) Ret the LME model using xed eects for Date only (a simple
way to do this is to use update(object, Yield ~ Date), where
object should be replaced with name of the previous lme object). Print the resulting object using summary and investigate
the dierences between the cutting dates (recall that, for the
treatment contrasts, the coecients represent dierences with
respect to the cutting date labelled None). Can you identify a
trend in the eect of cutting date on yield?
(f) Examine the plot of the residuals versus tted values and the
normal plot of the residuals. Can you identify any departures
from the LME models assumptions?
2
Theory and Computational Methods
for Linear Mixed-Eects Models
In this chapter we present the theory for the linear mixed-eects model
introduced in Chapter 1. A general formulation of LME models is presented and illustrated with examples. Estimation methods for LME models, based on the likelihood or the restricted likelihood of the parameters,
are described, together with the computational methods used to implement them in the lme function. Asymptotic results on the distribution of
the maximum likelihood estimators and the restricted maximum likelihood
estimators are used to derive condence intervals and hypotheses tests for
the models parameters.
The purpose of this chapter is to present an overview of the theoretical
and computational aspects of LME models that allows the evaluation of the
strengths and limitations of such models for practical applications. It is not
the purpose of this chapter to present a thorough theoretical description
of LME models. Such a comprehensive treatment of the theory of linear
mixed-eects models can be found, for example, in Searle, Casella and
McCulloch (1992) or in Vonesh and Chinchilli (1997).
Readers who are more interested in the applications of LME models
and the use of the functions and methods in the nlme library to t such
models can, without loss of continuity, skip this chapter and go straight
to Chapter 3. If you decide to skip this chapter at a rst reading, it is
recommended that you return to it (especially 2.1) at a later time to get
a good understanding of the LME model formulation and its assumptions
and limitations.
58
2. Theory and Computational Methods for LME Models
2.1 The LME Model Formulation

Linear mixed-eects models are mixed-eects models in which both the
xed and the random eects occur linearly in the model function. They extend linear models by incorporating random eects, which can be regarded
as additional error terms, to account for correlation among observations
within the same group.
In this section we present a general formulation for LME models proposed
by Laird and Ware (1982). The original single-level formulation is described
in 2.1.1 and its multilevel extension is described in 2.1.2.
2.1.1 Single Level of Grouping

For a single level of grouping, the linear mixed-eects model described by
Laird and Ware (1982) expresses the ni -dimensional response vector y i for
the ith group as
y i = X i + Z i bi + i ,
bi N (0, ),
i = 1, . . . , M,
i N (0, 2 I),
(2.1)
where is the p-dimensional vector of xed eects, bi is the q-dimensional

vector of random eects, X i (of size ni p) and Z i (of size ni q) are
known xed-eects and random-eects regressor matrices, and i is the
ni -dimensional within-group error vector with a spherical Gaussian distribution. The assumption Var(i ) = 2 I can be relaxed as shown in Chapter 5, where we describe extensions that allow us to model nonconstant
variances or special within-group correlation structures. The random effects bi and the within-group errors i are assumed to be independent for
dierent groups and to be independent of each other for the same group.
Because the distribution of the random eects vectors bi is assumed to
be normal (or Gaussian) with a mean of 0, it is completely characterized
by its variancecovariance matrix . This matrix must be symmetric and
positive semi-denite; that is, all its eigenvalues must be non-negative. We
will make the stronger assumption that it is positive-denite which is to
say that all its eigenvalues must be strictly positive. We can make this
restriction because an indenite model can always be re-expressed as a
positive-denite model of lower dimension.
The random eects bi are dened to have a mean of 0 and therefore any
nonzero mean for a term in the random eects must be expressed as part
of the xed-eects terms. Thus, the columns of Z i are usually a subset of
the columns of X i .
When computing with the model it is more convenient to express the
variancecovariance matrix in the form of a relative precision factor, ,
59
which is any matrix that satises

1
= T .
1/ 2
If is positive-denite then such a will exist, but it need not be unique.
The Cholesky factor (Thisted, 1988, 3.3) of 2 1 is one possible . The
matrix is called a relative precision factor because it factors the precision
matrix, 1 , of the random eects, expressed relative to the precision,
1/ 2 , of the i .
We use some of the examples in Chapter 1 to illustrate the general LME
model formulation.
Railway Rails Experiment
In the case of the rails data introduced in 1.1, M = 6, ni = 3, i = 1, . . . , 6,
p = q = 1, and the regressor matrices for the xed and random eects are
particularly simple:

1
X i = Z i = 1 = 1 , i = 1, . . . , 6.
1
The random eects bi , i = 1, . . . , 6 are scalars; hence their variance b2 is
also a scalar, as is the relative precision factor, . There is only one choice
for (up to changes in sign) and that is

= 2 /b2 .
Ergometric Experiment of Types of Stools
The data for the stools ergometric experiment of 1.2 are balanced, with
M = 6, ni = 4, i = 1, . . . , 6, p = 4, and q = 1. The xed-eects regressor
matrices X i are determined by the contrasts chosen to represent the types
of stool. For the Helmert contrasts parameterization used in the t of the
fm1Stool object in 1.2.1, we have
1 1 1 1
1
1 1 1
, i = 1, . . . , 6.
Xi =
1
0
2 1
1
0
0
3
The random-eects regression matrices Z i and the relative precision factor are the same as in the rails example.
Orthodontic Growth Curve in Girls
The orthodontic growth curve data for females presented in 1.4.1 are also
balanced, with M = 11, ni = 4, i = 1, . . . , 11. For the LME model with
60
random eects for both the intercept and the slope, used to t the fm2OrthF
object in 1.4.1, we have p = q = 2 and the xed- and random-eects
regressor matrices are identical and given by
1 8
1 10
Xi = Zi =
1 12 , i = 1, . . . , 11.
1 14
Any square-root of the 2 2 matrix 2 1 can be used as a relative
precision factor in this case.
2.1.2 A Multilevel LME Model

The LairdWare formulation for single-level LME models presented in
2.1.1 can be extended to multiple, nested levels of random eects. In the
case of two nested levels of random eects the response vectors at the innermost level of grouping are written y ij , i = 1, . . . , M, j = 1, . . . , Mi where
M is the number of rst-level groups and Mi is the number of second-level
groups within rst-level group i. The length of y ij is nij .
The xed-eects model matrices are X ij , i = 1, . . . , M, j = 1, . . . , Mi of
size nij p. Using rst-level random eects bi of length q1 and second-level
random eects bij of length q2 with corresponding model matrices Z i,j of
size ni q1 and Z ij of size ni q2 , we write the model as
bi N (0, 1 ),
i = 1, . . . , M,
bij N (0, 2 ),
j = 1, . . . , Mi ,
(2.2)
ij N (0, 2 I).
The level-1 random eects bi are assumed to be independent for dierent i,

the level-2 random eects bij are assumed to be independent for dierent
i or j and to be independent of the level-1 random eects, and the withingroup errors ij are assumed to be independent for dierent i or j and to
be independent of the random eects.
Extensions to an arbitrary number Q of levels of random eects follow
the same general pattern. For example, with Q = 3 the response for the
kth level-3 unit within the jth level-2 unit within the ith level-1 unit is
written
y ijk = X ijk + Z i,jk bi + Z ij,k bij + Z ijk bijk + ijk ,
i = 1, . . . , M,
bi N (0, 1 ),
j = 1, . . . , Mi ,
bij N (0, 2 ),
k = 1, . . . , Mij ,
bijk N (0, 3 ),
ijk N (0, 2 I).
Note that the distinction between, say, the kth horizontal section of the
regressor matrix for the level-2 random eect bij , written Z ij,k , and the
61
jkth horizontal section of the regressor matrix for the level-1 random eect
bi , written Z i,jk , is the position of the comma in the subscripts.
As with a single level of random eects, we will express the variance
covariance matrices, q , q = 1, . . . , Q, in terms of relative precision factors
q .
In this book, we only consider mixed-eects models with a multivariate
normal (or Gaussian) distribution for the random eects and the withingroup errors. Generally we assume that the variancecovariance matrix
q for the level-q random eects can be any positive-denite, symmetric
matrix. In some models we will further restrict the form of q , say by
requiring that it be diagonal or that it be a multiple of the identity.
Those familiar with the multilevel modeling literature (Bryk and Raudenbush, 1992; Goldstein, 1995) may notice that we count levels dierently.
In that literature the model (2.1) is called a two-level model because there
are two levels of random variation. Similarly, the model (2.2) is called a
three-level model. We prefer the terminology from the experimental design
literature and count the number of levels as the number of nested levels
of random eects.
Split-Plot Experiment on Varieties of Oats and Nitrogen Levels

We use the split-plot experiment on the yield of three dierent varieties
of oats measured at four dierent concentrations of nitrogen, described in
1.6, to illustrate the multilevel LME model formulation. The nal model
used in that section, corresponding to the tted object fm4Oats, represents
the yield yijk for the jth variety of oat at the kth nitrogen concentration
Nk in the ith block as
yijk = 0 + 1 Nk + bi + bij + ijk , i = 1, . . . , 6, j = 1, . . . , 3, k = 1, . . . , 4,

bi N 0, 12 , bij N 0, 22 , ijk N (0, 2 ).
The xed eects are the intercept 0 and the nitrogen slope 1 . The bi
denote the Block random eects, the bij denote the Variety within Block
random eects, and the ijk denote the within-group errors. This is an
example of a two-level mixed-eects model, with the bij random eects
nested within the bi random eects.
In this example, M = 6, Mi = 3, nij = 4, i = 1, . . . , 6 j = 1, . . . , 3,
p = 2, and q1 = q2 = 1. The regressor matrices are
X ij
1
1
=
1
1
0.0
0.2
,
0.4
0.6
Z i,j = Z i j

1
1
=
1 ,
1
i = 1, . . . , 6,
j = 1, . . . , 3.
62
Because all the random eects are scalars, the precision factors are
uniquely dened (up to changes in sign) as
1 =

2 /12
and 2 =

2 /22 .
2.2 Likelihood Estimation for LME Models

Several methods of parameter estimation have been used for linear mixedeects models. We will concentrate on two general methods: maximum
likelihood (ML) and restricted maximum likelihood (REML). Descriptions
and comparisons of the various estimation methods used for LME models
can be found, for example, in Searle et al. (1992) and Vonesh and Chinchilli
(1997).
2.2.1 The Single-Level LME Likelihood Function

Consider rst the model (2.1) that has a single level of random eects. The
parameters of the model are , 2 , and whatever parameters determine .
We use to represent an unconstrained set of parameters that determine
. We will discuss parameterizations of in 2.2.7for now we will simply
assume that a suitable parameterization has been chosen.
The likelihood function for the model (2.1) is the probability density for
the data given the parameters, but regarded as a function of the parameters
with the data xed, instead of as a function of the data with the parameters
xed. That is,

L , , 2 |y = p(y|, , 2 ),
where L is the likelihood, p is a probability
density, and y is the entire

n
.
N-dimensional response vector, N = M
i
i=1
Because the nonobservable random eects bi , i = 1, . . . , M are part of
the model, we must integrate the conditional density of the data given the
random eects with respect to the marginal density of the random eects
to obtain the marginal density for the data. We can use the independence
of the bi and the i to express this as
M

L , , 2 |y =
p y i |, , 2
i=1
i=1

p y i |bi , , 2 p bi |, 2 dbi ,
(2.3)
where the conditional density of y i is multivariate normal

exp y i X i Z i bi 2 /2 2

p y i |bi , , 2 =
(2 2 )ni /2
and the marginal density of bi is also multivariate normal

exp bTi 1 bi

p bi |, 2 =
q/2
(2)
||

2
exp bi /2 2
=
,
(2 2 )q/2 abs ||1
63
(2.4)
(2.5)
where |A| denotes the determinant of the matrix A. Substituting (2.4) and
(2.5) into (2.3) provides the likelihood as

L , , 2 |y =

exp y X i Z i bi 2 + bi 2 /2 2
M

i
abs ||
dbi
2 ni /2
(2 2 )q/2
i=1 (2 )

2

2
exp yi X i Z i bi /2
M

abs ||
=
dbi , (2.6)
q/2
2 ni /2
(2 2 )
i=1 (2 )
where

y
i = i ,
y
0

Xi
Xi =
,
0

Zi
Zi =
,
(2.7)
are augmented data vectors and model matrices. This approach of changing
the contribution of the marginal distribution of the random eects into
extra rows for the response and the design matrices is called a pseudodata approach because it creates the eect of the marginal distribution by
adding pseudo observations.
The exponent in the integral of (2.6) is in the form of a squared norm or,
more specically, a residual sum-of-squares. We can determine the conditional modes of the random eects given the data, written
bi , by minimizing
this residual sum-of-squares. This is a standard least squares problem for
which we could write the solution as
T 1 T

i .
Zi yi X
bi = Zi Zi
The squared norm can then be expressed as
2
2

2

i Zi bi
i Zi
bi
bi + Zi bi
= yi X
yi X
2

T T

i Zi
= yi X
bi Zi Zi bi
bi . (2.8)
bi + bi
64
The rst term in (2.8) does not depend on bi so its exponential can be factored out of the integral in (2.6). Integrating the exponential of the second
term in (2.8) is equivalent, up to a constant, to integrating a multivariate
normal density function. Note that

T T

2

T

exp
b
Z
Z
b
/2
i
i
i
i
i
i
Z
i|
|Z
i

dbi
q/2
(2 2 )
TZ
i|
|Z
i

T T

2
exp bi
bi Z i Z i bi bi /2
1

dbi
=
2 )q/2 / |Z
TZ
TZ
i|
i|
|Z
(2
i
i
1
1
=
=
. (2.9)
T
T
T
|Z
Z
+
|
|Z i Z i |
i
i
By combining (2.8) and (2.9) we can express the integral in (2.6) as

2

i Zi bi
exp yi X
/2 2
q/2
(2 2 )
dbi

2

i Zi
exp yi X
bi /2 2

=

T
Z i Z i
to give

L , , 2 |y
=
1
(2 2 )N/2
exp
2
M

Z
b
i
i i abs ||
i=1 i

.
2 2
T
i=1
Z i Z i
(2.10)
The expression (2.10) could be used directly in an optimization routine to

calculate the maximum likelihood estimates for , , and 2 . However, the
optimization is much simpler if we rst concentrate or prole the likelihood
so it is a function of only. That is, we calculate the conditional estimates

()
and
2 () as the values that maximize L(, , 2 ) for a given .
Notice that the parts of (2.10) involving and 2 are identical in form

to the likelihood for a linear regression model so ()
and
2 () can be
determined from standard linear regression theory.
65
We do need to be careful because the least squares estimates for will

depend on the conditional modes
bi and these, in turn, depend on . Thus,
we must determine these least squares values jointly as the least squares
solution to
T
T
T
T

bM ,
= arg min y e X e (b1 , . . . , bM , )T 2 ,
b1 , . . . ,
b1 ,...,bM ,
where
Z1
Xe = 0
..
.
0
0
0
0
Z2
..
.
...
...
...
...
..
.
0
0
0
0
..
.
0
0
...
...
ZM
X1
0
X2
..
.
XM
0
and
y1
0
y2
ye = 0 .
..
.
y M
0
(2.11)
Conceptually we could write

T
T
T
T

bM ,
= (X Te X e )1 X Te y e ,
b1 , . . . ,
but we denitely would not want to calculate these values this way. The
matrix X e is sparse and can be very large. If possible we want take advantage of the sparsity and avoid working directly with X e .
Linear regression theory also gives us the conditional maximum likelihood estimate for 2
T
T
T )T 2
b1 , . . . ,
bM ,
y e X e (
.
() =
N
2
(2.12)
Notice that the maximum likelihood estimate of 2 is the residual sum-ofsquares divided by N , not by N p.
Substituting these conditional estimates back into (2.10) provides the
proled likelihood
M
exp (N/2) abs ||

L() = L((),
,
2 ()) =
.
[2
2 ()]N/2 i=1 T
Z i Z i
(2.13)

bM or ()
to
We do not actually need to calculate the values of
b1 , . . . ,
evaluate the proled likelihood. We only need to know the norm of the
residual from the augmented least squares problem. The decomposition
methods described in 2.2.2 provide us with fast, convenient methods of
calculating this.
66
The pseudo-data representation of the marginal density p(y i |, , 2 )

used in (2.6) is just one way of expressing this density and deriving the
likelihood. It is also possible to describe this density as a normal distribution with mean 0 and a patterned variancecovariance matrix i a representation that is often used to derive the likelihood for the parameters in
a linear mixed-eects model. Although we will not use this representation
extensively in this chapter, we will use it in Chapter 5, so we present some
of this derivation of the likelihood here.
The model (2.1) can be re-expressed as
y i = X i + Z i bi + i = X i + i ,
i
i = 1, . . . , M,
(2.14)
i
where
= Z i bi + i . Because the
are the sum of two independent
multivariate normal random vectors, they are independently distributed as
multivariate normal vectors with mean 0 and variancecovariance matrix
2 i , where i = I + Z i Z Ti / 2 . It then follows from (2.14) that the y i
are independent multivariate normal random vectors with mean X i and
variancecovariance matrix 2 i . That is,
!
T
1

ni

(y i X i ) 1
2
2 2
i (y i X i )
exp
|i | 2 .
p y i |, , = 2
2 2
For a given value of , the values of and 2 that maximize the likelihood
could be written as
!1 M
M
"
"
T
1

X Xi
X T 1 y ,
()
=
i
i=1
2 () =
i=1

T
M
1

X
()
()
y
y
i
i
i
i
i
i=1
.
N
Computationally these expressions are much more dicult than (2.11) and

(2.12). Using these expressions for ()
and
2 () we could derive the
proled likelihood or log-likelihood.
We present these expressions for completeness only. We prefer to use
the expressions from the pseudo-data representation for computation, especially when the pseudo-data representation is combined with orthogonaltriangular decompositions described in the next section.
2.2.2 Orthogonal-Triangular Decompositions

Orthogonal-triangular decompositions of rectangular matrices are a preferred numerical method for solving least squares problems (Chambers,
1977; Kennedy and Gentle, 1980; Thisted, 1988). They are also called QR
decompositions as the decomposition is often written

R
X =Q
= Qt R,
0
67
where X is an n p matrix (n p) of rank p, Q is n n and orthogonal, R

is p p and upper triangular, and Qt (Q-truncated) consists of the rst p
columns of Q. To say that Q is orthogonal means that QT Q = QQT = I.
This implies that QTt Qt = I.
The S function qr is used to create a QR decomposition from a matrix.
For example, in 1.4.1 we present a model where the xed-eects model
matrices for each subject are
1 8
1 10
Xi =
1 12 , i = 1, . . . , 11.
1 14
We can generate such a matrix in S and create its decomposition by
> Xmat <- matrix( c(1, 1, 1,
> Xmat
[,1] [,2]
[1,]
1
8
[2,]
1
10
[3,]
1
12
[4,]
1
14
> Xqr <- qr( Xmat )
> qr.R( Xqr )
[,1]
[,2]
[1,]
-2 -22.0000
[2,]
0 -4.4721
> qr.Q( Xqr )
[,1]
[,2]
[1,] -0.5 0.67082
[2,] -0.5 0.22361
[3,] -0.5 -0.22361
[4,] -0.5 -0.67082
> qr.Q( Xqr, complete = TRUE
[,1]
[,2]
[,3]
[1,] -0.5 0.67082 0.023607
[2,] -0.5 0.22361 -0.439345
[3,] -0.5 -0.22361 0.807869
[4,] -0.5 -0.67082 -0.392131
1, 8, 10, 12, 14), ncol = 2 )
# creates a QR structure
# returns R
# returns Q-truncated
# returns the full Q

[,4]
0.54721
-0.71202
-0.21760
0.38240
Although we will write expressions that involve Q, this matrix is not

usually evaluated explicitly. Products such as QT y or Qy can be calculated directly from information about the decomposition without having
to generate this n n matrix. See Dongarra, Bunch, Moler and Stewart
(1979, Chapter 9) for details. The S functions qr.qty and qr.qy evaluate
these products directly.
An important property of orthogonal matrices is that they preserve
norms of vectors under multiplication either by Q or by QT . That is,
68
the transformation represented by Q is a generalization of a rotation or

a reection in the plane. In particular,
QT y2 = (QT y)T QT y = y T QQT y = y T y = y2 .
If we apply this to the residual vector in a least squares problem we get

2

y X2 = QT (y X)
= QT y QT X2

2

R
T
=
c
Q
Q

0

2

R

=
c 0
= c1 R2 + c2 2 ,
T

= QT y is the rotated residual vector. The components
where c = cT1 cT2
c1 and c2 are of lengths p and n p, respectively.
Because X has rank p, the p p matrix R is nonsingular and upper is easily evaluated as the solution
triangular. The least-squares solution
to
= c1
R
and the residual sum-of-squares is c2 2 . Notice that the residual sum-of
squares can be evaluated without having to calculate .
2.2.3 Evaluating the Likelihood Through Decompositions

Returning to the linear mixed-eects model, we take an orthogonal-triang i from (2.7) as
ular decomposition of the augmented model matrix Z

R11(i)
Z i = Q(i)
,
0
where Q(i) is (ni + q) (ni + q) and R11(i) is q q. Then

2

2

i Z
i bi
i Z
i bi
i X
i X

y
= QT(i) y

2
2
= c1(i) R10(i) R11(i) bi + c0(i) R00(i) ,
where the q p matrix R10(i) , the ni p
and the ni -vector c0(i) are dened by

R10(i)
i and
= QT(i) X
R00(i)
matrix R00(i) , the q-vector c1(i)

c1(i)
i.
= QT(i) y
c0(i)
69
Another way of thinking of these matrices is as components in an orthogonaltriangular decomposition of an augmented matrix

Zi
Xi
0

yi
R11(i)
= Q(i)
0
0
R10(i)
R00(i)

c1(i)
,
c0(i)
where the reduction to triangular form is halted after the rst q columns.
(The peculiar numbering scheme for the submatrices and subvectors is
designed to allow easy extension to more than one level of random eects.)
Returning to the integral in (2.6) we can now remove a constant factor and
reduce it to

exp y X i Z i bi 2 + bi 2 /2 2
i
dbi
2 2

c1(i) R10(i) R11(i) bi 2

2 exp
2
c0(i) R00(i)
2
= exp
dbi . (2.15)
q/2
2 2
(2 2 )
Because R11(i) is nonsingular, we can perform a change of variable to i =
(c1(i) R10(i) R11(i) bi )/ with dierential di = q abs |R11(i) | dbi
and write the integral as

exp c1(i) R10(i) R11(i) bi 2 /2 2
(2 2 )q/2
dbi

exp i 2 /2
1

=
di
q/2
abs R11(i)
(2)

= 1/ abs R11(i) .
(2.16)
This is the same result as (2.10) because

#

R11(i)
T
Z i Z = RT11(i) 0 QT(i) Q(i)

0

= RT11(i) R11(i)

= RT11(i) R11(i)

T 2
=
R11(i)

= abs R11(i) .
Because R11(i) is triangular, its determinant is simply the product of its
diagonal elements.
70
Substituting (2.16) into (2.15) into (2.6) provides the likelihood as

2
!
2

M exp c

0(i) R00(i) /2

||
2

abs
L , , |y =
R11(i)
(2 2 )ni /2
i=1

2
M
!
M
exp i=1 c0(i) R00(i) /2 2
||
.
=
abs
(2 2 )N/2
R11(i)
i=1
The term in the exponent has the form of a residual sum-of-squares for
pooled over all the groups. Forming another orthogonal-triangular decomposition
R00(1) c0(1)

..
.. = Q R00 c0
(2.17)
.
0
.
0
c1
R00(M ) c0(M )
produces the reduced form

L , , 2 |y
2
N/2

exp
= 2 2
c1 + c0 R00

2 2
abs
i=1
!
||

R11(i) .
(2.18)
For a given , the values of and 2 that maximize (2.18) are

()
= R1
00 c0
and
2 () =
c1
,
N
(2.19)
which give the proled likelihood

L(|y) = L (),
,
2 ()|y
=
!N/2
N
2
2 c1

M

||
N
exp
abs
,
2 i=1
|R11(i) |
(2.20)
or the proled log-likelihood

(|y) = log L(|y)
=

M
"
||
N
[log N log(2) 1] N log c1 +
log abs
.
2
|R11(i) |
i=1
(2.21)
The proled log-likelihood (2.21) is maximized with respect to , produc The maximum likelihood estimates
ing the maximum likelihood estimate .
2
in (2.19).

and
are then obtained by setting =
71
Although technically the random eects bi are not parameters for the
statistical model, they do behave in some ways like parameters and often
we want to estimate their values. The conditional modes of the random
eects, evaluated at the conditional estimate of , are the Best Linear Unbiased Predictor s or BLUP s of the bi , i = 1, . . . , M . They can be evaluated,
using the matrices from the orthogonal-triangular decompositions, as

R
()
.
(2.22)
bi () = R1
c
1(i)
10(i)
11(i)
In practice, the unknown vector is replaced by its maximum likelihood

producing estimated BLUPs
estimate ,
bi ().
The decomposition (2.17) is equivalent to calculating the QR decomposition of the potentially huge matrix X e dened in (2.11). If we determined
the least-squares solution to (2.11) using an orthogonal-triangular decomposition

Re
,
X e = Qe
0
the triangular part of the decomposition and the leading part of the rotated,
augmented response vector would be
R11(1)
c1(1)
0
...
0
R10(1)
0
c1(2)
R11(2) . . .
0
R10(2)
..
.
.
.
.
.
..
..
..
.. and c1 =
Re = .
.. .
0
c1(M )
0
. . . R11(M ) R10(M )
0
0
0
0
R00
c0

Thus, the ()
and
2 () from (2.19) are the same as those from (2.17)
and (2.12). The vector c1 is the residual vector in the coordinate system
determined by Qe . Because Qe is orthogonal, c1 2 is the residual sumof-squares for the least squares problem dened by X e and y e .
The proled log-likelihood (2.20) has the same form as (2.13). It consists
of three additive components; a constant, a scaled logarithm of the residual
sum-of-squares, and a sum of ratios of the logarithms of determinants. In
the next section we examine these terms in detail.
2.2.4 Components of the Proled Log-Likelihood

Returning to the example of the rails data of 1.1, let us consider the
dierent components of the proled log-likelihood as expressed in (2.21).
Recall that the relative precision factor will be a scalar in this case so
let us write it as . There are three additive terms in the proled loglikelihood:
72
1. The constant N2 [log N log(2) 1], which can be neglected for the
purposes of optimization.
2. N log c1 , a multiple of the logarithm of the norm of a residual
vector from the penalized least-squares t for , X i , Z i , and y i .

M
M
T
2
3.
i=1 log / abs |R11(i) | =
i=1 log / |Z i Z i + | . In the
general case this is the sum of the logarithms of the ratios of determinants.
In Figure 2.1 we show the two nonconstant terms and the resulting loglikelihood as a function of for the rails example
The shapes of the curves in Figure 2.1 indicate that it would be better to
optimize the proled log-likelihood with respect to = log instead of .
This transformation will also help to ensure that does not become negative during the course of the iterations of whatever optimization routine
we use. In Figure 2.2 we show the components and the log-likelihood as a
function of . We can see that the log-likelihood is closer to a quadratic
with respect to than with respect to .
There are patterns in Figure 2.2 that will hold in general for linear
mixed-eects models. The log of the norm of the residual is an increasing sigmoidal, or S-shaped, function with respect to . As (or
0), this log-norm approaches a horizontal asymptote at a value that
corresponds to the log residual norm from an unpenalized regression of the
form

b

Z1 0 . . .
0 X1 1
y1
b2
y2 0 Z 2 . . .
0 X2
..

y= . = .
..
..
.. . + .
.. ..
.
...
.
.
bn
yn
0
0 . . . Zn Xn
At the other extreme, large positive values of , and the correspondingly

large values of , put such a heavy penalty on the size of the bi terms in
the regression that these are forced to zero. Thus, as , the penalized
residual norm approaches that from a regression of the entire response
vector y on the X i matrices alone.

y1
X1
y2 X 2

y = . = . + .
.. ..
yn
Xn
In the ratio of determinants term, very large values of will dominate
Z Ti Z i in the denominator so the ratios approach / and the sum of the
73
-80
-75
-70
-65
log(likelihood)
3.0
3.5
4.0
4.5
log(residual norm)
-30
-20
-10
log(determinant ratio)
Delta
FIGURE 2.1. The proled log-likelihood as a function of for the rails example.
Two of the components of the log-likelihood, log 1 , the log of the length of
M
the residual, and
i=1 log / abs |11(i) | , the log of the determinant ratios,
are shown on the same scale.
74
-80
-75
-70
-65
log(likelihood)
3.0
3.5
4.0
4.5
log(residual norm)
-30
-20
-10
log(determinant ratio)
-4
-2
theta
FIGURE 2.2. The proled log-likelihood as a function of = log() for the rails
example. Two of the components of the log-likelihood, log 1 , the log of the
length of the residual, and M
i=1 log / abs |11(i) | , the log of the determinant
ratios, are shown on the same scale.
75
logarithms approaches zero. Very small values of will have little eect
on the denominator so the term has the form

M
M
"
"
T
log |Z Ti Z i |.
log |Z i Z i | = M
i=1
i=1
That is, when this term approaches a linear function of , as can

be seen in Figure 2.2.
Because the log of the determinant ratio approaches a linear function of
as while the log of the residual norm tends to a nite asymptote,
the log-likelihood approaches a linear function of . When , and hence
, becomes large the log-likelihood will usually decrease then approach a
constant. This does not always occur, however. For some data sets, the
log-likelihood will continue to increase with as . In these cases,
the maximum likelihood estimator of b2 is zero.
Both in the log of the ratio of determinants term and in the log of the
norm of the penalized residual term, the eect of is determined by its
Values of that are either much less than
size relative to the Z i matrices.

or much greater than Z Ti Z i will produce a log-likelihood that is near
an asymptote. If there is to be
a maximum for nite it will have to be
M
near 0 = log 0 =
Z Ti Z i /M . In the case of the rails data
i=1 log
0 = 0.549.
2.2.5 Restricted Likelihood Estimation

Maximum likelihood estimates of variance components, such as 2 and
b2 in the rails example, tend to underestimate these parameters. Many
analysts prefer the restricted (or residual) maximum likelihood (REML)
estimates (Patterson and Thompson, 1971; Harville, 1977) for these quantities.
There are several ways to dene the REML estimation criterion. One
denition that provides a convenient computational form (Laird and Ware,
1982) is
LR (, 2 |y) =
L(, , 2 |y) d,
which, within a Bayesian framework, corresponds to assuming a locally

uniform prior distribution for the xed eects and integrating them out
of the likelihood.
76
Using (2.18) and the same change-of-variable techniques as in (2.16) gives

the log-restricted-likelihood

R (, 2 |y) = log LR , 2 y)
=

M
"
||
N p
c1 2
log(2 2 )
log
abs
|R
|
+
log
abs
.
00
2
2 2
|R11(i) |
i=1
2
() = c1 2 /(N p) for 2 ,
This produces the conditional estimate
R
from which we obtain the proled log-restricted-likelihood
2
R
()|y)
R (|y) = R (,
= const (N p) log c1 log abs |R00 | +
M
"
i=1

log abs
||
|R11(i) |

.
(2.23)
The components of the proled log-restricted-likelihood in (2.23) are similar to those in the proled log-likelihood (2.21) except that the log of the
norm of the residual vector has a dierent multiplier
and there
is an ex

M
T 1
tra determinant term of log abs |R00 | = log i=1 X i i X i /2. A plot
of the components of the proled log-restricted-likelihood versus for the
rails example would be similar in shape to Figure 2.2.
The evaluation of the restricted maximum likelihood estimates is done
by optimizing the proled log-restricted-likelihood (2.23) with respect to
R to obtain the REML estionly, and using the resulting REML estimate
2
2
mate of ,
R ( R ). Similarly, the REML estimated BLUPs of the random
R in (2.22).
eects are obtained by replacing with
In some ways, it is blurring the denition of the REML criterion to speak
of the REML estimate of . The REML criterion only depends on and .
However, it is still useful, and perhaps even sensible, to evaluate the best
R has been determined using the REML
guess at from (2.19) once
criterion.
An important dierence between the likelihood function and the restricted likelihood function is that the former is invariant to one-to-one
reparameterizations of the xed eects (e.g., a change in the contrasts representing a categorical variable), while the latter is not. Changing the X i
matrices results in a change in log abs |R00 | and a corresponding change in
R (|y). As a consequence, LME models with dierent xed-eects structures t using REML cannot be compared on the basis of their restricted
likelihoods. In particular, likelihood ratio tests are not valid under these
circumstances.
77
2.2.6 Multiple Levels of Random Eects

The likelihood function and the restricted likelihood function for multilevel
LME models can be calculated using the same techniques described for
the single-level model in 2.2.1, 2.2.3, and 2.2.5. We use the two-level
LME model to illustrate the basic steps in the derivation of the multilevel
likelihood function.
The likelihood for a model with two levels of random eects is dened
as in (2.3), but integrating over both levels of random eects
L(, 1 , 2 , 2 |y) =

Mi
M

p(y ij |bij , bi , , 2 ) p(bij | 2 , 2 ) dbij p(bi | 1 , 2 ) dbi .
i=1
j=1
(2.24)
As with a single level of random eects, we can simplify the integrals in
(2.24) if we augment the Z ij matrices with 2 and form orthogonaltriangular decompositions of these augmented arrays. This allows us to
evaluate the inner integrals. To evaluate the outer integrals we iterate this
process.
That is, we rst form and decompose the arrays

Z ij Z i,j X ij y ij
R22(ij) R21(ij) R20(ij) c2(ij)
,
= Q(ij)
0
R11(ij) R10(ij) c1(ij)
2
0
0
0
i = 1, . . . , M,
j = 1, . . . , Mi . (2.25)
The matrix R22(ij) will be an upper-triangular matrix of dimension q2 q2 .

The other arrays in the rst row of the decomposition in (2.25) are used

only if the conditional estimates ()
or the conditional modes
bij () and

bi () are required. The arrays in the second row of the decomposition:
R11(ij) , R10ij , and c1(ij) each have nij rows.
To evaluate the outer integral in (2.24) we again form and decompose an
augmented array
R11(i1)
..
R11(iM )
i
1
R10(i1)
..
.
R10(iMi )
0
c1(i1)

..
R11(i)
.
= Q(i)
0
c1(iMi )
0
R10(i)
R00(i)
c1(i)
c0(i)
i = 1, . . . , M. (2.26)
The nal decomposition to produce R00 , c0 , and c1 is the same as that
in (2.17).
Using the matrices and vectors produced in (2.25), (2.26), and (2.17) and
following the same steps as for the single level of nesting we can express
78
the proled log-likelihood for 1 and 2 as

1 , 2 ), 1 , 2 ,
2 ( 1 , 2 )|y)
(1 , 2 |y) = log L((

M
"
|1 |
log abs
= const N log c1 +
|R11(i) |
i=1

M
M
i
""
|2 |
log abs
+
.
|R22(ij) |
i=1 j=1
Similarly, the proled log-restricted-likelihood is
2
(1 , 2 ), 1 , 2 ,
R
( 1 , 2 )|y)
R (1 , 2 |y) = log LR (
R
= const (N p) log c1 log abs |R00 |

"

Mi
M
M "
"
|1 |
|2 |
log abs
log abs
+
+
.
|R11(i) |
|R22(ij) |
i=1
i=1 j=1
The calculation methods extend in the obvious way to Q nested levels of

random eects.
2.2.7 Parameterizing Relative Precision Factors

In a model with Q nested levels of random eects, there are Q symmetric,
positive-denite, variancecovariance matrices k , k = 1, . . . , Q. For computational purposes we express these in terms of relative precision factors
k , k = 1, . . . , Q that satisfy
Tk k = 2 1
k ,
k = 1, . . . , Q.
To optimize the log-likelihood or log-restricted-likelihood we express the

scaled variancecovariance matrices k / 2 , or equivalently the relative
precision factors k , as a function of unconstrained parameter vectors
k , k = 1, . . . , Q. For example, when is a scalar, as in 2.2.4, we use the
unconstrained parameter = log when optimizing the log-likelihood.
For the general case where k / 2 is a positive-denite, symmetric matrix
of size q q, we parameterize it through its matrix logarithm. To dene this
parameterization, we note that any positive-denite, symmetric matrix A
can be expressed as the matrix exponential of another symmetric matrix
B. This means that
B3
B2
+
+ ....
2!
3!
If A is the matrix exponential of B, then B is the matrix logarithm of A.
Suppose A is q q, symmetric and positive-denite. One way of evaluating its matrix logarithm B is to calculate an eigenvalue-eigenvector
decomposition
A = eB = I + B +
A = U U T ,
79
where is q q and diagonal while U is q q and orthogonal. If A is

positive-denite, then all the diagonal elements of must be positive. The
matrix logarithm of is the diagonal matrix whose diagonal elements are
the logarithms of the corresponding elements of . We will denote this by
log . Finally
B = log A = U log U T .
We dene k to be the elements of the upper triangle of the matrix
logarithm of k / 2 . This gives a nonredundant, unconstrained parameter
vector for k / 2 .
Other unconstrained parameterizations for k / 2 are used when the
matrix is required to have a special structure beyond being symmetric and
positive-denite. For example, if k / 2 is to be diagonal and positivedenite then the diagonal elements must all be positive and an unconstrained parameterization uses the logarithms of these diagonal elements.
2.2.8 Optimization Algorithms

Optimization of the proled log-likelihood or the proled log-restrictedlikelihood of an LME model is usually accomplished through EM iterations
or through NewtonRaphson iterations (Laird and Ware, 1982; Lindstrom
and Bates, 1988; Longford, 1993).
The EM algorithm (Dempster, Laird and Rubin, 1977) is a popular iterative algorithm for likelihood estimation in models with incomplete data.
The EM iterations for the LME model are based on regarding the random
eects, such as the bi , i = 1, . . . , M , as unobserved data. At iteration w
we use the current variancecovariance parameter vector, (w) , to evaluate the distribution of b|y and derive the expectation of the log-likelihood
for a new value of given this conditional distribution. Because we are
taking an expectation, this step is called the E step. The M step consists
of maximizing this expectation with respect to to produce (w+1) . Each
iteration of the EM algorithm results in an increase in the log-likelihood
function, though a possibly small increase. Ecient implementations of the
EM algorithm for LME models are described in Bates and Pinheiro (1998).
The NewtonRaphson algorithm (Thisted, 1988, 4.2.2) is one of the
most widely used optimization procedures. It uses a rst-order expansion
of the score function (the gradient of the log-likelihood function) around the
current estimate (w) to produce the next estimate (w+1) . Each Newton
Raphson iteration requires the calculation of the score function and its
derivative, the Hessian matrix of the log-likelihood. Under general conditions usually satised in practice, the NewtonRaphson algorithm converges quadratically. Because the calculation of the Hessian matrix at each
iteration may be computationally expensive, simple, quicker to compute approximations are sometimes used, leading to the so-called QuasiNewton
algorithms (Thisted, 1988, 4.3.3.4) .
80
Any iterative optimization algorithm requires initial values for the parameters. Because we can express both the proled log-likelihood and the
proled log-restricted-likelihood as a function of the parameters, we only
need to formulate starting values for when performing iterative optimization for LME models. These may be obtained from a previous t for similar
data, or derived from the current data. A general procedure for deriving
initial values for from the data being t is described in Bates and Pinheiro
(1998) and is implemented in the lme function.
Individual iterations of the EM algorithm are quickly and easily computed. Although the EM iterations generally bring the parameters into the
region of the optimum very quickly, progress toward the optimum tends
to be slow when near the optimum. NewtonRaphson iterations, on the
other hand, are individually more computationally intensive than the EM
iterations, and they can be quite unstable when far from the optimum.
However, close to the optimum they converge very quickly.
We therefore recommend a hybrid approach of starting with an initial
(0) , performing a moderate number of EM iterations, then switching to
NewtonRaphson iterations. Essentially the EM iterations can be regarded
as rening the starting estimates before beginning the more general optimization routine. The lme function implements such a hybrid optimization scheme. It begins by calculating initial estimates of the parameters,
then uses several EM iterations to get near the optimum, then switches to
NewtonRaphson iterations to complete the convergence to the optimum.
By default 25 EM iterations are performed before switching to Newton
Raphson iterations.
When tting an LME model, it is often helpful to monitor the progress
of the NewtonRaphson iterations to identify possible convergence problems. This is done by including an optional control argument in the call
to lme. The value of control should be a list that can contain any of several ags or settings for the optimization algorithm. One of these ags is
msVerbose. When it is set to TRUE or T, diagnostic output on the progress
of the NewtonRaphson iterations in the indirect call of the ms function
(Bates and Chambers, 1992, 10.2) is produced.
If we set this ag in the rst t for the rails example of 1.1, the diagnostic output is not very interesting because the EM iterations leave
the parameter estimates so close to the optimum that convergence of the
NewtonRaphson iterations is declared almost immediately.
> fm1Rail.lme <+
control
Iteration: 0 ,
Parameters:
[1] -1.8196
Iteration: 1 ,
lme( travel ~ 1, data = Rail, random = ~ 1 | Rail,

= list( msVerbose = TRUE ) )
1 function calls, F= 61.049
function calls, F=
61.049
2.3 Approximate Distributions
81
Parameters:
[1] -1.8196
Note that the parameter listed in the iteration output is

= log (
= log()
/
b ) = log(4.0208/24.805) = 1.8196
and this is the only parameter being directly controlled by the optimization
algorithm. The function labelled F in the iteration output is the negative of
the log-restricted-likelihood but without the constant term
N p
2 [log(N p) log(2) 1], which is 0.0396 when N = 18 and p =
1. Thus, the value of F = 61.049 at convergence corresponds to a loglikelihood of (61.049 + 0.0396) = 61.089. Because most optimization
algorithms are designed to minimize rather than maximize a function of
the parameters, we minimize the negative of the log-likelihood instead of
maximizing the log-likelihood.
If we eliminate the EM iterations altogether with another control argument, niterEM, we can observe the progress of the NewtonRaphson iterations for .
> fm1Rail.lme <- lme( travel ~ 1, data = Rail, random = ~ 1 | Rail,
+
control = list( msVerbose = TRUE, niterEM = 0 ))
Iteration: 0 , 1 function calls, F= 67.894
Parameters:
[1] -0.43152
Parameters:
[1] -2.0007
Parameters:
[1] -1.8028
Parameters:
[1] -1.8195
The algorithm converged to a slightly dierent value of , but with essentially the same value of the log-likelihood.
2.3 Approximate Distributions

Inference on the parameters of a linear mixed-eects model usually relies on
approximate distributions for the maximum likelihood estimates and the
restricted maximum likelihood estimates derived from asymptotic results.
Pinheiro (1994) has shown that, under certain regularity conditions generally satised in practice, the maximum likelihood estimates in the general LME model described in 2.1 are consistent and asymptotically normal. The approximate variancecovariance matrix for the maximum likelihood estimates is given by the inverse of the information matrix (Cox
82
and Hinkley, 1974, 4.8) corresponding to the log-likelihood function =

(, 1 , . . . , Q , 2 ). Because

$
%
E 2 /Tq = 0, q = 1, . . . , Q and E 2 / 2 = 0,
the information matrix corresponding to an LME model with Q levels of
nesting is block diagonal and, therefore, the maximum likelihood estimates
of the xed eects are asymptotically uncorrelated with the maximum
likelihood estimates of 1 , . . . , Q and 2 .
The approximate distributions of the maximum likelihood estimates in
an LME model with Q levels of nesting are

T

,
N , 2 R1
00 R00

1
1
..
..
.
N . , I 1 ( 1 , . . . , Q , ) ,
Q
(2.27)
log
log

2
2
2

log
T
1 T
2 T
1
1
1
..
..
..
,
I (1 , . . . , Q , ) =
.
.
.
2
2
2

1 log
2 log
2 log
where = (1 , . . . , Q , 2 ) now denotes the log-likelihood function proled
on the xed eects, I denotes the empirical information matrix, and R00
is dened as in (2.17). We use log in place of 2 in (2.27) to give an
unrestricted parameterization for which the normal approximation tends
to be more accurate.
As shown by Pinheiro (1994), the REML estimates in an LME model also
are consistent and asymptotically normal, with approximate distributions
identical to (2.27) but with replaced by the log-restricted-likelihood R
dened in 2.2.5.
In practice, the unknown parameters 1 , . . . , Q and 2 are replaced by
their respective ML or REML estimates in the expressions for the approximate variancecovariance matrices in (2.27). The approximate distributions for the maximum likelihood estimates and REML estimates are used
to produce hypothesis tests and condence intervals for the LME model
parameters, as described in 2.4.
2.4 Hypothesis Tests and Condence Intervals

After we have t a statistical model to the data we usually want to assess
the precision of the estimates and the signicance of various terms in the
83
model or to compare how well one model ts the data relative to another
model. This section presents approximate hypothesis tests and condence
intervals for the parameters in an LME model.
2.4.1 Likelihood Ratio Tests

A general method for comparing nested models t by maximum likelihood
is the likelihood ratio test (Lehmann, 1986, 1.7). Such a test can also be
used with models t by REML, but only if both models have been t by
REML and if the xed-eects specication is the same for both models.
One statistical model is said to be nested within another model if it
represents a special case of the other model. For example, in the analysis
of the Machines data described in 1.3, we t one model, fm1Machine, with
a random eect for Worker only, then we t a second model, fm2Machine,
with random eects for Worker and for Machine %in% Worker. The model
fm1Machine is nested within fm2Machine because it represents a special case
of fm2Machine in which the variance of the Machine %in% Worker interaction
term is zero.
If L2 is the likelihood of the more general model (e.g., fm2Machine) and
L1 is the likelihood of the restricted model (e.g., fm1Machine) we must have
L2 > L1 and, correspondingly, log L2 > log L1 . The likelihood ratio test
(LRT) statistic
2 log(L2 /L1 ) = 2[log(L2 ) log(L1 )]
will be positive. If ki is the number of parameters to be estimated in model i,
then the asymptotic, or large sample, distribution of the LRT statistic,
under the null hypothesis that the restricted model is adequate, is a 2
distribution with k2 k1 degrees of freedom.
In Chapter 1 we show several examples of likelihood ratio tests performed with the anova function. When given two arguments representing
ts of nested models, this function displays the LRT statistic in the L.Ratio
column and gives the p-value from the 2k2 k1 distribution. The column labelled df is the number of parameters in each model. For example, using
model ts described in 1.3, we would have
> anova( fm1Machine, fm2Machine )
Model df
AIC
BIC logLik
fm1Machine
1 5 300.46 310.12 -145.23
fm2Machine
2 6 231.27 242.86 -109.64 1 vs 2 71.191 <.0001
The anova function also displays the values of the Akaike Information
Criterion (AIC ) (Sakamoto et al., 1986) and the Bayesian Information
Criterion (BIC ) (Schwarz, 1978). As mentioned in 1.1.1, these are model
84
comparison criteria evaluated as

AIC = 2 |y
+ 2npar ,

BIC = 2 |y
+ npar log(N )
(2.28)
for each model, where npar denotes the number of parameters in the model.
Under these denitions, smaller is better. That is, if we are using AIC to
compare two or more models for the same data, we prefer the model with
the lowest AIC. Similarly, when using BIC we prefer the model with the
lowest BIC. The REML versions of the AIC and the BIC simply replace

in (2.28).
(|y)
by R (|y)
We will generally use likelihood-ratio tests to evaluate the signicance
of terms in the random-eects structure. That is, we t dierent nested
models in which the random-eects structure changes and apply likelihoodratio tests. Stram and Lee (1994), using the results of Self and Liang (1987),
argued that tests on the random eects structure conducted in this way can
be conservative. That is, the p-value calculated from the 2k2 k1 distribution
is greater than it should be. As Stram and Lee (1994) explain, changing
from the more general model to the more specic model involves setting the
variance of certain components of the random eects to zero, which is on
the boundary of the parameter region. The asymptotic results for likelihood
ratio tests have to be adjusted for boundary conditions. In the next section
we use simulations to demonstrate the eect of these adjustments.
Simulating Likelihood Ratio Test Statistics
One way to check on the distribution of the likelihood ratio test statistic
under the null hypothesis is through simulation. The simulate.lme function
takes two model specications, the null model and the alternative model.
These may be given as lme objects corresponding to each model, or as
lists of arguments used to produce such ts. In the latter case, only those
characteristics that change between the two models need to be specied in
the argument list for the alternative model.
For example, in the analysis of the OrthoFem data presented in 1.4.1, the
fm1OrthF t to the OrthoFem data has the specication
> fm1OrthF <- lme( distance ~ age, data = OrthoFem,
+
random = ~ 1 | Subject )
while fm2OrthF is t as
> fm2OrthF <- update( fm1OrthF, random = ~ age | Subject )
Both models correspond to X i matrices of

X 1 = X 2 = = X 11
1 8
1 10
=
1 12
1 14
85
with a two-dimensional vector. In fm1OrthF the Z i matrices are

1
1
Z 1 = Z 2 = = Z 11 =
1
1
and the bi are one-dimensional random vectors
fm2OrthF the Z i matrices are
1
1
Z 1 = Z 2 = = Z 11 =
1
1
with variance = 12 . In
8
10
12
14
and the bi are two-dimensional random vectors with variancecovariance

matrix

2
1 1 2
.
=
1 2 22
In the terminology of hypothesis tests, fm1OrthF is the null model and
fm2OrthF is the alternative model. In this case, the null model is a special
case of the alternative model, with one fewer random eect. The model
being t as fm1OrthF is obtained from the model for fm2OrthF by requiring
the last row and column of the 2 2 to be zero. Although there are
three distinct entries in this row and column, these entries are determined
by only two parameters because must be symmetric. Notice that one of
these entries, 22 , must be non-negative so setting it to zero corresponds to
a boundary condition.
To simulate the likelihood ratio test statistic comparing model fm1OrthF
to model fm2OrthF we generate data according to the null model using
the parameter values from fm1OrthF. We then t both the null and the
alternative model to each set of simulated data and calculate the likelihood
ratio test statistic. This is repeated for nsim cases. By doing this we obtain
an empirical distribution of the likelihood ratio test statistic under the null
hypothesis. We can then compare the empirical distribution to dierent 2
distributions as in Figure 2.3, which is produced by
> orthLRTsim <- simulate.lme( fm1OrthF, fm2OrthF, nsim = 1000 )
> plot( orthLRTsim, df = c(1, 2) )
Figure 2.3 is a probabilityprobability plotsimilar to a quantilequantile

plot but on the p-value scale, rather than on the scale of the likelihood ratio
test (LRT) statistic. The nominal p-values for the simulated LRT statistics,
under 2 distributions with 1 and 2 degrees of freedom and an equal-weight
mixture of those 2 distributions (denoted Mix(1,2) in Figure 2.3), for both
86

0.0
REML
df=1
0.4
0.8
REML
Mix(1,2)
REML
df=2
1.0
0.8
Nominal p-value
0.6
0.4
0.2
ML
df=1
1.0
ML
Mix(1,2)
0.0
ML
df=2
0.8
0.6
0.4
0.2
0.0
0.0
0.4
0.8
0.0
0.4
0.8
Empirical p-value
FIGURE 2.3. Plots of the nominal versus empirical p-values for the likelihood
ratio test statistic comparing two models for the orthodontic data, female subjects
only. The null model, fm1OrthF, has a random eect for the intercept only. The
alternative model, fm2OrthF, has random eects for both the intercept and the
slope. The null model was simulated 1000 times, both models were t to the
simulated data, and the likelihood ratio test statistic was calculated for both
maximum likelihood and REML estimates. In each panel, the nominal p-values
for the LRT statistics under the corresponding distribution are plotted versus the
empirical p-values.
ML and REML estimation, are plotted versus the empirical p-values, obtained from the empirical distribution of the simulated LRT statistics.
For both REML and ML estimates, the nominal p-values for the LRT
statistics under a 2 distribution with 2 degrees of freedom are much greater
than the empirical p-values. This is the sense in which the likelihood ratio
test using 22 for the reference distribution will be conservativethe actual
p-value is smaller than the p-value that is reported. Stram and Lee (1994)
suggest a 0.521 + 0.522 mixture as a reference distribution, which is conrmed in Figure 2.3, for both ML and REML estimation. A 21 appears to
be anti-conservative in the sense that the nominal p-values are smaller
than the empirical p-values.
The adjustment suggested by Stram and Lee (1994) is not always this
successful. According to this adjustment, the null distribution of the likelihood ratio test statistic for comparing fm1Machine to fm2Machine should
have approximately a 0.520 + 0.521 mixture distribution, where 20 represents a distribution with a point mass at 0. When simulated
> machineLRTsim <- simulate.lme(fm1Machine, fm2Machine, nsim= 1000)
Nominal p-value
0.0
ML
Mix(0,1)
0.2
0.4
0.6
0.8
1.0
0.0
ML
df=1
REML
Mix(0,1)
0.2
0.4
0.6
87
0.8
1.0
REML
df=1
1.0
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Empirical p-value
FIGURE 2.4. Plots of the nominal versus empirical p-values for the likelihood ratio test statistic comparing two models for the Machines data. The null
model, fm1Machine, has a random eect for Worker only. The alternative model,
fm2Machine, has random eects for the Worker and a random interaction for
Machine %in% Worker. Both models were t to 1000 sets of data simulated from
the null model and the likelihood ratio test statistics were calculated.
> plot( machineLRTsim, df = c(0, 1),

+ layout = c(4,1), between = list(x = c(0, 0.5)) )
it produces a distribution for the LRT statistics that closely agrees with the
equal-weight mixture in the REML case, but which resembles a 0.6520 +
0.3521 mixture in the ML case.
It is dicult to come up with general rules for approximating the distribution of the LRT statistic for such nested mixed-eects models. The
naive approach of using a 2 distribution with the number of degrees of
freedom determined by the dierence in the number of nonredundant parameters in the models as the reference is easily implemented and tends to
be conservative. This is the reference distribution we use to calculate the
p-values quoted in the multiargument form of anova. One should be aware
that these p-values may be conservative. That is, the reported p-value may
be greater than the true p-value for the test and, in some cases, it may be
much greater.
2.4.2 Hypothesis Tests for Fixed-Eects Terms

When two nested models dier in the specication of their xed-eects
terms, a likelihood ratio test can be dened for maximum likelihood ts
only. As described in 2.2.5 a likelihood ratio test for REML ts is not
feasible, because there is a term in the REML criterion that changes with
the change in the xed-eects specication.
Even though a likelihood ratio test for the ML ts of models with dierent
xed eects can be calculated, we do not recommend using such tests. Such
88
Nominal p-value
0.0
ML
df=3
1.0
0.4
0.8
ML
Mix(3,4)
ML
df=4
0.8
0.6
0.4
0.2
0.0
0.0
0.4
0.8
0.0
0.4
0.8
Empirical p-value
ratio test statistic comparing two models for the ergoStool data. The alternative
model has a xed eect for Type but the null model does not. The random eects
specications are the same. Both models were t to 1000 sets of data simulated
from the null model and the likelihood ratio test statistics from the maximum
likelihood estimates were calculated.
likelihood ratio tests using the standard 2 reference distribution tend to

be anticonservativesometimes quite badly so.
As an example, consider the ergoStool example analyzed in 1.2.1. Suppose we compare fm1Stool, the model for the ergoStool data with a xed
eect for the Type factor, to a model without a xed eect for the Type
factor.
> stoolLRTsim <+
simulate.lme( m1 = list(fixed = effort ~ 1, data = ergoStool,
+
random = ~ 1 | Subject),
+
m2 = list(fixed = effort ~ Type),
+
method = "ML", nsim = 1000 )
> plot( stoolLRTsim, df = c(3, 4) )
# Figure 2.5
We can see from Figure 2.5 that, at 3 degrees of freedom, which is the
dierence in the number of parameters in the two models, the 2 distribution gives p-values that are anticonservative. At 4 degrees of freedom the
p-values will be conservative. The nominal p-values for the equal-weight
mixture of 23 and 24 distributions, represented in the middle panel of
Figure 2.5, are in close agreement with the empirical p-values.
In this case the slight anticonservative nature of the reported p-values
may not be too alarming. However, as the number of parameters being
removed from the xed eects becomes large, compared to the total number
of observations, this inaccuracy in the reported p-values can be substantial.
For example, Littell, Milliken, Stroup and Wolnger (1996, 1.5) provide
analyses of data from a partially balanced incomplete block (PBIB) design.
The design is similar to the randomized block design in the ergometric
experiment described in 1.2 except that not every level of the treatment
Nominal p-value
0.0
ML
df=14
1.0
0.4
89
0.8
ML
df=16
ML
df=18
0.8
0.6
0.4
0.2
0.0
0.0
0.4
0.8
0.0
0.4
0.8
Empirical p-value
ratio test statistic comparing two models for the PBIB data. The alternative model
has a xed eect for Type, but the null model does not. The random eects
specications are the same. Both models were t to 1000 sets of data simulated
from the null model, and the likelihood ratio test statistics from the maximum
likelihood estimates were calculated.
appears with every level of the blocking factor. This is the sense in which
it is an incomplete block design. It is partially balanced because every
pair of treatments occur together in a block the same number of times.
These data are described in greater detail in Appendix A.22 and are given
as an object called PBIB that is available with the nlme library.
The important point with regard to the likelihood ratio tests is that there
are 15 levels of the Treatment factor and only 60 observations in total. The
blocking factor also has 15 levels. If we simulate the likelihood ratio test
and plot the p-values calculated from the 214 distribution,
> pbibLRTsim <+
simulate.lme( m1 = list( fixed = response ~ 1, data = PBIB,
+
random = ~ 1 | Block ),
+
m2 = list( fixed = response ~ Treatment ),
+
method = "ML", nsim = 1000 )
> plot( pbibLRTsim, df = c(14,16,18), weights = FALSE ) # Figure 2.6
we can see, from Figure 2.6, that the p-values calculated using 214 as the
reference distribution are seriously anticonservative.
Another, perhaps more conventional, approach to performing hypothesis
tests involving terms in the xed-eects specication is to condition on
As
the estimates of the random eects variancecovariance parameters, .
described in 2.2.1, for a xed value of , the conditional estimates of

the xed eects, (),
are determined as standard least-squares estimates.
The approximate distribution of the maximum likelihood or the REML
estimates of the xed eects in (2.27) is exact for the conditional estimates

().
90
Conditional tests for the signicance of a term in the xed-eects specication are given by the usual F-tests or t-tests for linear regression models,
based on the usual (REML) conditional estimate of the variance
2
() = s2 =
R
c1 2
RSS
=
.
N p
N p
In practice, the unknown parameter vector is replaced by its maximum

likelihood estimate or its REML estimate so the conditional tests hold only
approximately.
The conditional t-tests are included in the output of the summary method
applied to lme objects. They test the marginal signicance of each xed
eect coecient when all other xed eects are present in the model. For
example the t-tests for the fm2Machine model
> summary( fm2Machine )
. . .
Fixed effects: score ~ Machine
(Intercept) 59.650
2.1447 36 27.813 <.0001
Machine1 3.983
1.0885 10
3.660 0.0044
Machine2 3.311
0.6284 10
5.269 0.0004
. . .
indicate that all the xed-eects terms are signicant.

The conditional F-tests are implemented in the single-argument form of
the anova method for tted models from lme. They test the signicance
of terms in the xed eects model, which may include several coecients.
By default, the terms are tested sequentially in the order they enter the
model, but the argument type to anova can be used to specify marginal
F-tests. For example, to jointly test the signicance of all 14 coecients
corresponding to Treatment in the PBIB example we use
> fm1PBIB <- lme( response ~ Treatment, data = PBIB, random = ~ 1 )
> anova( fm1PBIB )
(Intercept)
1
31 1654.2 <.0001
Treatment
14
31
1.5 0.1576
We will compare this result, a p-value of 15.8%, with that from the
likelihood ratio test. Because a likelihood ratio test for terms in the xedeects specication must be done on ML ts, we rst re-t fm1PBIB using
maximum likelihood, then modify the model.
> fm2PBIB <- update( fm1PBIB, method = "ML" )
> fm3PBIB <- update( fm2PBIB, response ~ 1 )
> anova( fm2PBIB, fm3PBIB )
Model df
AIC
BIC logLik
fm2PBIB
1 17 56.571 92.174 -11.285
fm3PBIB
2 3 52.152 58.435 -23.076 1 vs 2 23.581 0.0514
91
The simulation illustrated in Figure 2.6 shows that the 15.8% p-value from
the conditional F-test is much more realistic than the 5.1% p-value from
the likelihood ratio test.
For this reason, we prefer the conditional F-tests and t-tests for assessing
the signicance of terms in the xed eects.
These conditional tests for xed-eects terms require denominator degrees of freedom. In the case of the conditional F-tests, the numerator
degrees of freedom are also required, being dened by the term itself. The
denominator degrees of freedom are determined by the grouping level at
which the term is estimated. A term is called inner relative to a grouping
factor if its value can change within a given level of the grouping factor.
A term is outer to a grouping factor if its value does not change within
levels of the grouping factor. A term is said to be estimated at level i, if it
is inner to the i 1st grouping factor and outer to the ith grouping factor.
For example, the term Machine in the fm2Machine model is outer to Machine
%in% Worker and inner to Worker, so it is estimated at level 2 (Machine %in%
Worker). If a term is inner to all Q grouping factors in a model, it is estimated at the level of the within-group errors, which we denote as the
Q + 1st level.
The intercept, which is the parameter corresponding to the column of 1s
in the model matrices X i , is treated dierently from all the other parameters, when it is present. As a parameter it is regarded as being estimated
at level 0 because it is outer to all the grouping factors. However, its denominator degrees of freedom are calculated as if it were estimated at level
Q + 1. This is because the intercept is the one parameter that pools information from all the observations at a level even when the corresponding
column in X i doesnt change with the level.
Letting mi denote the total number of groups in level i (with the convention that m0 = 1 when the xed eects model includes an intercept
and 0 otherwise, and mQ+1 = N ) and pi denote the sum of the degrees
of freedom corresponding to the terms estimated at level i, the ith level
denominator degrees of freedom is dened as
denDF i = mi (mi1 + pi ) ,
i = 1, . . . , Q + 1.
This denition coincides with the classical decomposition of degrees of freedom in balanced, multilevel ANOVA designs and gives a reasonable approximation for more general mixed-eects models.
For example, in the fm2Machine model, Q = 2, m0 = 1, m1 = 6, m2 = 18,
m3 = 54, p0 = 1, p1 = 0, p2 = 2, and p3 = 0, giving denDF 1 = 5,
denDF 2 = 10, and denDF 3 = 36.
> anova( fm2Machine )
(Intercept)
1
36 773.57 <.0001
Machine
2
10
20.58
3e-04
92
Because Machine is estimated at level 2, its denominator degrees of freedom

is 10.
Another example is provided by the analysis of the Oats data, presented
in 1.6, which shows an anova of the form
> anova( fm2Oats )
(Intercept)
1
51 245.14 <.0001
ordered(nitro)
3
51
41.05 <.0001
Variety
2
10
1.49 0.2724
In this example, Q = 2, m0 = 1, m1 = 6, m2 = 18, m3 = 72, p0 = 1, p1 = 0,

p2 = 2, and p3 = 3, giving denDF 1 = 5, denDF 2 = 10, and denDF 3 = 51.
The nitro factor changes within levels of the rst-level grouping factor,
Block, and within levels of the second-level grouping factor, Variety %in%
Block. Thus, it is inner to each of these grouping factors and is estimated at
level 3, with 51 denominator degrees of freedom. By denition, the Variety
factor cannot change within levels of Variety %in% Block, but it changes
within levels of Block. It is therefore outer to Variety %in% Block and inner
to Block, being estimated at level 2 with 10 denominator degrees of freedom.
2.4.3 Condence Intervals

Approximate condence intervals on the variancecovariance components
and the xed eects are obtained using the approximate distributions for
the maximum likelihood estimates and the REML estimates described in
2.3 and the conditional t-tests described in 2.4.2.
Letting dfj denote the denominator degrees-of-freedom for the condi an
tional t-test corresponding to the the jth xed eect based on the ,
approximate condence interval of level 1 for j is

T

R1
R
,
j tdfj (1 /2)
00 R00
jj
where tdfj (1 /2) denotes the 1 /2 quantile of the t-distribution with

R denotes the REML estimate of . The matrix
dfj degrees of freedom and
R00 is evaluated at the estimated value of .
Condence intervals on the within-group standard deviation
% are ob$
tained from the approximate distribution in (2.27). Letting I 1 represent the last diagonal element of the inverse empirical information matrix
dened in (2.27), an approximate condence interval of level 1 for is

$
$
%
%
1
1
,
exp z(1 /2) I
,
exp z(1 /2) I
where z(1 /2) denotes the 1 /2 quantile of the standard normal

distribution. This condence interval formulation works for both ML and
REML estimates, with the obvious modications.
93
Condence intervals on the variancecovariance components for the random eects are a bit trickier to obtain. In practice, one is interested in
getting condence intervals on the original scale of the elements of and
not in the scale of the unconstrained parameters used in the optimization.
For some simple forms of , such as a diagonal structure or a multiple of
the identity structure, it is easy to transform the condence intervals obtained in the unconstrained scale (the logarithm of the standard deviations
in the two examples mentioned) back to the original parameter scale (by exponentiating the condence limits in the case of the diagonal and multiple
of the identity structures).
In the case of a general positive-denite , however, usually it is not
possible to transform back to the original scale the condence intervals
obtained for the unconstrained parameter used in the optimization. This
is true, for example, for the matrix logarithm parameterization described
in 2.2.7, when the dimension of is greater than one.
The approach used in lme is to consider a dierent parameterization
for general positive-denite when calculating condence intervals. This
parameterization, which we call the natural parameterization, uses the logarithm of the standard deviations and the generalized logits of the correlations. For a given correlation parameter 1 < < 1, its generalized logit
is log[(1 + )/(1 )] which can take any value in the real line. We denote
by the parameter vector determining the natural parameterization. The
elements of are individually unconstrained, but not jointly so. Therefore,
the natural parameterization cannot be used for optimization. However,
the elements of can be individually transformed into meaningful parameters in the original scale, so it is a useful parameterization for constructing
condence intervals.
If j corresponds to the logarithm of a standard deviation in and letting [I 1 ]jj denote its associated diagonal element in the inverse empirical
information matrix, an approximate level 1 condence interval for the
corresponding standard deviation is

$
$

%
%
exp j z(1 /2) I 1 jj , exp j + z(1 /2) I 1 jj .
An approximate condence interval for a correlation coecient represented

by j in the natural parameter vector is

$
$
%
%
exp j z(1 2 ) I 1 jj 1 exp j + z(1 2 ) I 1 jj 1
.

$
$
,
%
%
exp j z(1 2 ) I 1 jj + 1 exp j + z(1 2 ) I 1 jj + 1
94
2.5 Fitted Values and Predictions

Fitted values, which are the predicted values for the observed responses
under the tted model, are often of interest for model checking. Predicted
values for new observations are one of the primary quantities of interest
when making inferences from a tted model.
In a mixed-eects model, tted values and predictions may be obtained
at dierent levels of nesting, or at the population level. Population level
predictions estimate the marginal expected value of the response. For example, letting xh represent a vector of xed eects covariates, the marginal
expected value of the corresponding response yh is
E [yh ] = xTh .
(2.29)
Predicted values at the kth level of nesting estimate the conditional expectation of the response, given the random eects at levels k. For example, letting z h (i) denote a vector of covariates corresponding to random
eects associated with the ith group at the rst level of nesting, the level-1
predictions estimate
E [yh (i)|bi ] = xTh + z h (i)T bi .
(2.30)
Similarly, letting z h (i, j) denote a covariate vector associated with the jth
level-2 group within the ith level-1 group, the level-2 predicted values estimate
E [yh (i)|bi , bij ] = xTh + z h (i)T bi + z h (i, j)T bij .
(2.31)
This extends naturally to an arbitrary level of nesting.

The Best Linear Unbiased Predictors (BLUP s) of the population expected values and the conditional expectations given the random eects are
obtained by replacing, in the expressions dening the expectations, with

its conditional estimate ()
and the random eects with their BLUPs.
For example, the BLUPs corresponding to the expected values in (2.29),
(2.30), and (2.31) are

yh = xTh ()
T
yh (i) = xh () + z h (i)T
bi ()
T
T
yh (i, j) = xh () + z h (i) bi () + z h (i, j)T
bij ().
In practice, the unknown parameter vector is replaced by its maximum
likelihood estimate or its REML estimate, producing estimated BLUPs of
the expected values.
2.6 Chapter Summary

This chapter presents the theory and computational methods for linear
mixed-eects models. We express linear mixed-eects models in the Laird
2.6 Chapter Summary
95
Ware formulation. For a single grouping factor that divides the observations
into M groups of ni , i = 1, . . . , M observations, the model is written
y i = X i + Z i bi + i ,
bi N (0, ),
i = 1, . . . , M,
i N (0, 2 I).
The parameters in the model are the p-dimensional xed eects, , the qq
variancecovariance matrix, , for the random eects, and the variance 2
of the noise i . We estimate these parameters by maximum likelihood
(ML) or by restricted maximum likelihood (REML).
Although the random eects bi , i = 1, . . . , M are not formally parameters
in the model, we will often want to formulate our best guess for these
values given the data. We use the Best Linear Unbiased Predictors (BLUPs)
for this.
For computational purposes the variancecovariance matrix is reexpressed in terms of the relative precision factor which satises
T = / 2
and the matrix is expressed as a function of an unconstrained parameter
vector .
The proled log-likelihood function with respect to can be easily calculated using matrix decompositions. That is, the log-likelihood correspond
ing to the conditionally best estimates ()
and
2 () can be evaluated as a
function of alone. This simplies the problem of optimizing the likelihood
to get maximum likelihood estimates because it reduces the dimension of
the optimization. The same simplication applies to REML estimation.
We describe approximate distributions for the maximum likelihood estimates and the REML estimates using results from asymptotic theory for
linear mixed-eects models.
We compare models that dier in the random eects specication by
likelihood ratio tests or by simulation-based parametric bootstrap evaluations.
We assess the signicance of terms in the xed-eects specication by
These tests
standard linear regression tests conditional on the value of .
include t-tests for individual coecients or F-tests for more complicated
terms or linear combinations of coecients. The degrees of freedom for
a t-test (or the denominator degrees of freedom for an F-test) depend on
whether the factor being considered is inner to the grouping factor (changes
within levels of the grouping factor) or outer to the grouping factor (is
invariant within levels of the grouping factor).
Approximate condence intervals for the xed eects and the variance
covariance parameters are produced from the approximate distributions of
the maximum likelihood estimates and REML estimates.
96
All these results extend to multiple nested levels of random eects. A

model with two levels of nested random eects, for example, is written
bi N (0, 1 ),
i = 1, . . . , M,
bij N (0, 2 ),
j = 1, . . . , Mi ,
ij N (0, 2 I).
The two variancecovariance matrices 1 and 2 are written in terms

of relative precision factors 1 and 2 , parameterized by unconstrained
parameter vectors 1 and 2 . The proled log-likelihood or the proled logrestricted-likelihood, a function of 1 and 2 only, is maximized to produce

&1 , and
&2 .
the estimates ,
2 ,
Exercises
1. The simulation results presented in Figure 2.4 (p. 87) indicate that
the null distribution of the REML likelihood ratio test statistic comparing a null model with a single level of scalar random eects to
an alternative model with nested levels of scalar random eects is
approximately an equally weighted mixture of a 20 and a 21 .
Conrm this result by simulating a LRT statistic on the Oats data,
considered in 1.6. The preferred model for those data, fm4Oats, was
dened with random = ~Block/Variety. Re-t this model with random
= ~Block. Using this t as the null model and fm4Oats as the alternative model, obtain a set of simulated LRT statistics with simulate.lme.
Plot these simulated LRT statistics setting df = c(0,1) to obtain a
plot like Figure 2.4. Are the conclusions from this simulation similar
to those from the simulation shown in Figure 2.4?
Note that simulate.lme must t both models to nsim simulated sets
of data. By default nsim = 1000, which could tie up your computer
for a long time. You may wish to set a lower value of nsim if the
default number of simulations will take too long.
3
Describing the Structure of
Grouped Data
As illustrated by the examples in Chapter 1, we will be modeling data from

experiments or studies in which the observations are grouped according to
one or more nested classications. Often this classication is by Subject
or some similar experimental unit. Repeated measures data, longitudinal
data, and growth curve data are examples of this general class of grouped
data.
A common and versatile way of organizing data in S is as data.frame
objects. These are in the form of tables where each row corresponds to an
observation and each column corresponds to one of the variables being observed. We extend the data.frame class to the class of groupedData objects,
which are data frames with additional information about the grouping of
the observations and, possibly, other special roles of some variables.
In this chapter we describe creating, summarizing and displaying groupedData objects with a single level of grouping or with multiple levels of grouping.
3.1 The Display Formula and Its Components

A groupedData object contains the data values themselves, stored as a data
frame, and a formula that designates special roles for some of the variables
in the data frame. The most important of the special roles is that of a
grouping factor that divides the observations into the distinct groups of
98
3. Describing the Structure of Grouped Data
observations. The formula also designates a response and, when available,

a primary covariate. It is given as
response ~ primary | grouping
where response is an expression for the response, primary is an expression

for the primary covariate, and grouping is an expression for the grouping
factor. Most often these expressions are simply the name of a variable in
the data frame, but they could also be functions of one or more variables.
For example, log(conc) would be a legitimate expression for the response
if conc is one of the variables in the data frame.
The formula function extracts the formula from a grouped data object.
Applied to some of the data sets used in Chapter 1 it produces
> formula( Rail )
travel ~ 1 | Rail
> formula( ergoStool )
effort ~ Type | Subject
> formula( Machines )
score ~ Machine | Worker
> formula( Orthodont )
distance ~ age | Subject
> formula( Pixel )
pixel ~ day | Dog/Side
> formula( Oats )
yield ~ nitro | Block
Notice that there is not primary covariate in the Rail data so we use the
constant expression 1 in that position in the formula. In data with multiple, nested grouping factors, such as the Pixel data, the grouping factors
are separated by /. Factors that are nested within other factors appear
further to the right so an expression like Dog/Side indicates that Side is
nested within Dog.
The formula of a grouped data object has the same pattern as the formula used in a call to a trellis graphics function, such as xyplot. This is
intentional. Because such a formula is available with the data, the plot
method for objects in the groupedData class can produce an informative
trellis display from the object alone. It may, in fact, be best to think of the
formula stored with the data as a display formula for the data because it
provides a meaningful default graphical display method for the data.
The formula function shown above is an example of an extractor function
for this class. It returns some property of the objectthe display formula
in this casewithout requiring the user to be aware of how that property is
stored. We provide other extractor functions for each of the components of
the display formula. The getGroups extractor returns the value of the grouping factor. A companion function, getGroupsFormula, returns the formula
that is evaluated to produce the grouping factor. The extractors for the
other components of the display formula are getResponse and getCovariate,
3.1 The Display Formula and Its Components

-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
24
22
12
13
14
19
15
20
18
23
11
99
1.0
170
160
150
140
130
21
Height (cm)
170
160
150
140
130
10
26
25
17
16
170
160
150
140
130
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
0.0
1.0
Centered age
FIGURE 3.1. Heights of 26 boys from Oxford, England, each measured on nine
occasions. The ages have been centered and are in an arbitrary unit.
which return numeric vectors or factors, and getResponseFormula and

getCovariateFormula, which return formulas.
It is safer to use these extractor functions instead of checking the display
formula for the object and extracting a variable from the object. For example, suppose we wish to check for balance in the Oxboys data, shown in
Figure 3.1, and consisting of the heights at dierent ages of several boys
100
from Oxford, England (see Appendix A.19 for more detail). We could use
the table function on the grouping factor,
> table( Oxboys$Subject )
10 26 25 9 2 6 7 17 16 15 8 20 1 18 5 23 11 21 3 24 22 12 13
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
14 19 4
9 9 9
to check if each boys height is recorded the same number of times. To do

this, we must know that the grouping factor for the Oxboys data is named
Subject. A form that is easier to remember, and also more general, is
> table( getGroups( Oxboys ) )
10 26 25 9 2 6 7 17 16 15 8 20 1 18 5 23 11 21 3 24 22 12 13
9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
14 19 4
9 9 9
> unique( table( getGroups( Oxboys ) ) ) # a more concise result
[1] 9
Because there are exactly nine observations for each subject, the data are
balanced with respect to the number of observations. However, if we also
check for balance in the covariate values, we nd they are not balanced.
> unique( table( getCovariate( Oxboys ), getGroups( Oxboys ) ) )
[1] 1 0
> length( unique( getCovariate( Oxboys ) ) )
[1] 16
Further checking reveals that there are 16 unique values of the covariate age.
The boys are measured at approximately the same ages, but not exactly the
same ages. This imbalance could aect some analysis methods for repeated
measures data. It does not aect the methods described in this book.
The isBalanced function in the nlme library can be used to check a
groupedData object for balance with respect to the grouping factor(s) or
with respect to the groups and the covariate. It is built from calls to
getGroups and table like those above.
When applied to data with multiple, nested grouping factors, the getGroups extractor takes an optional argument level. Levels are counted from
the outside inward so, in the Pixel data where the grouping is Dog/Side,
Dog is the rst level and Side is the second level. When the argument level
species a single level the result is returned as a vector
> unique( getGroups(Pixel, level = 1) )
[1] 1 2 3 4 5 6 7 8 9 10
> unique( getGroups(Pixel, level = 2) )
[1] 1/R 2/R 3/R 4/R 5/R 6/R 7/R 8/R
1/R < 2/R < 3/R < 4/R < 5/R < ...
9/R
10/R 1/L
...
3.2 Constructing groupedData Objects
101
Notice that the groups at level = 2, the Side within Dog factor, are
coerced to an ordered factor with distinct levels for each combination of
Side within Dog.
If we extract the groups for multiple levels the result is returned as a
data frame with one column for each level. Any inner grouping factors are
preserved in their original form in this frame rather than being coerced to
an ordered factor with distinct levels as above. For example,
> Pixel.groups <- getGroups( Pixel, level = 1:2 )
> class( Pixel.groups )
[1] "data.frame"
> names( Pixel.groups )
[1] "Dog"
"Side"
> unique( Pixel.groups[["Side"]] )
[1] R L
In a call to a linear mixed-eects modeling function, lme or lmList, or

to a nonlinear mixed-eects modeling function, nlsList or nlme (discussed
in Chapter 8), the default values for the response, for a covariate, and for
the grouping factor are obtained from the formula stored with the data.
These are, however, only the default values. They can be overridden with an
explicit model formula. For example, during the course of model building
we may wish to change our idea of what constitutes the response, say
by transforming from a measure of concentration to the logarithm of the
concentration. It is not necessary to change the formula stored with the data
when doing this. We can use an explicit model formula as an argument to
the model tting function and override the formula stored with the data.
If we do decide to make a permanent change in the formula of a groupedData object, we can use the update function to do this. For example, we
could change the covariate in the PBG data (discussed in 3.2.1 and Appendix A.21) from dose to log(dose) by updating the object.
> formula( PBG )
deltaBP ~ dose | Rabbit
> PBG.log <- update( PBG, formula = deltaBP ~ log(dose) | Rabbit )
> formula(PBG.log)
deltaBP ~ log(dose) | Rabbit
> unique( getCovariate(PBG.log) )
[1] 1.8326 2.5257 3.2189 3.9120 4.6052 5.2983
> unique( getCovariate(PBG) )
[1]
6.25 12.50 25.00 50.00 100.00 200.00

Constructing a groupedData object requires the data to be available as a
data frame. There are several ways that data can be imported into S and
102
formed into a data frame. One of the simplest ways is using the read.table
function on data stored in an external le (Venables and Ripley, 1999, 2.4).
For example, if the Oxford boys height data are stored in a text le
named oxboys.dat of the form
Subject
1
1
1
1
1
1
1
1
1
2
26
26
26
26
26
age
-1.0000
-0.7479
-0.4630
-0.1643
-0.0027
0.2466
0.5562
0.7781
0.9945
-1.0000
...
-0.0027
0.2466
0.5562
0.7781
1.0055
height
140.50
143.40
144.80
147.10
147.70
150.20
151.70
153.30
155.80
136.90
138.40
138.90
141.80
142.60
143.10
we can create a data frame with

> Oxboys.frm <- read.table( "oxboys.dat", header = TRUE )
> class( Oxboys.frm )
# check the class of the result
[1] "data.frame"
> dim( Oxboys.frm )
# check the dimensions
[1] 234 3
The argument header = TRUE in the call to read.table indicates that the
rst line of the le is to be used to create the names for the variables in
the frame.
The result of read.table is of the data.frame class. It has two dimensions:
the number of rows (cases) and the number of columns (variables).
A function to create objects of a given class is called a constructor for that
class. The primary constructor function for a class is often given the same
name as the class itself. Thus the default constructor for the groupedData
class is the groupedData function. Its required arguments are a formula and
a data frame. Optional arguments include labels, where display labels for
the response and the primary covariate can be given, and units, where the
units of these variables can be given. The default axis labels for data plots
are constructed by pasting together components of labels and units. The
reason for separating the units from the rest of the display label is to permit
propagation of the units to derived quantities such as the residuals from a
tted model.
103
For example, reading the Oxboys data from a le, converting it to a

groupedData object, and establishing default labels could be accomplished
in a single call of
> Oxboys <- groupedData( height ~ age | Subject,
+
data = read.table("oxboys.dat", header = TRUE),
+
labels = list(x = "Centered age", y = "Height"),
+
units = list(y = "(cm)") )
> Oxboys
# display the object
Grouped Data: height ~ age | Subject
Subject
age height
1
1 -1.0000 140.50
2
1 -0.7479 143.40
3
1 -0.4630 144.80
...
234
26 1.0055 143.10
By default the groupedData constructor also converts the grouping factor

(the factor Subject in the Oxboys data) to an ordered factor. The order is
determined by applying a summary function to the response within each
group. The default summary function is max, the maximum.
When the grouping factor has been converted to an ordered factor, the
panels in the trellis plots are arranged in that order. The order of the panels
is from left to right across the rows, starting with the bottom row. In the
default ordering described above the maximum value of the response in
each panel will increase across the rows starting from the lower left panel.
Most of the data plots in this book are ordered in this way.
Conversion of the grouping factor to an ordered factor is done only if
the expression for the grouping factor is simply the name of a variable in
the data frame. The conversion does not cause the rows of the data frame
themselves to be reordered; it merely changes the class of the grouping
factor and attaches the ordering to it. One way to examine the ordering is
by requesting the unique values of the grouping factor
> unique( getGroups( Oxboys ) )
[1] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
[21] 21 22 23 24 25 26
10 < 26 < 25 < 9 < 2 < 6 < 7 < 17 < 16 < 15 < 8 < 20 < 1 < 18 ...
The rst value of Subject in the data is 1, but the value of Subject with
the smallest maximum height is 10 so this subjects data occupy the lower
left panel in Figure 3.1.
The labels and units arguments are optional. We recommend using them
because this makes it easier to create more informative trellis plots.
104

0 30
0 30
0 30
0 30
0 30
11
0 30
10
12
0 30
13
0 30
15
14
16
Body weight (g)
600
500
400
300
0 30
0 30
0 30
0 30
0 30
0 30
0 30
0 30
Time (days)
FIGURE 3.2. Weight versus time of rats on three dierent diets. The rst group of
eight rats is on the control diet. There were four rats each in the two experimental
diets.
3.2.1 Roles of Other Experimental or Blocking Factors

Although the display formula can be used to designate a response, a covariate, and a grouping factor, these may not be sucient to describe the
structure of the experiment or study completely. Many experiments impose
additional structure on the data. For example, the observations in the orthodontic data, described in 1.4.1, are grouped according to the subject
on which the measurements were made. In most analyses of this type of
data we would want to include the subjects sex as an explanatory factor.
Because Sex is a characteristic of the subject, it is invariant within the observations for a single subject. A factor that is invariant within the groups
determined by the grouping factor is said to be outer to the grouping factor.
When outer factors are present they can, and should, be designated as
such when constructing a groupedData object using; for example,
outer = ~ Sex
Multiple outer factors can be specied by separating their names with *

in the formula.
Outer factors can be characteristics of the subject (more generally, of
the experimental unit) such as Sex in this example. They can also be
experimental factors that are applied to this level of experimental unit.
For example, the BodyWeight data, shown in Figure 3.2 and described in
Appendix A.3, contain measurements of the weights of 16 rats over time.
Eight of the rats were given a control diet, four rats were given one exper-

0
1
20
40
105
60
Body weight (g)
600
500
400
300
20
40
60
20
40
60
Time (days)
FIGURE 3.3. Weight of rats versus time for three dierent diets.
imental diet, and four rats were given another experimental diet. The Diet
factor is an experimental factor that is outer to the grouping factor Rat.
One benet of specifying outer factors to the constructor is that the constructor will modify the way in which the groups are ordered. Reordering
of the groups is permitted only within the same level of an outer factor (or
within the same combination of levels of outer factors, when there is more
than one). This ensures that groups at the same levels of all outer factors
will be plotted in a set of adjacent panels.
The plot method for the groupedData class allows an optional argument
outer that can be either a logical value or a formula. When this argument is
used the panels are determined by the factor or combination of factors given
in the outer formula. A logical value of TRUE or T can be used instead of a
formula, indicating that the outer formula stored with the data should be
used to determine the panels in the plot. For example, we can get a stronger
visual comparison of the dierences between the diets for the BodyWeight
data with
> plot( BodyWeight, outer = ~ Diet, aspect = 3 )
# Figure 3.3
106
The aspect argument, discussed in 3.3.1, is used to enhance the visualization of patterns in the plot. Because this outer formula was stored with
the BodyWeight data we would produce the same plot with
> plot( BodyWeight, outer = TRUE, aspect = 3 )
In plots grouped by the levels of an outer formula, the original grouping

factor stored with the data determines which points are associated with
each other in the panels. Points in the same group are joined by lines when
each panel is a scatter plot of the response versus a continuous covariate. If
there is no primary covariate the data will be plotted as a dot plot where
points in the same group will be rendered with the same symbol and color.
When there is more than one outer factor in the data the arrangement of
the panels depends on the order in which the factors are listed in the outer
formula for the plot. For example, in Chapter 7 we show several plots
of the results of an experiment in which the weights of soybean plants,
grown in dierent experimental plots, were measured several times during
the growing season. There were two dierent varieties of soybeans in the
experiment, which was carried out over three consecutive growing seasons.
The grouping factor is Plot and the outer factors are Variety and Year.
The plot produced by
> plot( Soybean, outer = ~ Year * Variety )
# Fig 6.10 (p. 288)
arranges panels of the same Variety on the same row, making it easy to
compare the results for each variety across years. Conversely, the plot produced by
> plot( Soybean, outer = ~ Variety * Year )
(not shown) arranges panels of the same Year on the same row, making it
easy to compare varieties within each year.
When specifying outer factors in the constructor or in a plot call we
should ensure that they are indeed constant or invariant within each level
of the grouping factor. The gsummary function with the optional argument
invariantsOnly = TRUE allows us to check this. It returns the values of only
those variables that are invariant within each level of the grouping factor.
These values are returned as a data frame with one row for each level of
the grouping factor. For the BodyWeight data we get
> gsummary( BodyWeight, invar = TRUE )
Rat Diet
2
2
1
3
3
1
4
4
1
1
1
1
8
8
1
5
5
1
Change in blood pressure (mm Hg)
8
4
16
32
64
107
128
1
30
20
10
30
20
10
0
8
16
32
64
128
16
32
64
128
Dose of phenylbiguanide (ug)

FIGURE 3.4. Change in blood pressure versus dose of phenylbiguanide (PBG)
for ve rabbits. Each rabbit was exposed to increasing doses of PBG, once after treatment with the H35 -agonist MDL 72222 and once after treatment with a
placebo.
6
7
11
9
10
12
13
15
14
16
6
7
11
9
10
12
13
15
14
16
1
1
2
2
2
2
3
3
3
3
indicating that Diet is an invariant of the grouping structure determined

by the Rat factor. Notice that the grouping factor itself, Rat in this case,
must be one of the invariants.
An outer factor is a characteristic of the experimental unit or an indicator
of a treatment applied to the entire unit. In some experiments one level of a
treatment may be applied to the experimental unit, say Subject, for some
of the observations, then another level applied for other observations. If
such an inner factor is distinct from the primary covariate, we may want
to indicate it somehow on a data plot. An example of an inner factor is
provided by the phenylbiguanide (PBG) data, shown in Figure 3.4 and
described in Appendix A.21. These data are from a cross-over trial where
each experimental animal was exposed to increasing doses of PBG under
108
two treatments; once with the MD5 -antagonist MDL 72222 and once with a
placebo. The change in blood pressure was measured for each dose on each
occasion.
In Figure 3.4, produced by
> plot( PBG, inner = ~ Treatment, scales = list(x = list(log = 2)))
the lines in each panel joint points with the same Treatment. (The scales
argument to the plot call will be described in 3.3.) This plot provides a
strong indication that the eect of the PBG treatment is to shift the dose
response curve to the right. We will discuss methods for modeling this in
Chapter 7.
The PBG data is similar in structure to the Pixel data. In both these data
sets there is a continuous response, a continuous covariate (day for Pixel and
dose for PBG), a major grouping factor corresponding to the experimental
animal (Dog for Pixel and Rabbit for PBG) and a factor that varies within
this major grouping factor (Side for Pixel and Treatment for PBG). In the
case of the Pixel data we used nested grouping factors to represent this
structure. In the case of the PBG data we used a single grouping factor with
an inner treatment factor. These two structures are quite similarin fact,
for the purposes of plotting the data, they are essentially equivalent. The
choice of one structure or the other is more an indication of how we think
the inner factor should be modeled. For the Pixel data we modeled the
eect of the Side factor as a random eect because the Side just represents
a random selection of lymph node for each Dog. In other experiments there
may be important physiological dierences between the left side and the
right side of the animal so we would model it as a xed eect. In the PBG
data the Treatment factor is a factor with xed, repeatable levels, and we
model it as a xed eect.
If we decide that an inner factor should be modeled as a random eect
we should specify it as part of a nested grouping structure. If it should be a
xed eect we specify it as an inner factor. These choices can be overridden
when constructing models.
3.2.2 Constructors for Balanced Data

The ergoStool data is an example of a data set that is balanced both with
respect to the grouping factor and with respect to the primary covariate.
That is, each Subject has the same number of observations on each Subject
and each subject uses each stool type the same number of times.
It can be convenient to represent a balanced, unreplicated set of responses
as a matrix. For example,
> ergoStool.mat <- asTable( ergoStool )
109
> ergoStool.mat
T1 T2 T3 T4
8 7 11 8 7
5 8 11 8 7
4 7 11 10 9
9 9 13 10 8
6 9 11 11 10
3 7 14 13 9
7 8 12 12 11
1 12 15 12 10
2 10 14 13 12
The asTable function, which can only be used with balanced and unreplicated data, produces a table of the responses in the form of a matrix where
columns correspond to the unique values of the primary covariate and rows
correspond to groups. The dimnames of the matrix are the unique levels of
the grouping factor and the covariate.
This table provides a compact representation of balanced data. Often
the data from a balanced experiment are provided in the form of a table
like this. The balancedGrouped function convert data from a table like this
to a groupedData object.
> ergoStool.new <- balancedGrouped( effort ~ Type | Subject,
+
data = ergoStool.mat )
Warning messages:
4 missing values generated coercing from character to numeric
in: as.numeric(dn[[1]])
> ergoStool.new
Grouped Data: effort ~ Type | Subject
Type Subject effort
1
T1
8
7
2
T2
8
11
3
T3
8
8
4
T4
8
7
5
T1
5
8
...
36
T4
2
12
The formula given as the rst argument to balancedGrouped is used to

assign names to the response, the primary covariate, and the grouping
factor. The data values are extracted from the matrix itself and from its
dimnames attribute. The matrix should be arranged so each row contains the
data from one group. The covariate values, corresponding to the dierent
columns, can be the levels of a factor or of a continous covariate. If the
column names can all be converted to numeric values, the covariate is
assumed to be continuous and is coerced to a numeric vector. Otherwise
it is left as a factor. The process of checking for numeric values in these
names is what generates the warning message in the previous example.
110
It is a common practice to label the levels of a factor like the Type factor
as 1, 2, . . . , which would result in its being coerced to a numeric variable.
Unless this is detected and the numeric variable is explicitly converted to
a factor, models t to such data will be nonsensical. It is always a good
idea to check that the variables in a groupedData object have the expected
classes. We describe how to do this in 3.4.
The optional labels and units arguments can be used with balancedGrouped just as in the groupedData constructor.
As seen in the example, the balancedGrouped constructor produces an
object like any other groupedData object. The matrix of response values
is converted to a vector and both the primary covariate and the grouping
factor are expanded to have the same length as the response vector. Later
manipulations of the object or plots created from the object do not rely on
its having been generated from balanced data. This is intentional. Although
there is a certain amount of redundancy in storing multiple copies of the
same covariate or grouping factor values, it is oset by the exibility of
the data.frame structure in dealing with missing data and with general,
unbalanced data.
Many methods for the analysis of longitudinal data or repeated measurements data depend on having balanced data. The analysis techniques
described in this book do not.
3.3 Controlling Trellis Graphics Presentations of

Grouped Data
Trellis graphics presentations of grouped data allow easy evaluation of the
behavior of the response with respect to the primary covariate within each
group. They also allow comparisons between groups. Because they are so effective at illustrating both within-group and between-group behavior, they
are the default plot method for groupedData objects.
The defaults chosen in the trellis graphics library and in the plot method
for groupedData objects will usually provide an informative and visually
appealing plot. Sometimes, however, the default plot can be made even
more informative by adjusting one or two of the trellis graphics parameters.
In this section we describe some of the trellis parameters that are helpful in
enhancing plots of grouped data. For a full discussion of the trellis graphics
parameters and controls see Becker, Cleveland and Shyu (1996) or the online documentation for the trellis library.
3.3.1 Layout of the Trellis Plot

A trellis plot consists of one or more panels arranged in a regular array
on one or more pages. In the default data plot for a groupedData object
111
with a continuous primary covariate, there is one panel for each level of the
grouping factor. The horizonal axis in the panel is the primary covariate,
the vertical axis is the response, and the data are represented both as points
and by a connecting line. If the primary covariate is a factor, such as in the
Machine data, or if there is no primary covariate, such as in the Rail data,
the plot is a dotplot with one row for each level of the grouping factor. In
this case the response is on the horizontal axis.
For numeric covariates the aspect ratio of each panel, which is the ratio
of the physical size of the vertical axis to that of the horizontal axis, is
determined by the 45-degree banking rule described in Cleveland (1994,
3.1). We have found that this rule produces appealing and informative
aspect ratios in a wide variety of cases. If you wish to override this choice
of aspect ratio, you can give an numerical value as the optional aspect
argument in the call to plot. A value greater than 1.0 produces tall, narrow
panels, while a value between 0.0 and 1.0 produces short, wide panels.
The arrangement of panels in rows and columns on the page is calculated
to make the specied number of panels of the chosen aspect ratio ll as
much as possible of the available area. This does not always create a good
arrangement for comparing patterns across outer factors. For example, the
grouping factor Plant in the CO2 data, described in Appendix A.5, has
twelve dierent levels. The plants themselves come from one of two types
and have been subjected to one of two treatments. Because the aspect ratio
chosen by the banking rule creates panels that are taller than they are wide,
a more-or-less square plot area will be lled with three rows of four panels
each (Figure 3.5).
For some combinations of grass type and treatment the panels are spread
across more than one row in Figure 3.5. It would be better to keep these
combinations on the same row so we can more easily compare treatments
and grass types. That is, we would prefer the panels to be arranged in four
rows of three or, perhaps, two rows of six. If we have four rows of three,
we may wish to indicate visually that the lower two rows represent one
type of plant (Quebec) and the upper two rows represent the other type
(Mississippi). We can do this by specifying a list as the optional between
argument. A component named x in this list indicates the sizes of gaps to
leave between columns while a component named y species gaps between
rows. (When forming a between argument for the rows, remember that the
trellis convention is to count from the bottom to the top, not from the top
to the bottom.) The gaps are given in units of character heights. Generally
a gap of 0.5 is sucient to distinguish groups without using too much space.
An arrangement of the CO2 data in two rows of six panels with a gap
between the third and fourth columns, shown in Figure 3.6, is produced by
> plot(CO2, layout=c(6,2), between=list(x=c(0,0,0.5,0,0))) # Fig 3.6
Assembling the panels on a single page is eective when there is a small

or moderate number of groups. If there is a large number of groups, the
112

200
600
1000
200
600
Mn1
Mc2
Mc3
Mc1
Qc3
Qc2
Mn3
Mn2
1000
40
CO2 uptake rate (umol/m^2 s)
30
20
10
40
30
20
10
Qn1
Qn2
Qn3
Qc1
40
30
20
10
200
600
1000
200
600
1000
Ambient carbon dioxide concentration (uL/L)

FIGURE 3.5. Carbon dioxide uptake versus ambient CO2 concentration for
Echinochloa crus-galli plants, six from Quebec and six from Mississippi. Half the
plants of each type were chilled overnight before the measurements were taken.
The labels of each panel show the origin of the plant (Quebec or Mississippi)
and the treatment (chilled or nonchilled). This plot shows a default layout of the
panels.
single page plot may result in the panels being too small to be informative.
In these cases a third component can be added to the layout argument
causing the plot to be spread over several pages.
If the number of groups in some level of an outer factor does not t
exactly into the rectangular array, an optional skip argument can be used
to skip over selected panel locations.
The use of both of these arguments is illustrated in the code for the gures
of the spruce tree growth data (Appendix A.28). There were four groves
of trees; two exposed to an ozone-rich atmosphere, and two exposed to a
normal atmosphere. In the rst and second groves 27 trees were measured,
but in the third and fourth groves only 12 and 13 trees were measured,
respectively.

200
Mn3
600
1000
Mn2
200
Mn1
600
1000
Mc2
200
Mc3
600
113
1000
Mc1
40
30
20
10
Qn1
Qn2
Qn3
Qc1
Qc3
Qc2
40
30
20
10
200
600
1000
200
600
1000
200
600
1000

FIGURE 3.6. Carbon dioxide uptake versus ambient CO2 concentration for
Echinochloa crus-galli plants. This plot shows an alternative layout of the panels.
The three pages of gures (Figures A.8A.10, pages 445447) in an array

of four rows of seven columns per page were created with
> plot( Spruce, layout = c(7, 4, 3),
+
skip = c(rep(FALSE, 27), TRUE, rep(FALSE, 27), TRUE,
+
rep(FALSE, 12), rep(TRUE, 2), rep(FALSE,13)) )
An alternative arrangement (not shown) of three pages in an array of three

rows of nine columns per page can be created with
> plot( Spruce, layout = c(9, 3, 3),
+
skip = c(rep(FALSE, 66), TRUE, TRUE, rep(FALSE, 13)) )
On the rst two pages of this plot the array would be lled. On the third
page there would be a gap in the middle of the array to separate the panels
corresponding to the two dierent groves of trees exposed to a normal
atmosphere.
3.3.2 Modifying the Vertical and Horizontal Scales

It is often helpful to present quantities such as concentrations on a logarithmic scale. The optional scales argument for trellis graphics functions
allows specication of logarithmic scales on either the vertical or horizontal
axes. The axis annotations can be at powers of 10 or at powers of 2.
For example, the default plot of the DNase assay data shown in Figure 3.7
squeezes most of the data onto the left-hand side of the panel. (These data
are described in more detail in Appendix A.7.)
114

0
8 10
8 10
3
2.0
Optical density
1.5
1.0
0.5
0.0
10
11
2.0
1.5
1.0
0.5
0.0
0
8 10
8 10
8 10
DNase concentration (ng/ml)

FIGURE 3.7. Optical density versus DNase concentration, arithmetic scale
Both the facts that the unique values of the DNase concentration are,
for the most part, logarithmically spaced
> unique( getCovariate(DNase) )
[1] 0.048828 0.195312 0.390625 0.781250 1.562500 3.125000
[7] 6.250000 12.500000
> log( unique(getCovariate(DNase)), 2 )
[1] -4.35614 -2.35614 -1.35614 -0.35614 0.64386 1.64386 2.64386
[8] 3.64386
and the experimenters desire to t a logistic response function (described

in Chapter 6 and Appendix C.7) to the logarithm of the concentration
indicate that we should use a logarithmic scale on the horizontal axis. This
change is incorporated in Figure 3.8, produced with
> plot( DNase, layout=c(6,2), scales = list(x=list(log=2)) )
3.3.3 Modifying the Panel Function

This is an advanced topic. You can consider skipping this section unless
you want to modify the way the data are presented within each panel.
The presentation of the data within each panel is controlled by a panel
function. If no primary covariate is available, or if the primary covariate
is a categorical variable, the default panel function for the plot method
for groupedData objects is panel.dotplot. When the primary covariate is
numeric, the default panel function is
function(x, y) {
panel.grid()
panel.xyplot(x, y)

2^-4 2^-2
2^-4 2^-2
115
3
2.0
Optical density
1.5
1.0
0.5
0.0
10
11
2.0
1.5
1.0
0.5
0.0
2^-4 2^-2
2^-4 2^-2
2^-4 2^-2
DNase concentration (ng/ml)

FIGURE 3.8. Optical density versus DNase concentration for eleven runs of an
assay. The concentration is shown on a logarithmic scale.
y.avg <- tapply(y, x, mean)

# average y for each distinct x
xvals <- as.numeric(names(y.avg))
ord <- order(xvals)
panel.xyplot(xvals[ord], y.avg[ord], type = "l")
}
The rst two lines of this function draw the background grid and place
symbols at the data values. The actual symbol that is drawn is determined
by the trellis device that is active when the plot is displayed. It is usually
an open circle.
The last four lines of the panel function add a line through the data
values. Some care needs to be taken when doing this. In the DNase assay
data, for example, there are duplicate observations at each concentration in
each run. Rather than joining the dots, it makes more sense to draw the
line through the average response in each set of replicates. In the default
panel function xvals is dened to be the unique values of x and y.avg is
calculated as the average of the y values at these distinct x values. Finally,
the xvals vector is put into increasing order and the line drawn with the
points in this order.
This panel function can be overridden with an explicit panel argument to
the plot call if, for example, you want to omit the background grid. If you
do override the default panel function, it would be a good idea to follow
the general pattern of this function. In particular, you should draw the grid
rst (if you choose to do so) then add any points or lines. Also, be careful
to handle replicates or unusual ordering of the (x, y) pairs gracefully.
116
3.3.4 Plots of Multiply-Nested Data

The plot method for multiply-nested groupedData objects takes an optional
argument displayLevel that denes the display level to be used. By default,
the innermost level of grouping is used as the display level. For the Pixel
data, this is Side within Dog, as shown in Figure 3.9.
> plot( Pixel, layout = c(4,5),
+
between = list(x = c(0, 0.5), y = 0.5) )
# Figure 3.9
A more meaningful trellis display of these data, using Dog as the display
level, is obtained with
> plot( Pixel, displayLevel = 1 )
# Figure 3.10
When the display level for a multiply-nested groupedData object is smaller

than the maximum grouping level, dierent actions can be taken with respect to the inner grouping levels. The default action is to preserve, as
much as possible, the original structure of the data and use the combination of the inner grouping factors as a single inner factor, as in Figure 3.10.
Alternatively, observations within a given value of the display level may
be collapsed over some, or all inner levels by giving a collapse argument
in the plot call. Another optional argument, FUN, can specify the summary
function to be use when collapsing the data. This summary function should
take a numeric vector as its argument and return a single numeric value.
The default is the mean function.
Using collapse can help to reduce clutter in a plot. For example, the
Wafer data, from an experiment in semiconductor manufacturing, give current intensity versus the applied voltage at eight sites on each of ten wafers
(see Appendix A.30 for details). Dierences between wafers and between
sites within wafers are too subtle to be noticeable on a plot at displayLevel
= 2 that has separate panels for each Site within each Wafer. Displaying
at the Wafer level with separate curves for each Site within each Wafer, as
in Figure 3.11, produces a cluttered plot because the eight curves for each
wafer nearly overlay each other. It is more informative to examine the mean
curve for each wafer and the standard deviation about this mean curve, as
in Figures 3.12 and 3.13
> plot( Wafer, display = 1, collapse = 1 )
# Fig. 3.12
> plot( Wafer, display = 1, collapse = 1,
# Fig. 3.13
+
FUN = function(x) sqrt(var(x)), layout = c(10,1) )
In general there is a trend for the standard deviation about the curve to
be greater when the response is greater. This is a common occurrence. A
less obvious eect is that the standard deviation is greater in the wafers
with overall higher current intensity, even though the dierences in the
current intensity are small.
5 10
20
5 10
7/l
7/r
10/r
10/l
4/l
4/r
5/r
5/l
117
20
1160
1140
1120
1100
1080
1060
1040
1160
1140
1120
1100
1080
1060
Pixel intensity
1040
8/r
8/l
6/r
6/l
3/l
3/r
9/r
9/l
1160
1140
1120
1100
1080
1060
1040
1160
1140
1120
1100
1080
1060
1040
1/r
1/l
2/r
2/l
1160
1140
1120
1100
1080
1060
1040
0
5 10
20
5 10
20

FIGURE 3.9. Mean pixel intensity of lymph nodes in the axillary region versus
time by Side within Dog.
118
Pixel intensity
10
15
20
10
15
20
10
1160
1140
1120
1100
1080
1060
1040
1160
1140
1120
1100
1080
1060
1040
0
10
15
20
10
15
20
10
15
20

FIGURE 3.10. Mean pixel intensity of lymph nodes in the axillary region versus
time by Dog. The two curves on each plot correspond to the left and the right
sides of the Dog.
1.0
Intensity of current (mA)
1.5
2.0
1.0
10
1.5
2.0
7
15
10
15
10
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
Voltage (V)
FIGURE 3.11. Current versus voltage for the Wafer data. The panels correspond
to wafers. With each wafer the current was measured at eight dierent sites.
1.0
1.5
2.0
1.0
1.5
10
119
2.0
7
15
10
15
10
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
Voltage (V)
Current standard deviation
FIGURE 3.12. Mean current versus voltage at the wafer level for the Wafer data.
Each point on each curve is the average of the current for that voltage at eight
sites on the wafer.
1.0
2.0
1.0
2.0
1.0
2.0
1.0
10
2.0
1.0
2.0
0.4
0.3
0.2
0.1
1.0
2.0
1.0
2.0
1.0
2.0
1.0
2.0
1.0
2.0
Voltage (V)
FIGURE 3.13. Standard deviation of current versus voltage at the wafer level for
the Wafer data. Each point on each curve is the standard deviation of the current
at that voltage at eight sites on the wafer.
120
3.4 Summaries
In addition to creating graphical presentations of the data we may wish to
summarize the data numerically, either by group or across groups. In 3.1
we demonstated the use of the table, gsummary, and unique functions to
summarize data by group. In this section we expand on the usage of these
functions and introduce another groupwise summary function gapply.
The gapply function is part of the nlme library. It joins several standard
S functions in the apply family. These include apply, lapply, tapply, and
sapply. They all apply a function to subsections of some data structure
and gather the results in some way. Both lapply and sapply can be used
to apply a function to the components of a list. The result of lapply is
always a list; sapply will create a more compact result if possible. Because
the groupedData class inherits from the data.frame class and a data frame
can be treated as a list whose components are the columns of the frame, we
can apply a function to the columns of a groupedData object. For example,
we can use sapply to check the data.class of the columns of a groupedData
object by
> sapply( ergoStool, data.class )
effort
Type
Subject
"numeric" "factor" "ordered"
We see that effort, the response, is a numeric variable; Type, the covariate,
is a factor, and Subject, the grouping factor, is an ordered factor. We could
replace sapply with lapply and get the same information, but the result
would be returned as a list and would not print as compactly.
Checking the data.class of all variables in a data.frame or a groupedData
object is an important rst step in any data analysis. Because factor levels
are often coded as integers, it is a common mistake to leave what should
be a factor as a numeric variable. Any linear models using such a factor
will be meaningless because the factor will be treated as a single numeric
variable instead of being expanded into a set of contrasts. Another way of
checking on the data.class of variables in a frame is to use the summary
function shown later in this section.
The table and unique functions are not solely intended for use with
grouped data, but often are useful when working with grouped data. The
gsummary function, however, is specically designed for use with groupedData objects. In 3.2.1 we used gsummary with the optional argument
invariantsOnly=TRUE to extract only those variables in the groupedData
object that are invariant within levels of the grouping factor. These could
be experimental factors, like Diet in the BodyWeight data, or they could
simply be additional characteristics of the groups, like Sex in the Orthodont
data. The Theoph data are another example of a medical study where the
grouping is by Subject. They are from a study of the pharmacokinetics of
the drug theophylline, which is used to treat asthma. Each subjects weight
3.4 Summaries
121
and dose of theophylline are given in the data. As this is a short-duration

study (about 24 hours) and only one dose of theophylline was given, both
Wt and Dose are invariants.
> gsummary( Theoph, inv = TRUE )
Subject
Wt Dose
6
6 80.0 4.00
7
7 64.6 4.95
8
8 70.5 4.53
11
11 65.0 4.92
3
3 70.5 4.53
2
2 72.4 4.40
4
4 72.7 4.40
9
9 86.4 3.10
12
12 60.5 5.30
10
10 58.2 5.50
1
1 79.6 4.02
5
5 54.6 5.86
Sometimes it is distracting to have the grouping factor itself included

as one of the invariants. The optional argument omitGroupingFactor=TRUE
suppresses this. The combination of omit = TRUE and inv = TRUE can be
used to check if there are any nontrivial invariants. The value returned will
be NULL unless there are invariants other than the grouping factor.
> gsummary( Theoph, omit = TRUE, inv = TRUE )
Wt Dose
6 80.0 4.00
7 64.6 4.95
8 70.5 4.53
11 65.0 4.92
3 70.5 4.53
2 72.4 4.40
4 72.7 4.40
9 86.4 3.10
12 60.5 5.30
10 58.2 5.50
1 79.6 4.02
5 54.6 5.86
> is.null(gsummary(Theoph, inv = T, omit = T)) # invariants present
[1] F
> is.null(gsummary(Oxboys, inv = T, omit = T)) # no invariants
[1] T
When the invariantsOnly argument is omitted or given the value FALSE, a

summary of all the variables in the object is returned. The default summary
is a representative value for each variable within each group: numeric
variables are represented by their mean within each group and non-numeric
variables (e.g. factors) by their modes (the most frequently occuring value)
122
within each group. When multiple modes are present, the rst element of
the sorted modes is returned.
> gsummary( Theoph )
Subject
Wt Dose
6
6 80.0 4.00
7
7 64.6 4.95
8
8 70.5 4.53
11
11 65.0 4.92
3
3 70.5 4.53
2
2 72.4 4.40
4
4 72.7 4.40
9
9 86.4 3.10
12
12 60.5 5.30
10
10 58.2 5.50
1
1 79.6 4.02
5
5 54.6 5.86
time
5.888182
5.865455
5.890000
5.871818
5.907273
5.869091
5.940000
5.868182
5.876364
5.915455
5.950000
5.893636
conc
3.525455
3.910909
4.271818
4.510909
5.086364
4.823636
4.940000
4.893636
5.410000
5.930909
6.439091
5.782727
By giving a numeric summary function, which is a function that calculates a single numerical value from a numeric vector, as the argument
FUN, we can produce other summaries. For example, we can check that
the Theoph data are sorted according to increasing values of the maximum
response with
> gsummary( Theoph, FUN = max, omit = TRUE )
Wt Dose time conc
6 80.0 4.00 23.85 6.44
7 64.6 4.95 24.22 7.09
8 70.5 4.53 24.12 7.56
11 65.0 4.92 24.08 8.00
3 70.5 4.53 24.17 8.20
2 72.4 4.40 24.30 8.33
4 72.7 4.40 24.65 8.60
9 86.4 3.10 24.43 9.03
12 60.5 5.30 24.15 9.75
10 58.2 5.50 23.70 10.21
1 79.6 4.02 24.37 10.50
5 54.6 5.86 24.35 11.40
This ordering of the subjects does indeed give increasing maximum concentration of theophylline.
The FUN argument to gsummary is applied only to numeric variables in the
grouped data object. Any non-numeric variables are represented by their
modes within each group. A variable that is invariant within each group
is represented by the (single) value that it assumes within each group. In
other words, the value returned for each variable is determined according
to:
If the variable is an invariant, its value within each group is returned.
3.4 Summaries
123
If the variable is a factor (ordered or unordered) or of mode character,

the mode for each group is returned.
If the variable is numeric and not invariant, the summary function
FUN is applied within each group and those values are returned.
When there are a large number of groups in the data, the result of
gsummary may itself be so large as to be unwieldy. The Quinidine data,
described in Appendix A.25, is from a study where thirteen dierent variables were recorded on 136 subjects.
> Quin.sum <- gsummary( Quinidine, omit = TRUE, FUN = mean )
> dim( Quin.sum )
[1] 136 13
Upon examining the rst few rows

> Quin.sum[1:10, ]
time conc dose interval
109 30.2633
NA
NA
NA
70
0.7500
NA
NA
NA
23 52.0263
NA
NA
NA
92
8.8571
NA
NA
NA
111 18.1638
NA
NA
NA
5 24.3750
NA
NA
NA
18 196.8438
NA
NA
NA
24 31.2500
NA
NA
NA
2 12.2000
NA
NA
NA
88
4.7900
NA
NA
NA
Ethanol
Heart Creatinine
109
none No/Mild
>= 50
70 former No/Mild
>= 50
23
none No/Mild
>= 50
92 former No/Mild
>= 50
111 former No/Mild
>= 50
5
none
Severe
>= 50
18
none No/Mild
< 50
24 former No/Mild
>= 50
2 current Moderate
>= 50
88
none Moderate
>= 50
Age Height Weight

Race Smoke
70
67 58.000 Caucasian
no
68
69 75.000 Caucasian
no
75
72 108.000 Caucasian
yes
68
72 65.000 Caucasian
yes
68
66 56.000
Latin
yes
62
71 66.000 Caucasian
yes
87
69 85.375 Caucasian
no
55
69 89.000
Latin
no
58
69 85.000
Latin
no
85
72 77.000 Caucasian
no
glyco
0.46000
1.15000
0.83875
1.27000
1.23000
1.39000
1.26000
0.57000
0.82000
0.61000
we see some unusual results. The summarized values of conc, dose, and
interval are always recorded as missing values (NA). A less obvious peculiarity in the data is an apparent inconsistency in the numeric values of
height and weight; for some reason height was recorded in inches while
weight was recorded in kilograms.
Returning to the question of the missing values in conc, dose, and interval, the data from a single subject indicate the reason for this
124
> Quinidine[Quinidine[["Subject"]] == 3, 1:8]

Subject
time conc dose interval Age Height Weight
17
3
0.00
NA 201
NA 67
69
69
18
3
8.00
NA 201
NA 67
69
69
19
3
16.00
NA 201
NA 67
69
69
20
3
24.00
NA 201
NA 67
69
69
21
3
32.00
NA 201
NA 67
69
69
22
3
41.25 2.4
NA
NA 67
69
69
23
3 104.00
NA 201
8 67
69
69
24
3 113.00 2.3
NA
NA 67
69
69
25
3 3865.00
NA 201
6 67
69
62
26
3 3873.00
NA 201
NA 67
69
62
27
3 3881.00
NA 201
NA 67
69
62
28
3 3889.00
NA 201
NA 67
69
62
29
3 3897.00
NA 201
NA 67
69
62
30
3 3900.00
NA
NA
NA 67
69
62
31
3 3905.00
NA 201
NA 67
69
62
32
3 3909.00 4.7
NA
NA 67
69
62
33
3 4073.00
NA 201
8 67
69
62
Each observation is either a record of a dosage or a record of a concentration measurement but never both. Thus, whenever dose is present, conc is
missing and whenever conc is present, both dose and interval are missing.
Because any subject in the experiment must have at least one dosage record
and at least one concentration record, every subject has missing data in
the conc, dose, and interval variables.
The default behavior of most summary functions in S is to return the
value NA if the input vector contains any missing values. The mean function
behaves like this and returns NA for every subject in each of these three
variables. The behavior can be overridden in mean (and in several other
summary functions) by giving the optional argument na.rm = TRUE. The
default value of FUN in gsummary is mean with na.rm = TRUE.
> Quin.sum1 <- gsummary( Quinidine, omit = TRUE )
> Quin.sum1[1:10, 1:7]
time
conc
dose interval
Age Height Weight
1
92.817 2.2000 268.71
6.0000 60.000
69 106.000
2
12.200 1.2000 166.00
NA 58.000
69 85.000
3 2090.015 3.1333 201.00
7.3333 67.000
69 65.294
4 137.790 3.3667 236.39
7.3333 88.000
66 95.185
5
24.375 0.7000 301.00
NA 62.000
71 66.000
6
3.625 2.6000 166.00
6.0000 76.000
71 93.000
7 1187.320 2.7833 256.22
6.0000 60.097
66 85.484
8
20.019 2.5000 166.00
NA 52.000
71 75.000
9
69.200 3.9500 498.00
6.0000 68.000
70 79.000
10 1717.261 3.1667 201.00
8.0000 73.154
69 79.462
Notice that there are still some NAs in the groupwise summary for interval. For these subjects every value of interval was missing.
3.4 Summaries
125
The function summary can be used with any data frame to summarize the
columns according to their class. In particular, we can use it on Quin.sum1 to
obtain some summary statistics for each variable with each group (subject)
counted only once.
> summary( Quin.sum1 )
time
conc
Min.
:
0.065
Min.
:0.50
1st Qu.: 19.300
1st Qu.:1.70
Median : 47.200
Median :2.24
Mean
: 251.000
Mean
:2.36
3rd Qu.: 171.000
3rd Qu.:2.92
Max.
:5360.000
Max.
:5.77
dose
Min.
: 83
1st Qu.:198
Median :201
Mean
:224
3rd Qu.:249
Max.
:498
Age
Height
Weight
Min.
:42.0
Min.
:60.0
Min.
: 41.0
1st Qu.:61.0
1st Qu.:67.0
1st Qu.: 67.8
Median :66.0
Median :70.0
Median : 79.2
Mean
:66.9
Mean
:69.6
Mean
: 79.2
3rd Qu.:73.0
3rd Qu.:72.0
3rd Qu.: 88.2
Max.
:92.0
Max.
:79.0
Max.
:119.0
Smoke
Ethanol
Heart
Creatinine
no :94
none
:90
No/Mild :55
< 50 : 36
yes:42
current:16
Moderate:40
>= 50:100
former :30
Severe :41
interval
Min.
: 5.00
1st Qu.: 6.00
Median : 6.00
Mean
: 6.99
3rd Qu.: 8.00
Max.
:12.00
NAs
:29.00
Race
Caucasian:91
Latin
:35
Black
:10
glyco
Min.
:0.390
1st Qu.:0.885
Median :1.170
Mean
:1.210
3rd Qu.:1.450
Max.
:2.990
Contrast this with the result of

> summary( Quinidine )
...
time
conc
dose
interval
Min.
:
0
Min.
:
0.40
Min.
: 83
Min.
:
4.00
1st Qu.: 16
1st Qu.:
1.60
1st Qu.:166
1st Qu.:
6.00
Median : 60
Median :
2.30
Median :201
Median :
6.00
Mean
: 373
Mean
:
2.45
Mean
:225
Mean
:
7.11
3rd Qu.: 241
3rd Qu.:
3.00
3rd Qu.:249
3rd Qu.:
8.00
Max.
:8100
Max.
:
9.40
Max.
:603
Max.
: 12.00
NAs
:1110.00
NAs
:443
NAs
:1222.00
Age
Height
Weight
Race
Min.
:42.0
Min.
:60.0
Min.
: 41.0
Caucasian:968
1st Qu.:60.0
1st Qu.:67.0
1st Qu.: 69.5
Latin
:384
Median :66.0
Median :69.0
Median : 78.0
Black
:119
Mean
:66.7
Mean
:69.2
Mean
: 79.7
3rd Qu.:74.0
3rd Qu.:72.0
3rd Qu.: 89.0
Max.
:92.0
Max.
:79.0
Max.
:119.0
Smoke
Ethanol
Heart
Creatinine
no :1024
none
:991
No/Mild :598
< 50 : 418
126
yes: 447
current:191
former :289
Moderate:375
Severe :498
>= 50:1053
glyco
Min.
:0.39
1st Qu.:0.93
Median :1.23
Mean
:1.28
3rd Qu.:1.54
Max.
:3.16
The rst summary tells us that there are 94 nonsmokers in the study and
42 smokers while the second tells us that there are 1024 total observations
on the nonsmokers and 447 on the smokers.
Both summaries are useful to us in understanding these data. Because
NAs in conc and dose are mutually exclusive, we can see that there are
at most 1110 dosage records and at most 443 concentration measurements.
Because there are 136 subjects in the study, this means there are on average
fewer than three concentration measurements per subject.
We can get an exact count of the number of concentrations by counting
the number of nonmissing values in the entire conc variable. One way to
do this is
> sum( ifelse(is.na(Quinidine[["conc"]]), 0, 1) )
[1] 361
This is equivalent to the somewhat more terse expression

> sum( !is.na(Quinidine[["conc"]]) )
[1] 361
because the logical values TRUE and FALSE are interpreted as 1 and 0 in
arithmetic expressions.
A similar expression
> sum( !is.na(Quinidine[["dose"]]) )
[1] 1028
tells us that there are 1028 dosage events. The 164 rows that are neither
dosage records nor concentration measurements are cases where values of
other variables changed for the subject.
With only 361 concentration measurements for 136 subjects there are
fewer than three concentration measurements per subject. To explore this
further, we would like to determine the distribution of the number of concentration measurements per subject. We need to divide the data by subject
(or by group in our general terminology) and count the number of nonmissing values in the conc variable for each subject. The function gapply is
available to perform calculations such as this. It applies another function
to some or all of the variables in a grouped data object by group.
3.4 Summaries
127
> gapply( Quinidine, "conc", function(x) sum(!is.na(x)) )

109 70 23 92 111 5 18 24 2 88 91 117 120 13 89 27 53 122 129 132
1 1 3 1
1 2 3 1 1 1 1
3
2 1 3 1 1
1
2
3
16 106 15 22 57 77 115 121 123 11 48 126 223 19 38 42 52 56 63 83
1
1 1 1 3 1
4
1
1 2 2
2
6 1 1 2 1 1 4 1
104 118 137 17 29 34 46 73 87 103 138 45 44 97 36 37 72 100 8 71
2
2
1 1 1 1 3 2 2
1
2 3 7 2 2 3 1
3 1 5
6 14 26 75 20 96 99 134 12 49 67 85 112 127 55 68 124 1 35 47 79
1 3 1 3 2 3 2
1 1 3 3 1
3
3 6 3
1 2 2 5 3
95 114 135 105 116 62 65 107 130 66 139 33 80 125 110 128 136 21
3
2
2
1
3 4 7
4
3 1
3 3 2
1 11
2 11 2
43 90 102 40 84 98 30 82 93 108 119 32 133 7 9 76 94 58 113 50 39
1 1
2 2 6 2 1 3 4
1
3 1
2 6 2 6 5 1
2 3 2
78 25 61 3 64 60 59 10 69 4 81 54 41 74 28 51
10 2 2 3 4 4 3 6 2 6 11 4 3 3 4 6
The result can be a bit confusing when printed in this way. It is a named
vector of the counts where the names are the values of the Subject variable.
Thus, subjects 124 and 125 both had only a single concentration measurement. To obtain the distribution of the measurements, we apply the table
function to this vector
> table( gapply(Quinidine, "conc", function(x) sum(!is.na(x))) )
1 2 3 4 5 6 7 10 11
46 33 31 9 3 8 2 1 3
We see that most of the subjects in the study have very few measurements
of the response. A total of 110 out of the 136 subjects have fewer than four
response measurements. This is not uncommon in such routine clinical data
(data that are collected in the routine course of treatment of patients). A
common consequence of having so few observations for most of the subjects
is that the information gained from such a study is imprecise compared to
that gained from a controlled experiment.
The second argument to gapply is the name or index of the variable in
the groupedData object to which we will apply the function. When only a
single name or index is given, the value passed to the function to be applied
is in the form of a vector. If more than one variable is to be passed, they
are passed as a data.frame. If this argument is missing, all the variables in
the groupedData object are passed as a data.frame so the eect is to select
subsets of the rows corresponding to unique values of the grouping factor
from the groupedData object.
To illustrate this, let us determine those subjects for whom there are
records that are neither dosage records nor concentration records. For each
subject we must determine whether there are any records with both the
conc variable and the dose variable missing.
> changeRecords <- gapply( Quinidine, FUN = function(frm)
+
any(is.na(frm[["conc"]]) & is.na(frm[["dose"]])) )
128
> changeRecords
109 70 23 92 111 5 18 24 2 88 91 117 120 13 89 27 53 122 129 132
F F F F
F F T F F F F
F
F F F F F
F
F
T
...
78 25 61 3 64 60 59 10 69 4 81 54 41 74 28 51
F F T T T F F T F T T T T F T F
As we see, the printed representation as a named vector of length 136 is

not easy to read. It is more informative if we convert the result to a vector
of levels of the Subject factor for which there are records that are neither
dosage records nor concentration records.
> sort( as.numeric( names(changeRecords)[changeRecords] ) )
[1]
3
4
7 10 14 18 28 33 37 40 41 44 45 46 47
[16] 48 50 54 55 61 62 63 64 65 71 75 76 77 79 80
[31] 81 82 84 94 95 96 97 98 110 112 114 118 119 127 132
[46] 133 135 136 139 223
Notice that subject 3 is one of those with such a change record. We

printed some of the variables from this subjects data on page 123. We can
see there that the record in question is row 30. If we look at this row and
the adjacent rows
> Quinidine[29:31,]
Grouped Data: conc ~ time | Subject
Subject time conc dose interval Age Height Weight
Race
29
3 3897
NA 201
NA 67
69
62 Caucasian
30
3 3900
NA
NA
NA 67
69
62 Caucasian
31
3 3905
NA 201
NA 67
69
62 Caucasian
Smoke Ethanol
Heart Creatinine glyco
29
yes former Moderate
< 50 1.71
30
yes former Moderate
< 50 1.71
31
yes former Moderate
< 50 1.71
we can see that there are no changes in any of the variables except for time
so this row is redundant. We did not cull such redundant rows from the
data set because we wanted to be able directly to compare results based
on these data with analyses done by others.
The data for subject 4 provide a better example.
> Quinidine[Quinidine[["Subject"]] == 4, ]
Grouped Data: conc ~ time | Subject
Subject
time conc dose interval Age Height Weight Race Smoke
45
4
0.00
NA 332
NA 88
66
103 Black
yes
46
4
7.00
NA 332
NA 88
66
103 Black
yes
47
4 13.00
NA 332
NA 88
66
103 Black
yes
48
4 19.00
NA 332
NA 88
66
103 Black
yes
49
4 21.50 3.1
NA
NA 88
66
103 Black
yes
50
4 85.00
NA 249
6 88
66
103 Black
yes
51
4 91.00 5.8
NA
NA 88
66
103 Black
yes
3.4 Summaries
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
4
Ethanol
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
none
91.08
NA 249
NA
97.00
NA 249
NA
103.00
NA 249
NA
105.00
NA
NA
NA
109.00
NA 249
NA
115.00
NA 249
NA
145.00
NA 166
NA
151.00
NA 166
NA
156.00 3.1
NA
NA
157.00
NA 166
NA
163.00
NA 166
NA
169.00
NA 166
NA
174.75
NA 201
NA
177.00
NA
NA
NA
181.50 3.1
NA
NA
245.00
NA 201
8
249.00
NA
NA
NA
252.50 3.2
NA
NA
317.00
NA 201
8
326.00 1.9
NA
NA
Heart Creatinine glyco
Severe
>= 50 1.48
Severe
>= 50 1.48
Severe
>= 50 1.48
Severe
>= 50 1.48
Severe
>= 50 1.48
Severe
>= 50 1.61
Severe
>= 50 1.61
Severe
>= 50 1.61
Severe
>= 50 1.61
Severe
>= 50 1.61
Severe
>= 50 1.61
Severe
>= 50 1.61
Severe
>= 50 1.61
Severe
>= 50 1.88
Severe
>= 50 1.88
Severe
>= 50 1.88
Severe
>= 50 1.88
Severe
>= 50 1.88
Severe
>= 50 1.88
Severe
>= 50 1.88
Severe
>= 50 1.68
Severe
>= 50 1.68
Severe
>= 50 1.87
Severe
>= 50 1.87
Severe
>= 50 1.87
Severe
>= 50 1.83
Severe
>= 50 1.83
88
88
88
88
88
88
88
88
88
88
88
88
88
88
88
88
88
88
88
88
66
66
66
66
66
66
66
66
66
66
66
66
66
66
66
66
66
66
66
66
103
103
103
92
92
92
92
92
92
92
92
92
92
92
92
92
86
86
86
86
Black
Black
Black
Black
Black
Black
Black
Black
Black
Black
Black
Black
Black
Black
Black
Black
Black
Black
Black
Black
129
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
130
Here rows 55, 65, and 68 are in fact change rows. Both rows 55 and 68
record changes in the subjects weight. Row 65 records a change in the
glyco variable (i.e., the serum concentration of alpha-1-acid glycoprotein).
3.5 Chapter Summary

In this chapter we have shown examples of constructing, summarizing, and
graphically displaying groupedData objects. These objects include the data,
stored as a data frame, and a formula that designates dierent variables
as a response, a primary covariate, and as one or more grouping factors.
Other variables can be designated as outer or inner factors relative to the
grouping factors. Accessor or extractor functions are available to extract
either the formula for these variables or the value of these variables.
Informative and visually appealing trellis graphics displays of the data
can be quickly and easily generated from the information that is stored
with the data. The regular data summary functions in S can be applied to
the data as well as the gsummary and gapply functions that are especially
designed for these data.
Informative plots and summaries of the data are very useful for the preliminary phase of the statistical analysis. Many important features of the
data are identied at this stage, but usually one is interested in going a
step further in the analysis and tting parametric models, such as the linear
mixed-eects models described in the next chapter.
Exercises
1. In Figure 3.6 (p. 113), the twelve panels in the plot of the CO2 data
were laid out as two rows of six columns. Create a plot of these
data arranged as four rows of three columns. You should insert some
space between the second and third rows to separate the panels for
the Mississippi plants from those for the Quebec plants.
2. Use the outer argument to the plot function to produce a plot of the
CO2 data similar to Figure 8.15 (p. 369) where each panel displays
the data for all three plants at some combination of Variety and
Treatment. Produce two such plots of these data, each laid out as two
rows by two columns. One plot should have the rows determined by
Variety and the columns by Treatment. This arrangement makes it
easy to assess the eect of Treatment within a Variety. The other plot
should have rows determined by Treatment and columns by Variety,
allowing easy assessment of the eect of Variety within Treatment.
Exercises
131
3. The Dialyzer data, described in Appendix A.6 and used in some

examples in 5.4 and 8.3.3, consists of observations of ultraltration
rates at dierent transmembrane pressures on dierent subjects. The
QB variable, which indicates the blood ow rate used for the subject,
is outer to the grouping factor Subject.
(a) Check if these data are balanced with respect to the number of
observations on each Subject.
(b) Check if these data are balanced with respect to the number
of observations and the values of the transmembrane pressure.
Verify your result using the isBalanced function.
(c) Produce a plot of the ultraltration rate versus transmembrane
pressure by subject.
(d) Check with gsummary that QB is invariant within each Subject.
Determine which Subjects are at which QB levels.
(e) Recreate the plot from part (c) arranging for the panels corresponding to subjects at a QB of 200 dl/min to be separated from
those at a QB of 300 dl/min.
(f) Use the outer argument to plot as outer = TRUE or outer = ~QB
to create a plot of ultraltration rate versus transmembrane
pressure by subject divided into two panels according to blood
ow rate. Compare this plot to Figure 5.1 (p. 215). Note the
change in the aspect ratio of the panels relative to the plots in
parts (c) and (d).
(g) Use the aspect argument in a plot like the last one to produce
a plot similar to Figure 5.1.
(h) Use gsummary or gapply to determine the maximum observed
ultraltration rate by subject. Produce dotplots or boxplots of
this maximum rate dividing the subjects according to blood ow
rate (QB). Does the maximum ultraltration rate appear to be
related to the blood ow rate?
4. In the DNase data, described in 3.3.2, there are two measurements
of the optical density at each DNase concentration within each run.
When such replicate observations are available, one can check the
assumption of constant variance for the ij in a linear mixed-eects
model by plotting the logarithm of the standard deviation of the
replicate measurements versus the logarithm of their average (Box,
Hunter and Hunter, 1978, 7.8).
(a) Determine the average optical density at each set of replicate
observations. One way to do this would be to create a copy of
the groupedData object DNase redening the display formula to
be density ~1 | Run/conc. Applying gsummary to this object will
produce the average optical density for each pair of replicates.
132
(b) Determine the standard deviation of the optical density at each

set of replicate observations. Note that some of these standard
deviations should be zero because the replicate observations are
equal. Due to numerical round-o some equal replicates may
produce a standard deviation that is very small but not exactly
zero. Use a boxplot of the logarithm of the standard deviations
to check for these. You may wish to replace those small values
with zero.
(c) Plot the logarithm of the standard deviation versus the logarithm of the average. Estimate the slope of a straight line t to
these points.
4
Fitting Linear Mixed-Eects Models
As seen in Chapter 1, mixed-eects models provide a exible and powerful

tool for analyzing balanced and unbalanced grouped data. These models
have gained popularity over the last decade, in part because of the development of reliable and ecient software for tting and analyzing them. The
linear and nonlinear mixed-eects (nlme) library in S is an example of such
software. We describe the lme function from that library in this chapter,
as well the methods for displaying and comparing tted models created by
this function.
The rst section gives a brief review of the standard linear modeling
facilities in S to introduce the general style of the S modeling functions,
classes, and methods that are used with the nlme library. The lmList function, used to obtain separate lm ts according to the levels of a grouping
variable, is described and illustrated through examples.
The next section describes the linear mixed-eects modeling capabilities
in S. The S modeling function lme is described, together with its associated methods. Its use is illustrated through examples, including single- and
multilevel grouped data.
After a model has been t to the data, it is important to examine whether
the underlying assumptions appear to be violated. Graphical methods and
numerical summaries for assessing the validity of the assumptions in a linear
mixed-eects model are described in 4.3.
An inside-out model building approach is adopted here, starting with
individual ts by group, using plots of the individual coecients to decide
on the random-eects structure, and nally tting a mixed-eects model
134
4. Fitting Linear Mixed-Eects Models
to the complete data. We make extensive use of examples to introduce and

illustrate the available functions and methods.
In this chapter we will restrict our attention to models in which the
within-group errors are independent and have equal variance. Models with
more complex within-group covariance structures, such as the heterogeneous AR(1) structure, will be explored in detail in Chapter 5.

S oers a variety of functions and methods for tting and manipulating
linear models. The two main modeling functions are lm, for linear regression
models, and aov, for analysis of variance models. These two functions have
similar syntax and generate similar tted objects. We concentrate here on
lm. A typical call to lm is of the form
lm(formula, data)
where formula species the linear model to be tted and data gives a data
frame in which the variables in formula are to be evaluated. Several other
arguments to lm are available and are described in detail in Chambers and
Hastie (1992, Chapter 4) and also in Venables and Ripley (1999, Chapter
6).
The formula language used in the formula argument gives lm great exibility in specifying linear models. The formulas use a version of the syntax
dened by Wilkinson and Rogers (1973), which translates a linear model
like y = 0 + 1 x1 + 2 x2 + into the S expression
y ~ 1 + x1 + x2
The ~ is read is modeled as. The expression on the left-hand side of the
~ species the response to be modeled. The expression on the right-hand
side describes the covariates used and the ways they will be combined to
form the model. The expression does not include the coecients (the s).
They are implicit.
The constant term 1 is included by default and does not need to be given
explicitly in the model. If a model without an intercept term is desired, a
-1 must be included in the formula. The covariates can be factors, ordered
factors, continuous variables, matrices, or data frames. Any function of a
covariate that returns one of these types of covariate can also be used on
the right-hand side of the formula. Function calls are also allowed on the
left-hand side of the formula. For example,
log(y) ~ exp(x1) + cos(x2)
is a valid lm formula. An interaction between two covariates is denoted by

the : operator. Nesting of a covariate within a factor is denoted by the
135
%in% operator. More detailed references on the formula language include

Chambers and Hastie (1992, Chapter 2) and Venables and Ripley (1999,
6.2).
The lm function operates in a style common to most modeling functions
in S, in particular lme and nlme. A call to lm returns a tted object of class
lm to which several generic functions can be applied. These can display
the results of the t (print and summary), produce diagnostic plots (plot),
return predictions (predict), extract components (fitted, residuals, and
coef), update the original model (update), or compare dierent tted objects (anova).
To illustrate some of these capabilities, we revisit the orthodontic growth
curve data of 1.4. Suppose that we initially ignore the grouping structure
in the data and t a single linear regression model of distance on age to
the data from all the subjects. The corresponding call to lm is
> fm1Orth.lm <- lm( distance ~ age, Orthodont )
A brief description of the results is provided by the print method which is

called implicitly when the tted object is to be displayed.
> fm1Orth.lm
# equivalent to print( fm1Orth.lm )
Call:
lm(formula = distance~age, data = Orthodont)
Coefficients:
(Intercept)
age
16.761 0.66019
Diagnostic plots for assessing the quality of the t are obtained using the
method for the plot generic function
> par( mfrow=c(3,2) )
> plot( fm1Orth.lm )
# arrange 6 separate plots on a page

# Figure 4.1
There is considerable variability remaining after the t, as shown in the

residual plots. This is not surprising, as we know that the simple linear
regression model does not represent the structure of the data well. Apparently there are some observations with unusual inuence on the t,
especially observations 39 and 104. Furthermore, the normal probability
plot of the residuals suggests that the error distribution has heavier tails
than expected from normally distributed variates.
Suppose that we now want to test for possible dierences in intercept or
in slope between boys and girls. We can use the following model,
distance = 0 + 1 Sex + 2 age + 3 age Sex + ,
(4.1)
-4
-6
2.0
1.5
23
24
25
0.0
22
23
24
6
0
Residuals
-4
-6
23
24
25
26
-2
-1
-2
0.0
f-value
0.08
0.06
0.0
-4
-6
-4
-6
101
40
0.04
0.8
104
Cooks Distance
0.4
0.02
4
2
Quantiles of Standard Normal
Residuals
Fitted Values
distance
0
Fitted : age
-2
104
22
0.0
26
35 39
-2
25
Fitted : age
30
25
20
distance
26
Fitted : age
104
22
104
39
35
sqrt(abs(resid(fm1Orth.lm)))
4
2
0
-2
Residuals
2.5
1.0
39
35
0.5
136
0.4
0.8
20
40
60
80
100
Index
FIGURE 4.1. Diagnostic plots for the simple linear regression model t of the
orthodontic growth curve data.
137
with Sex representing a binary variable taking values 1 for boys and 1 for
girls. The parameters 1 and 3 represent, respectively, the intercept and
slope gender eects. We can t this model in S with another call to lm or
by using update on the previous tted model and redening the formula.
> fm2Orth.lm <- update( fm1Orth.lm, formula = distance ~ Sex*age )
The expression Sex*age in a linear model formula crosses the Sex and age
factors. This means it generates the main eects for these factors and their
interaction. It is equivalent to Sex + age + Sex:age.
The summary method displays the results in more detail. In particular,
it provides information about the marginal signicance of the parameter
estimates.
> summary( fm2Orth.lm )
Call: lm(formula = distance ~Sex + age + Sex:age, data = Orthodont)
Residuals:
Min
1Q Median
3Q Max
-5.62 -1.32 -0.168 1.33 5.25
Coefficients:
(Intercept)
Sex
age
Sex:age
Value Std. Error t value Pr(>|t|)

16.857
1.109
15.194
0.000
0.516
1.109
0.465
0.643
0.632
0.099
6.394
0.000
-0.152
0.099
-1.542
0.126
Residual standard error: 2.26 on 104 degrees of freedom

Multiple R-Squared: 0.423
F-statistic: 25.4 on 3 and 104 d.o.f., the p-value is 2.11e-12
Correlation of Coefficients:
(Intercept)
Sex
age
Sex 0.185
age -0.980
-0.181
Sex:age -0.181
-0.980 0.185
The p-values for the Sex and Sex:age coecients suggest that there is no
gender eect on the orthodontic measurement growth. Because the t-test
is only measuring the marginal signicance of each term in the model, we
should proceed with caution and delete one term at a time from the model.
Deleting rst the least signicant term, Sex, we get:
> fm3Orth.lm <- update( fm2Orth.lm, formula = . ~ . - Sex )
. . .
Coefficients:
(Intercept) 16.761
1.086
15.432
0.000
138

F11
F04
F03
F08
F02
F07
F05
F01
F06
F09
F10
M10
M01
M04
M06
M15
M09
M14
M13
M12
M03
M08
M07
M11
M02
M05
M16
-6
-4
-2
resid(fm2Orth.lm)
FIGURE 4.2. Residual plots corresponding to the fm2Orth.lm tted object, by

subject.
age
Sex:age
. . .
0.640
-0.107
0.097
0.020
6.613
-5.474
0.000
0.000
By convention, the .~. expression represents the formula in the object being
updated and the - operator is used to delete terms from the model.
The Sex:age coecient now becomes very signicant, indicating that the
growth patterns are dierent for boys and girls. Because the lm t is not
adequate for these data, we will postpone further discussion of these modelbuilding issues until the linear mixed-eects model has been described.
The grouped nature of these data, with repeated measures on each subject at four dierent years, violates the basic assumption of independence
that underlies the statistical methods used in lm. Boxplots of the fm2Orth.lm
residuals by subject show this.
> bwplot( getGroups(Orthodont)~resid(fm2Orth.lm) )
# Figure 4.2
The most important feature observed in Figure 4.2 is that residuals corresponding to the same subject tend to have the same sign. This indicates the
need for a subject eect in the model, which is precisely the motivation
for linear mixed-eects models.
139
4.1.1 Separate lm Fits per Group: the lmList Function

The rst step in the model-building process for a linear mixed-eects model,
after the functional form of the model has been decided, is choosing which
parameters in the model, if any, should have a random-eect component
included to account for between-group variation. The lmList function and
the methods associated with it are useful for this.
A typical call to lmList is
lmList( formula, data )
where the right-hand side of the formula consists of two parts separated by
the | operator. The rst part species the linear model to be tted to each
subset of data; the second part species the grouping factor. Any linear
formula allowed in lm can also be used as a model formula with lmList.
The data argument gives the data frame in which to nd the variables used
in formula.
Continuing with the analysis of the orthodontic data, we see from a Trellis
plot of these data (Figure 1.11, page 31) that a simple linear regression
model of distance as a function of age may be suitable. We t this by
> fm1Orth.lis <- lmList( distance ~ age | Subject, Orthodont )
If data is a groupedData object (see Chapter 3), the grouping variable can
be omitted from formula, being extracted from the group formula in data.
> getGroupsFormula( Orthodont )
~ Subject
so an alternative call to lmList to obtain the same tted object is

> fm1Orth.lis <- lmList( distance ~ age, Orthodont )
Because the lmList function is a generic function (Chambers and Hastie,

1992, Appendix A) with dierent methods for arguments of dierent classes,
this same t can be specied in an even simpler way. If the rst argument
to lmList is a groupedData object, the display formula for this object is
used to create a default model formula and to extract the grouping variable expression. Because we are using the same grouping, response, and
covariate in our lmList t as in the display formula
> formula( Orthodont )
we can obtain the same tted model object with the simpler call
> fm1Orth.lis <- lmList( Orthodont )
Objects returned by lmList are of class lmList, for which several display
and plot methods are available. Table 4.1 lists some of the most important
methods for class lmList. We illustrate the use of some of these methods
below.
The print method displays minimal information about the tted object.
140
TABLE 4.1. Main lmList methods.

augPred
coef
fitted
fixef
intervals
lme
logLik
pairs
plot
predict
print
qqnorm
ranef
resid
summary
update
predictions augmented with observed values

coecients from individual lm ts
tted values from individual lm ts
average of individual lm coecients
condence intervals on coecients
linear mixed-eects model from lmList t
sum of individual lm log-likelihoods
scatter-plot matrix of coecients or random eects
diagnostic Trellis plots
predictions for individual lm ts
brief information about the lm ts
normal probability plots
deviations of coecients from average
residuals from individual lm ts
more detailed information about lm ts
update the individual lm ts
> fm1Orth.lis
Call:
Model: distance ~ age | Subject
Data: Orthodont
Coefficients:
(Intercept)
age
M16
16.95 0.550
. . .
F11
18.95 0.675
The residual standard error given in the output is the pooled estimate of
the standard error calculated from the individual lm ts by group. More
detailed output can be obtained using the summary method.
> summary( fm1Orth.lis )
Call:
Model: distance ~ age | Subject
Data: Orthodont
Coefficients:
(Intercept)
Value Std. Error t value
Pr(>|t|)
M16 16.95
3.2882 5.15484 3.6952e-06

2.0
141
M13
age
1.5
1.0
0.5
10
15
20
25
(Intercept)
FIGURE 4.3. Pairs plot for fm1Orth.lis.
M05 13.65
. . .
F11 0.675
3.2882 4.15124 1.1817e-04

0.29293 2.30428 2.5081e-02
Diagnostic plots can be obtained using the plot method, as described in

4.3.
The main purpose of the preliminary analysis provided by lmList is to
give an indication of what random-eects structure to use in a linear mixedeects model. We must decide which random eects to include in a model
for the data, and what covariance structure these random eects should
have. The pairs method provides one view of the random-eects covariance
structure.
> pairs( fm1Orth.lis, id = 0.01, adj = -0.5 )
# Figure 4.3
The id argument is used to identify outlierspoints outside the estimated

probability contour at level 1-id/2 will be marked in the plot. We see that
subject M13 has an unusually low intercept, compensated by a large slope.
There appears to be a negative correlation between the intercept and slope
estimates. Those with experience analyzing regression models may already
have guessed why this pattern occurs. It is because all the data were collected between age 8 and age 14, but the intercept represents the measurement at age 0. This causes a high negative correlation (0.98) between
estimates of the slopes and the intercepts. We can remove this correlation
by centering the data. In this case, we t distance as a linear function of
142

2.0
M13
I(age - 11)
1.5
1.0
0.5
20
22
24
26
28
(Intercept)
FIGURE 4.4. Pairs plot for fm2Orth.lis with ages centered at 11 years.
age - 11. The two quantities being estimated then are the distance at 11
years of age and the slope, or growth rate. We t this revised model with
> fm2Orth.lis <- update( fm1Orth.lis, distance ~ I(age-11) )
The corresponding pairs plot (Figure 4.4) does not suggest any correlation
between the intercept estimates and the slope estimates. It is clear that the
orthodontic distance for subject M13 has grown at an unusually fast rate, but
his orthodontic distance at age 11 was about average. Both intercept and
slope estimates seem to vary with individual, but to see how signicantly
they vary among subjects we need to consider the precision of the lmList
estimates. This can be evaluated with the intervals method.
> intervals( fm2Orth.lis )
, , (Intercept)
lower
est.
M16 21.687 23.000
M05 21.687 23.000
M02 22.062 23.375
. . .
F04 23.562 24.875
F11 25.062 26.375
upper
24.313
24.313
24.688
26.188
27.688
, , I(age - 11)
lower est. upper
M16 -0.037297 0.550 1.1373
M05 0.262703 0.850 1.4373

(Intercept)
F04
F03
F02
F07
F05
F01
|
|
F06
F09
|
|
|
M13
|
|
22
|
|
|
|
26
28
30
-0.5
|
|
|
|
|
|
|
|
0.0
24
|
|
|
|
|
|
M08
|
|
M14
|
|
|
|
M02
|
|
20
M07
|
|
M04
M03
|
|
M12
M06
M11
|
|
M01
18
|
|
|
|
|
|
|
|
|
|
|
M16
|
|
|
|
M15
M09
|
|
M10
M05
|
|
F08
F10
I(age - 11)
F11
143
|
|
0.5
1.0
1.5
2.0
2.5
FIGURE 4.5. Ninety-ve percent condence intervals on intercept and slope for
each subject in the orthodontic distance growth data.
M02 0.187703 0.775 1.3623

. . .
F04 -0.112297 0.475 1.0623
F11 0.087703 0.675 1.2623
As often happens, displaying the intervals as a table of numbers is not very

informative. We nd it much more eective to plot these intervals using
> plot( intervals(fm2Orth.lis) )
# Figure 4.5
The individual condence intervals in Figure 4.5 give a clear indication

that a random eect is needed to account for subject-to-subject variability
in the intercept. Except for subject M13, all condence intervals for the
slope overlap, so perhaps this parameter can be regarded as a xed eect
in the mixed-eects model. We will explore these questions in 4.2.1, while
describing the lme function.
To further illustrate the capabilities of lmList, we consider data on radioimmunoassays of the protein Insulin-like Growth Factor (IGF-I) presented
in Davidian and Giltinan (1995, 3.2.1, p. 65). The data are from quality
control radioimmunoassays for ten dierent lots of a radioactive tracer used
144
IGF-I concentration (ng/ml)
0
8
10
20
30
40
50
10
20
30
40
50
10
8
6
4
2
10
10
8
6
4
2
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Tracer age (days)
FIGURE 4.6. Estimated concentration of the protein Insulin-like Growth Factor

(IGF-I) versus age of radioactive tracer for ten lots of tracer.
in the calibration of IGF-I concentration measurements. They are described

in more detail in Appendix A.11.
> IGF
Grouped Data: conc ~ age | Lot
Lot age conc
1
1
7 4.90
2
1
7 5.68
. . .
236 10 11 5.30
237 10 13 5.63
This data set, displayed in Figure 4.6, is an example of unbalanced, repeated

measures data. We do not consider these data to be longitudinal because
dierent tracer samples are used at each radioimmunoassay. This reduces
the potential for serial correlation in the responses.
The primary purpose of the IGF-I experiment was to investigate possible
trends in control values with tracer age, which would indicate tracer decay
within the usual storage period. We can investigate this by testing if the
slope of a simple linear regression model is signicantly dierent from zero.
We must account for both the within-lot and the between-lot variability
when tting the model and testing for the signicance of the slope. A
linear mixed-eects model will do this, but rst we investigate the sources
of variation in the data by tting separate regression lines to each lot. As
the xed-eects formula coincides with the display formula in IGF, we can
use the simple form of the call to lmList
> fm1IGF.lis <- lmList( IGF )
> coef( fm1IGF.lis )
(Intercept)
age
9
5.0986 0.0057276
6
4.6300 0.1700000

(Intercept)
Lot
|||
|||
|
|
1
6
|||
|
|||
|
|
2
10
|||
|
|||
|
age
145
|||
6
-1.0
-0.5
0.0
0.5
1.0
FIGURE 4.7. Ninety-ve percent condence intervals on intercept and slope for
each lot in the IGF data.
1
10
2
8
5
4
3
7
5.4929
6.0516
5.4764
5.5922
5.3732
5.5768
5.2788
5.2069
-0.0077901
-0.0473282
-0.0144271
0.0060638
-0.0095140
-0.0166578
0.0100830
0.0093136
A quick look at the individual coecient estimates indicates that Lot 6 is

unusual. It has a low intercept compensated by a high slope. Examination
of Figure 4.6 shows that this lot has only four observations and that these
are at nearby tracer ages. The coecients from the individual t to this lot
are unreliable. We will return to this issue in 4.2.1.
The plot of the individual 95% condence intervals, shown in Figure 4.7,
provides some insight about lot-to-lot variation in the parameter estimates.
> plot( intervals(fm1IGF.lis) )
# Figure 4.7
Because of the imbalance in the data, these condence intervals have very
dierent lengths. There is little indication of lot-to-lot variation in either
the intercept or the slope estimates, since all condence intervals overlap.
A xed-eects model seems adequate to represent the IGF data.
> fm1IGF.lm <- lm( conc ~ age, data = IGF )
> summary( fm1IGF.lm )
. . .
Coefficients:
(Intercept)
5.351
0.104
51.584
0.000
age -0.001
0.004
-0.170
0.865
. . .
146
There does not appear to be a signicant tracer decay within the 50-day
period over which the data were collected. A linear mixed-eects model for
the IGF data will be considered in the next section.
4.2 Fitting Linear Mixed-Eects Models with lme

The general formulation of a linear mixed-eects model, as well as the estimation methods used to t it, have been described in Chapter 2. In this
section, we concentrate on the capabilities available in the nlme library for
tting such models. We initially consider lme ts for single-level grouped
data with general covariance structures for the random eects. Fitting models with patterned covariance structures for the random eects is described
in 4.2.2. In 4.2.3, we describe how to t multilevel models with lme.
4.2.1 Fitting Single-Level Models

We use the lme function to t linear mixed-eects models by maximum likelihood or by restricted maximum likelihood (the default). Several optional
arguments can be used with this function, but a typical call is
lme( fixed, data, random )
The rst argument is a two-sided linear formula specifying the xed eects
in the model. The third argument is typically given as a one-sided linear
formula, specifying the random eects and the grouping structure in the
model. For the orthodontic data, with the ages centered at 11 years, these
formulas are:
fixed = distance ~ I(age-11), random = ~ I(age-11) | Subject
Note that the response is specied only in the fixed formula. If the random
formula is omitted, its default value is taken as the right-hand side of the
fixed formula. This describes a model in which every xed eect has an
associated random eect. To use this default, data must be a groupedData
object, so the formula for the grouping structure can be obtained from the
display formula.
The argument data species a data frame in which the variables named in
fixed and random can be evaluated. When data inherits from class groupedData, the expression dening the grouping structure can be omitted in
random.
A simple call to lme to t the orthodontic data model is
> fm1Orth.lme <- lme( distance ~ I(age-11), data = Orthodont,
+
random = ~ I(age-11) | Subject )
or, because Orthodont is a groupedData object and, by default, the random

eects have the same form as the xed eects
147
TABLE 4.2. Main lme methods.

ACF
anova
augPred
coef
fitted
fixef
intervals
logLik
pairs
plot
predict
print
qqnorm
ranef
resid
summary
update
Variogram
empirical autocorrelation function of within-group residuals

likelihood ratio or conditional tests
estimated coecients for dierent levels of grouping
tted values for dierent levels of grouping
xed-eects estimates
condence intervals on model parameters
log-likelihood at convergence
predictions for dierent levels of grouping
brief information about the t
random-eects estimates
residuals for dierent levels of grouping
more detailed information about the t
update the lme t
semivariogram of within-group residuals
> fm1Orth.lme <- lme( distance ~ I(age-11), data = Orthodont )
Because the lme function is generic, the model can be described in several
dierent ways. For example, there is an lme method for lmList objects. When
an lmList object, such as fm2Orth.lis in 4.1.1, is given as the rst argument
to lme, it provides default values for fixed, random, and data. We can create
the same tted model with the simple call
> fm1Orth.lme <- lme( fm2Orth.lis )
One advantage of this method is that initial estimates for the parameters
in the proled (restricted-)likelihood of the mixed-eects model are automatically calculated from the lmList object.
The tted object is of the lme class, for which several methods are available to display, plot, update, and further explore the estimation results.
Table 4.2 lists the most important methods for class lme. We illustrate the
use of these methods through the examples in the next sections.
Orthodontic Growth Curve
As for all the classes of objects representing tted models, the print method
for the lme class returns a brief description of the estimation results. It
prints the estimates of the standard deviations and the correlations of the
148
random eects, the within-group standard error, and the xed eects. For
the fm1Orth.lme object it gives
> fm1Orth.lme
Data: Orthodont
Fixed: distance ~ I(age - 11)
(Intercept) I(age - 11)
24.023
0.66019
Random effects:
Formula: ~ I(age - 11) | Subject
StdDev
Corr
(Intercept) 2.13433 (Inter
I(age - 11) 0.22643 0.503
Residual 1.31004
One of the questions of interest for the orthodontic growth data is whether
boys and girls have dierent growth patterns. We can assess this by tting
the model
> fm2Orth.lme <- update(fm1Orth.lme,fixed = distance~Sex*I(age-11))
Note that lmList cannot be used to test for gender dierences in the orthodontic growth data, as it estimates individual coecients for each subject. In general, we will not be able to use lmList to test for dierences due
to factors that are invariant with respect to the groups.
Some more detailed output is supplied by summary.
> summary( fm2Orth.lme )
Data: Orthodont
AIC
BIC logLik
451.35 472.51 -217.68
Random effects:
StdDev
Corr
I(age - 11) 0.18035 0.206
Residual 1.31004
Fixed effects: distance ~ Sex + I(age - 11) + Sex:I(age - 11)
(Intercept)
Sex
I(age - 11)
Sex:I(age - 11)
Correlation:
149

23.808
0.38071 79 62.537 <.0001
-1.161
0.38071 25 -3.048 0.0054
0.632
0.06737 79
9.381 <.0001
-0.152
0.06737 79 -2.262 0.0264
(Intrc
Sex I(-11)
Sex 0.185
I(age - 11) 0.102 0.019
Sex:I(age - 11) 0.019 0.102 0.185
Min
Q1
Med
Q3
Max
-3.1681 -0.38594 0.0071041 0.44515 3.8495
The small p-values associated with Sex and Sex:I(age-11) in the summary
output indicate that boys and girls have signicantly dierent orthodontic
growth patterns.
The fitted method is used to extract the tted values from the lme
object, using the methodology described in 1.4.2. By default, the withingroup tted values, that is, the tted values corresponding to the individual
coecient estimates, are produced. Population tted values, based on the
xed-eects estimates alone, are obtained setting the level argument to 0
(zero). Both types of tted values can be simultaneously obtained with
> fitted( fm2Orth.lme, level = 0:1 )
fixed Subject
1 22.616 24.846
2 24.184 26.576
3 25.753 28.307
. . .
Residuals are extracted with the resid method, which also takes a level
argument.
> resid( fm2Orth.lme, level = 1 )
M01
M01
M01
M01
M02
M02
M02
M02
1.1543 -1.5765 0.69274 0.96198 0.22522 -0.29641 -1.318 0.66034
. . .
F10
F10
F10
F10
F11
F11
F11
-1.2233 0.44296 -0.39073 -0.72443 0.28277 -0.37929 1.4587
F11
0.29661
attr(, "label"):
[1] "Residuals (mm)"
By default, the raw, or response residuals, given by the observed responses

minus the tted values, are calculated. Standardized, or Pearson residuals,
150
corresponding to the raw residuals divided by the estimated within-group

standard deviation, are obtained using
> resid( fm2Orth.lme, level = 1, type = "pearson" )
M01
M01
M01
M01
M02
M02
M02
0.88111 -1.2034 0.5288 0.73431 0.17192 -0.22626 -1.0061
. . .
F09
F10
F10
F10
F10
F11
F11
-0.76369 -0.93382 0.33813 -0.29826 -0.55298 0.21585 -0.28952
F11
F11
1.1135 0.22641
attr(, "label"):
[1] "Standardized residuals"
Partial matching of arguments is used throughout the nlme library, so type

= "p" would suce in this case.
Predicted values are obtained with the predict method. For example, to
predict the orthodontic distance for boy M11 and girl F03 at ages 16, 17 and
18, we rst dene a data frame with the relevant information
> newOrth <- data.frame( Subject = rep(c("M11","F03"), c(3, 3)),
+
Sex = rep(c("Male", "Female"), c(3, 3)),
+
age = rep(16:18, 2) )
and then use

> predict( fm2Orth.lme, newdata = newOrth )
M11
M11
M11
F03
F03 F03
26.968 27.612 28.256 26.614 27.207 27.8
attr(, "label"):
[1] "Predicted values (mm)"
By default, the within-group predictions are produced. To obtain both

population and within-group predictions we use
> predict( fm2Orth.lme, newdata = newOrth, level = 0:1 )
Subject predict.fixed predict.Subject
1
M11
28.891
26.968
2
M11
29.675
27.612
3
M11
30.459
28.256
4
F03
25.045
26.614
5
F03
25.525
27.207
6
F03
26.005
27.800
The predict.fixed column gives the population predictions, while

predict.Subject gives the within-group predictions. We see that M11 is below the boys average, while F11 is above the girls average.
The fm2Orth.lme object corresponds to a restricted-maximum likelihood
t, which tends to produce more conservative estimates of the variance
components. A maximum likelihood t is obtained with
151
> fm2Orth.lmeM <- update( fm2Orth.lme, method = "ML" )

> summary( fm2Orth.lmeM )
Data: Orthodont
AIC
BIC logLik
443.81 465.26 -213.9
Random effects:
StdDev
Corr
I(age - 11) 0.15414 0.234
Residual 1.31004
Fixed effects: distance ~ Sex + I(age - 11) + Sex:I(age - 11)
(Intercept) 23.808
0.37332 79 63.775 <.0001
Sex -1.161
0.37332 25 -3.109 0.0046
I(age - 11)
0.632
0.06606 79
9.567 <.0001
Sex:I(age - 11) -0.152
0.06606 79 -2.307 0.0237
. . .
As expected, the ML estimates of the random-eects standard deviations

are smaller than the corresponding REML estimates. The estimated withingroup residual standard deviations are identical, which generally need not
occur. In general, the xed-eects estimates obtained using ML and REML
will be similar, though not identical, as in this example. Inferences regarding
the xed eects are essentially the same for the two estimation methods,
in this case.
It is instructive, at this point, to compare the individual coecient estimates obtained with lmList to those obtained with lme. The function
compareFits can be used for this. The resulting object has a plot method
that displays these coecients side-by-side.
> compOrth <+
compareFits( coef(fm2Orth.lis), coef(fm1Orth.lme) )
> compOrth
, , (Intercept)
coef(fm2Orth.lis) coef(fm1Orth.lme)
M16
23.000
23.078
M05
23.000
23.128
M02
23.375
23.455
. . .
F04
24.875
24.764
F11
26.375
26.156
, , I(age - 11)
M16
0.550
0.59133
152

+
coef(fm2Orth.lis)
coef(fm1Orth.lme)
(Intercept)
F11
F04
F03
F08
F02
F07
F05
F01
F06
F09
F10
M10
M01
M04
M06
M15
M09
M14
M13
M12
M03
M08
M07
M11
M02
M05
M16
+
+
+
+
+
+
+
+
20
22
+
+
+
+
I(age - 11)
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
24
26
28
+
+
+
0.5
1.0
1.5
2.0
FIGURE 4.8. Individual estimates from an lmList t and from an lme t of the
orthodontic distance growth data
M05
0.850
0.68579
M02
0.775
0.67469
. . .
M13
1.950
1.07385
. . .
F04
0.475
0.63032
F11
0.675
0.74338
> plot( compOrth, mark = fixef(fm1Orth.lme) )
# Figure 4.8
The mark argument to the plot method indicates points in the horizontal
axis where dashed vertical lines should be drawn.
The plots in Figure 4.8 indicate that the individual estimates from the lme
t tend to be pulled toward the xed-eects estimates, when compared
to the lmList estimates. This is typical of linear mixed-eects estimation:
the individual coecient estimates from the lme t represent a compromise
between the coecients from the individual ts corresponding to the lmList
t and the xed-eects estimates, associated with the population averages.
For this reason, these estimates are often called shrinkage estimates, in
the sense that they shrink the individual estimates toward the population
average.
The shrinkage toward the xed eects is particularly noticeable for the
slope estimate of subject M13. As pointed out in 4.2, this subject has an
outlying orthodontic growth pattern, which leads to an abnormally high
estimated slope in the lm t. The pooling of subjects in the lme estimation
gives a certain amount of robustness to individual outlying behavior. This
feature is better illustrated by the comparison of the predicted values from
the two ts, which is obtained with the comparePred function.
> plot( comparePred(fm2Orth.lis, fm1Orth.lme, length.out = 2),

fm2Orth.lis
8
F03
10
153
fm1Orth.lme
13
F04
F11
30
25
20
F10
F09
F06
F01
F05
F07
F02
F08
M13
M14
M09
M15
M06
M04
M01
M10
30
25
20
30
25
20
M16
M05
M02
M11
M07
M08
M03
M12
30
25
20
10
13
10
13
10
13
10
13
Age (yr)
FIGURE 4.9. Individual predicted values from separate lm ts and from an lme
t of the orthodontic distance growth data
layout = c(8,4), between = list(y = c(0, 0.5)) )
# Figure 4.9
The length.out argument species the number of predictions for each tted
object. In this case, because the model is a straight line, only two points
are needed. The plot of the individual predictions for the lmList and lme
ts, shown in Figure 4.9, clearly indicates the greater sensitivity of the
individual lm ts to extreme observations.
It is also interesting to compare the fm2Orth.lme and the fmOrth.lmeM
objects, corresponding, respectively, to REML and ML ts of the same
model. We compare the estimated random eects for each t with the
compareFits function.
> plot( compareFits(ranef(fm2Orth.lme), ranef(fm2Orth.lmeM)),
+
mark = c(0, 0) )
# Figure 4.10
The ML random-eects estimates tend to be closer to zero than the REML

estimates, especially the slope random eects. This will usually occur in
mixed-eects models, because REML estimation generally produces larger
154

+
ranef(fm2Orth.lme)
ranef(fm2Orth.lmeM)
(Intercept)
F11
F04
F03
F08
F02
F07
F05
F01
F06
F09
F10
M10
M01
M04
M06
M15
M09
M14
M13
M12
M03
M08
M07
M11
M02
M05
M16
+
+
+
+
-2
+
+
+
+
+
+
+
+
+
I(age - 11)
+
+
+
+
+
+
+
+
+
+
4
+
+
+
+
+
+
-0.1
+
+
0.0
0.1
0.2
0.3
FIGURE 4.10. Individual random-eects estimates from restricted maximum likelihood (REML) and maximum likelihood (ML) ts of the orthodontic distance
growth data
estimates for the random-eects variances which, in turn, result in less

shrinkage toward zero for the random-eects estimates.
We can also compare the lme t to an lm t without random eects:
> fm4Orth.lm <- lm( distance ~ Sex * I(age-11), Orthodont )
. . .
Coefficients:
(Intercept)
23.808
0.221
107.731
0.000
Sex
-1.161
0.221
-5.251
0.000
I(age - 11)
0.632
0.099
6.394
0.000
Sex:I(age - 11)
-0.152
0.099
-1.542
0.126
. . .
The pointwise estimates of the xed eects are almost identical, but their
standard errors are quite dierent. The lm t has smaller standard errors
for the (Intercept) and Sex xed eects and larger standard errors for the
xed eects involving age. This is because the model used in lm ignores the
group structure of the data and incorrectly combines the between-group
and the within-group variation in the residual standard error. Fixed eects
that are associated with invariant factors (factors that do not vary within
groups) are actually estimated with less precision than suggested by the lm
output, because the contribution of the between-group variation to their
standard error is larger than that included in the lm residual standard
error. Conversely, the precision of the xed eects related to variables that
vary within group are less aected by the between-group variation. In the
terminology of split-plot experiments, (Intercept) and Sex are associated
155
with whole-plot treatments and should be compared to the whole-plot error,

while I(age11) and Sex:I(age11) are related to subplot treatments and
should be tested against the subplot error.
The anova method can be used to compare lme and lm objects.
> anova( fm2Orth.lme, fm4Orth.lm )
Model df
AIC
BIC logLik
fm2Orth.lme
1 8 451.35 472.51 -217.68
fm4Orth.lm
2 5 496.33 509.55 -243.17 1 vs 2 50.977 <.0001
In this case, as evidenced by the low p-value for the likelihood ratio test,
the linear mixed-eects model provides a much better description of the
data than the linear regression model.
Radioimmunoassays of IGF-I
The linear mixed-eects model corresponding to the simple linear regression
of the estimated concentration of IGF-I (yij ) in the jth tracer sample within
the ith lot on the age of the tracer sample (xij ) is
yij = (0 + b0i ) + (1 + b1i ) xij + ij ,

b0i
N (0, ) , ij N 0, 2 ,
bi =
b1i
(4.2)
where 0 and 1 are, respectively, the xed eects for the intercept and the
slope; the bi are the random-eects vectors, assumed to be independent
for dierent lots; and the ij are the independent, identically distributed
within-group errors, assumed to be independent of the random eects.
We t the linear mixed-eects model (4.2) with
> fm1IGF.lme <- lme( fm1IGF.lis )
> fm1IGF.lme
Data: IGF
Fixed: conc ~ age
(Intercept)
age
5.375 -0.0025337
Random effects:
Formula: ~ age | Lot
StdDev
Corr
age 0.0080862 -1
Residual 0.8206310
156
The xed-eects estimates are similar to the ones obtained with the single lm t of 4.1.1. The within-group standard errors are also similar in
the two ts, which suggests that not much is gained by incorporating random eects into the model. The estimated correlation between the random
eects ( 1) gives a clear indication that the estimated random-eects
covariance matrix is ill-conditioned, suggesting that the model may be overparameterized. The condence intervals for the standard deviations and
correlation coecient reinforce the indication of overparameterization.
> intervals( fm1IGF.lme )
Fixed effects:
lower
est.
upper
(Intercept) 5.163178 5.3749606 5.5867427
age -0.012471 -0.0025337 0.0074039
Random Effects:
Level: Lot
lower
est.
upper
sd((Intercept)) 0.0011710 0.0823594 5.792715
sd(age) 0.0013177 0.0080862 0.049623
cor((Intercept),age) -1.0000000 -0.9999640 1.000000
lower
est.
upper
0.7212 0.82063 0.93377
The 95% condence interval for the correlation coecient covers all possible
values for this parameter. There is also evidence of large variability in the
estimates of the (Intercept) and age standard deviations. These issues are
explored in more detail in 4.3.
The primary question of interest in the IGF-I study is whether the tracer
decays with age. We can investigate it with the summary method.
> summary( fm1IGF.lme )
. . .
Fixed effects: conc ~ age
(Intercept) 5.3750
0.10748 226 50.011 <.0001
age -0.0025
0.00504 226 -0.502 0.6159
. . .
As with the lm results of 4.1.1, there is no signicant evidence of tracer

decay with age, for the 50-day period in which the observations were collected. Note that the standard errors for the estimates are very similar to
the ones in the lm t.
The plots of the coecient estimates corresponding to fm1IGF.lis and
fm1IGF.lme are shown in Figure 4.11. Once again we observe a shrinkage

+
coef(fm1IGF.lis)
coef(fm1IGF.lme)
(Intercept)
age
+
+
7
3
4
5
8
2
10
1
6
9
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
4.6
4.8
5.0
5.2
157
5.4
5.6
5.8
6.0 0.05
0.0
0.05
0.10
0.15
FIGURE 4.11. Individual estimates from separate lm ts and from an lme t of

the IGF-I radioimmunoassays data.
toward the population mean pattern for the individual lme coecients.
The IGF-I data contain several outlying observations and the dramatic
shrinkage in the coecient estimates observed for some of the lots reects
the greater robustness of the lme t. This is better illustrated by comparing
the individual predictions under each t, as presented in Figure 4.12. The
dierences in the predicted values for the two ts are particularly dramatic
for lots 6 and 10, both of which have observations only over a very limited
time range. For lot 6 a single low observation at one of the earliest times
causes a dramatic change in the estimate of both the slope and intercept
when this lot is t by itself. When it is combined with the other lots in a
mixed-eects model the eect of this single observation is diminished. Also
notice that the outlying observations for lots 4, 3, and 7 have very little
eect on the parameter estimates because in each of these lots there are
several other observations at times both above and below the times of the
aberrant observations.
4.2.2 Patterned VarianceCovariance Matrices for the

Random Eects: The pdMat Classes
The models considered in 4.2.1 do not assume any special form for the
random-eects variancecovariance matrix . In many practical applications, however, we will wish to restrict to special forms of variance
covariance matrices that are parameterized by fewer parameters. For example, we may be willing to assume that the random eects are independent,
in which case would be diagonal, or that, in addition to being independent, they have the same variance, in which case would be a multiple of
the identity matrix.
The nlme library provides several classes of positive-denite matrices,
the pdMat classes, that are used to specify patterned variancecovariance
matrices for the random eects. Table 4.3 lists the standard pdMat classes
included in the nlme library. The default class of positive-denite matrix
158

fm1IGF.lis
0
10
20
30
40
fm1IGF.lme
50
10
20
30
40
50
IGF-I concentration (ng/ml)
12
10
8
6
4
2
9
10
12
10
8
6
4
2
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Tracer age (days)
FIGURE 4.12. Individual predicted values from separate lm ts and from an lme
t of the IGF-I radioimmunoassays data.
TABLE 4.3. Standard pdMat classes.

pdBlocked
pdCompSymm
pdDiag
pdIdent
pdSymm
block-diagonal
compound-symmetry structure
diagonal
multiple of an identity
general positive-denite matrix
for the random eects in the nlme library is pdSymm, corresponding to a

general symmetric positive-denite matrix.
A function that creates an object of a given class is called a constructor
for that class. The pdMat constructors have the same name as their corresponding classes so, for example, the constructor for the pdDiag class is
also called pdDiag.
Because initial values for can be derived internally in the lme function,
the pdMat constructors are typically used only to specify a pdMat class and
a formula for the random-eects model. For example,
> pd1 <- pdDiag( ~ age )
> pd1
Uninitialized positive definite matrix structure of class pdDiag
> formula( pd1 )
~ age
159
creates an object of class pdDiag, with a formula attribute specifying a

random-eects model, but with no initial value assigned to the matrix.
The pdMat object returned by the constructor is passed through the random
argument to the lme function. For example, to specify a diagonal variance
covariance structure for the random eects in the IGF-I example of 4.2.1,
we use
> fm2IGF.lme <- update( fm1IGF.lme, random = pdDiag(~age) )
> fm2IGF.lme
Data: IGF
Fixed: conc ~ age
(Intercept)
age
5.369 -0.0019301
Random effects:
Structure: Diagonal
(Intercept)
age Residual
StdDev: 0.00031074 0.0053722
0.8218
. . .
With the exception of the standard deviation for the (Intercept) random
eect, all estimates are similar to the ones in fm1IGF.lme. We can compare
the two ts with
> anova( fm1IGF.lme, fm2IGF.lme )
Model df
AIC
BIC logLik
fm1IGF.lme
1 6 606.37 627.12 -297.18
fm2IGF.lme
2 5 604.80 622.10 -297.40 1 vs 2 0.43436 0.5099
The large p-value for the likelihood ratio test and the smaller AIC and
BIC values for the simpler model fm2IGF.lme indicate that it should be
preferred.
Because IGF is a groupedData object, the grouping structure does not
need to be given explicitly in random. In cases when both the grouping
structure and a pdMat class are to be declared in random, we use a named
list, with the name specifying the grouping factor.
> update( fm1IGF.lme, random = list(Lot = pdDiag(~ age)) )
The value argument to the constructor is used to assign a value to the

positive-denite matrix. In this case, the random-eects formula needs to
be specied through the form argument.
> pd2 <- pdDiag( value = diag(2), form = ~ age )
160
> pd2
Positive definite matrix structure of class pdDiag representing
[,1] [,2]
[1,]
1
0
[2,]
0
1
> formula( pd2 )
~ age
This can be used to provide initial values for the scaled variancecovariance
matrix of the random eects, D = / 2 in the lme call.
> lme( conc ~ age, IGF, pdDiag(diag(2), ~age) )
Split-Plot Experiment on Varieties of Oats

We now revisit the Oats example of 1.6 and describe alternative ways of
analyzing the split-plot data using pdMat classes. The nal mixed-eects
model resulting from the analysis presented in that section is
yijk = 0 + 1 Nk + bi + bi,j + ijk , i = 1, . . . , 6,
j = 1, . . . 3, k = 1, . . . , 4,

2
ijk N (0, ), bi N 0, 12 , bi,j N 0, 22 ,
(4.3)
where i indexes the Blocks, j indexes the Varieties, and k indexes the Nitrogen concentrations Nk . The intercept is represented by 0 , the Nitrogen
slope by 1 and the yield by yijk . The bi denote the Block random eects,
the bi,j denote the Variety within Block random eects, and the ijk denote the within-group errors. This is an example of a two-level mixed-eects
model, with the bi,j random eects nested within the bi random eects.
The multilevel model capabilities of lme were used in 1.6 to t (4.3). We
recommend tting the model this way, as it uses ecient computational
algorithms designed specically for this type of model. Nevertheless, to
further illustrate the use of the pdMat classes, we consider equivalent singlelevel representations of the same model.
By dening
yi =
yi11
yi12
..
.
i =
yi34
N1
1
0
Xi = 1
0
1
0
i11
i12
..
.
i34
0
N2
0
0
0
0
N3
0

=
0
0
,
0
N4
0
1
bi + bi,1
bi = bi + bi,2 ,
bi + bi,3

,
1
1 0 0
1
Zi = 0 1 0
1 ,
0 0 1
1
161
with denoting the Kronecker product, we can rewrite (4.3) as the singlelevel model

y i = X i + Z i bi + i , bi N (0, ) , i N 0, 2 I ,
(4.4)
where
12 + 22
12
=
12
12
2
1 + 22
12
12
.
12
2
1 + 22
The matrix has a compound symmetry structure, represented in nlme

by the pdCompSymm class. We t (4.4) with
> fm4OatsB <- lme( yield ~ nitro, data = Oats,
+
random =list(Block = pdCompSymm(~ Variety - 1)))
> summary( fm4OatsB )
Data: Oats
AIC
BIC logLik
603.04 614.28 -296.52
Random effects:
Formula: ~ Variety - 1 | Block
Structure: Compound Symmetry
StdDev Corr
VarietyGolden Rain 18.208
VarietyMarvellous 18.208 0.635
VarietyVictory 18.208 0.635 0.635
Residual 12.867
(Intercept) 81.872
6.9453 65 11.788 <.0001
nitro 73.667
6.7815 65 10.863 <.0001
Correlation:
(Intr)
nitro -0.293
Min
Q1
Med
Q3
Max
-1.7438 -0.66475 0.017103 0.54299 1.803
Comparing this with the output of summary(fm4Oats) in 1.6, we see that,

except for the random-eects variancecovariance components, the results
are nearly identical.
Verifying the equivalence of the random-eects variancecovariance components requires some extra work. Note that the variance in the compound symmetric matrix is b2 = 12 + 22 and the correlation is
162

= 12 / 12 + 22 . Therefore, 12 = b2 and 22 = b2 12 . We can
then derive the REML estimates of 1 and 2 from fm4OatsB

1 = 0.63471 18.208 = 14.506,

2 = 18.2082 14.5062 = 11.005,
verifying that they are identical to the estimates corresponding to fm4Oats
in 1.6. Because the REML estimate of in the summary(fm4OatsB) output
was displayed with only three decimal places, we used
> corMatrix( fm4OatsB$modelStruct$reStruct$Block )[1,2]
[1] 0.63471
to obtain the more accurate used to obtain

1 and
2 .
Yet another representation of (4.3) as a single-level model is obtained by
dening
1
bi
1
1
0
0
bi,1
, Wi = 1 0 1 0 1 ,
b
i = b
1
i,2
1 0 0 1
bi,3
1
2
1 0
0
0
0 22 0
0
=
2
0
0 2 0
0
0
0 22
and writing
y i = X i + W i b
i + i ,
b
i N (0, ) ,

i N 0, 2 I . (4.5)
The parameters in (4.3) are equivalent to those in (4.5).

The matrix is structured as a block diagonal matrix, with blocks 12
and 22 I. The pdBlocked class is used to represent block diagonal matrices in
the nlme library. It takes as an argument a list of pdMat objects, specifying
the dierent blocks in the order they appear in the main diagonal of the
corresponding block-diagonal matrix. In the case of , both blocks are
expressed as multiples of the identity matrix, represented by the pdIdent
class in nlme. We can then t (4.5) with
> fm4OatsC <- lme( yield ~ nitro, data = Oats,
+
random=list(Block=pdBlocked(list(pdIdent(~ 1),
+
pdIdent(~ Variety-1)))))
> summary( fm4OatsC )
Data: Oats
AIC
BIC logLik
603.04 614.28 -296.52
Random effects:
163
Composite Structure: Blocked

Block 1: (Intercept)
StdDev:
(Intercept)
14.505
Block 2: VarietyGolden Rain, VarietyMarvellous, VarietyVictory

Formula: ~ Variety - 1 | Block
Structure: Multiple of an Identity
VarietyGolden Rain VarietyMarvellous VarietyVictory
StdDev:
11.005
11.005
11.005
Residual
StdDev:
12.867
(Intercept) 81.872
6.9451 65 11.788 <.0001
nitro 73.667
6.7815 65 10.863 <.0001
Correlation:
(Intr)
nitro -0.293
. . .
Comparing this to the output of summary(fm4Oats) in 1.6 we verify that,

as expected, the two ts are nearly identical.
Cell Culture Bioassay with Crossed Random Eects
Data grouped according to crossed classication factors induce a variance
covariance structured for the observations which can be exibly represented
by mixed-eects models with crossed random eects. These models can also
be t with the lme function. However, unlike in the case of nested random
eects, the underlying estimation algorithm is not optimized to take full
advantage of the sparse structure of design matrices for crossed random
eects.
The crossed random-eects structure is represented in lme by a combination of pdBlocked and pdIdent objects. We illustrate its use with an example
of a cell culture plate bioassay conducted at Searle, Inc. The data, courtesy
of Rich Wolfe and David Lansky, come from a bioassay run on a cell culture
plate with two blocks of 30 wells each. The wells in each block are labeled
according to six rows and ve columns, corresponding to a crossed classication. Within each block, six dierent samples are randomly assigned
to rows and ve serial dilutions are randomly assigned to columns. The
response variable is the logarithm of the optical density measure on a well.
The cells are treated with a compound that they metabolize to produce
the stain. Only live cells can make the stain, so the optical density is a
164

5
dba f
c edb
fc b
e a b cf d
Block
b
a b
fc
f cd
2
1
ae
bf c
a
c
bf
d
d
1
a e
e
-0.2
b fc
fa b
d
cd
0.0
0.2
0.4
0.6
Log-optical density
FIGURE 4.13. Log-optical density measured on the central 60 wells of a cell

culture plate. The wells are divided into two blocks with six rows and ve columns,
with samples being assigned to rows and dilutions to columns. Panels in the plot
correspond to the serial dilutions and symbols refer to the samples.
measure of the number of cells that are alive and healthy. These data are
described in detail in Appendix A.2 and are included in the nlme library
as the groupedData object Assay.
The plot of the log-optical densities, displayed in Figure 4.13, indicates
that the response increases with dilution and is generally lower for treatments a and e. There does not appear to be any interactions between sample
and dilution.
A full factorial model is used to represent the xed eects and three
random eects are used to account for block, row, and column eects, with
the last two random eects nested within block, but crossed with each
other. The corresponding mixed-eects model for the log-optical density
yijk in the jth row, kth column of the ith block, for i = 1, . . . , 2, j =
1, . . . , 6, k = 1, . . . , 5, is
yijk = + j + k + jk + bi + rij + cik + ijk ,

2
bi N 0, 1 , rij N 0, 22 , cik N 0, 32 , ijk N 0, 2 .
(4.6)
The xed eects in (4.6) are , the grand mean, j and k , the sample and
dilution main eects, and jk , the sample-dilution interaction. To ensure
identiability of the xed eects, it is conventioned that 1 = 1 = 1k =
165
j1 = 0, for j = 1, . . . , 6, k = 1, . . . , 5. The random eects in (4.6) are

bi , the block random eect, rij , the row within block random eect, and
cik , the column within block random eect. All random eects are assumed
independent of each other and independent of the within-groups errors ijk .
The crossed random-eects structure in (4.6) can alternatively be represented as the random-eects structure corresponding to a single-level
model, with block as the single grouping variable, and random-eects vecT
tor bi = (bi , ri1 , . . . , ri6 , ci1 , . . . , ci5 ) , i = 1, . . . , 2, with
2
0
0
1
0 .
Var (bi ) = 0 22 I
0
0
32 I
That is, bi has a block diagonal variancecovariance matrix, with diagonal
blocks given by multiples of the identity matrix. This type of variance
covariance structure is represented in S by a pdBlocked object with pdIdent
elements. We t the linear mixed-eects model (4.6) with lme as
> ## establishing the desired parameterization for contrasts
> options( contrasts = c("contr.treatment", "contr.poly") )
> fm1Assay <- lme( logDens ~ sample * dilut, Assay,
+
random = pdBlocked(list(pdIdent(~ 1), pdIdent(~ sample - 1),
+
pdIdent(~ dilut - 1))) )
> fm1Assay
Data: Assay
Log-restricted-likelihood: 38.536
Fixed: logDens ~ sample * dilut
(Intercept) sampleb samplec sampled
samplee samplef dilut2
-0.18279 0.080753 0.13398 0.2077 -0.023672 0.073569 0.20443
dilut3 dilut4 dilut5 samplebdilut2 samplecdilut2
0.40586 0.57319 0.72064
0.0089389
-0.0084953
sampleddilut2 sampleedilut2 samplefdilut2 samplebdilut3
0.0010793
-0.041918
0.019352
-0.025066
samplecdilut3 sampleddilut3 sampleedilut3 samplefdilut3
0.018645
0.0039886
-0.027713
0.054316
samplebdilut4 samplecdilut4 sampleddilut4 sampleedilut4
0.060789
0.0052598
-0.016486
0.049799
samplefdilut4 samplebdilut5 samplecdilut5 sampleddilut5
0.063372
-0.045762
-0.072598
-0.17776
sampleedilut5 samplefdilut5
0.013611
0.0040234
Random effects:
166
StdDev:
(Intercept)
0.0098084
Block 2: samplea, sampleb, samplec, sampled, samplee, samplef

Formula: ~ sample - 1 | Block
samplea sampleb samplec sampled samplee samplef
StdDev: 0.025289 0.025289 0.025289 0.025289 0.025289 0.025289
Block 3: dilut1, dilut2, dilut3, dilut4, dilut5
Formula: ~ dilut - 1 | Block
dilut1
dilut2
dilut3
dilut4
dilut5
StdDev: 0.0091252 0.0091252 0.0091252 0.0091252 0.0091252
Residual
StdDev: 0.041566
Number of Groups: 2
The REML estimates of the standard deviation components in this example

2 = 0.0253,
3 = 0.0091, and
= 0.0416.
are
1 = 0.0098,
The primary question of interest for this experiment is whether there are
signicant dierences among the xed eects, which we investigate with
> anova( fm1Assay )
(Intercept)
1
29 538.03 <.0001
sample
5
29
11.21 <.0001
dilut
4
29 420.80 <.0001
sample:dilut
20
29
1.61 0.1193
As suggested by Figure 4.13, there are signicant dierences among samples

and among dilutions, but no signicant interaction between the two factors.
The small estimated standard deviations in the fm1Assay t suggest that
some, or perhaps all, of the random eects can be eliminated from (4.6).
However, because our purpose here is just to illustrate the use of lme with
crossed random eects, we do not pursue the analysis of the Assay data any
further.
New pdMat classes, representing user-dened positive-denite matrix
structures, can be added to the set of standard classes in Table 4.3 and
used with the lme and nlme functions. For this, one must specify a constructor function, generally with the same name as the class, and, at a
minimum, methods for the functions pdConstruct, pdMatrix, and coef. The
pdDiag constructor and methods can serve as templates for these.
167
4.2.3 Fitting Multilevel Models

Linear mixed-eects models with nested grouping factors, generally called
multilevel models (Goldstein, 1995) or hierarchical linear models (Bryk and
Raudenbush, 1992), can be tted with the lme function, just like single-level
models. The only dierence is in the specication of the random argument,
which must provide information about the nested grouping structure and
the random-eects model at each level of grouping. We describe the multilevel model capabilities in lme through the analyses of two examples from
Integrated Circuiting (IC) manufacturing.
Thickness of Oxide Coating on a Semiconductor
Littell et al. (1996) describe data from a passive data collection study in the
IC industry in which the thickness of the oxide coating layer was measured
on three randomly selected sites in each of three wafers from each of eight
lots randomly selected from the population of lots. There are two nested
grouping levels in this example: lot and wafer within lot. The objective
of the study was to estimate the variance components associated with the
dierent levels of nesting and the within-group error, to evaluate assignable
causes of variability in the oxide deposition process. These data are also
described in Appendix A.20 and are included in the groupedData object
Oxide in the nlme library.
The plot of the data, shown in Figure 4.14, suggests that the lot-tolot variability of the oxide layer thickness is greater than the wafer-towafer variability within a lot, which, in turn, is greater than the site-to-site
variation within a wafer.
A multilevel model to describe the oxide thickness yijk measured on the
kth site of the jth wafer within the ith lot is
yijk = + bi + bi,j + ijk , i = 1, . . . , 8, j, k = 1, 2, 3,

bi N 0, 12 , bi,j N 0, 22 , ijk N 0, 2 ,
(4.7)
where the lot random eects bi are assumed to be independent for dierent
i, the wafer within lot random eects bi,j are assumed to be independent
for dierent i and j and to be independent of the bi , and the within-group
errors ijk are assumed to be independent for dierent i, j, and k and to
be independent of the random eects.
The most general form of the argument random when lme is used to t
a multilevel model is as a named list where the names dene the grouping
factors and the formulas describe the random-eects models at each level.
The order of nesting is taken to be the order of the elements in the list,
with the outermost level appearing rst. In the case of (4.7) we write
random = list( Lot = ~ 1, Wafer = ~ 1 )
When the random-eects formulas are the same for all levels of grouping,
we can replace the named list by a one-sided formula with the common
168

+
1
6
>>>
+ +
+
> >
3
+
+
> +> >
> >
>+
>
+ +
>> >+ ++
> + +> >
7
2
>
+ >>+> +
3
Lot
> > +++

>
1980
1990
2000
2010
2020
2030
Thickness of oxide layer
FIGURE 4.14. Thickness of oxide layer measured on dierent sites of wafers

selected from a sample of manufacturing lots. Symbols denote dierent wafers
within the same lot.
random-eects formula and an expression dening the grouping structure

separated by a | operator.
random = ~ 1 | Lot/Wafer
Because Oxide contains this grouping structure in its display formula

> formula( Oxide )
Thickness ~ 1 | Lot/Wafer
the grouping structure expression can be omitted from random. In fact,

because, by default, random is equal to the right-hand side of the fixed
formula, ~1 in this case, we can omit random all together from the lme call
and t the model with
> fm1Oxide <- lme( Thickness ~ 1, Oxide )
> fm1Oxide
Data: Oxide
Fixed: Thickness ~ 1
(Intercept)
2000.2
Random effects:
Formula: ~ 1 | Lot
(Intercept)
StdDev:
11.398
169
Formula: ~ 1 | Wafer %in% Lot

StdDev:
5.9888
3.5453
Number of Groups:
Lot Wafer %in% Lot
8
24
The REML estimates of the variance components are the square of the
12 = 11.3982 = 129.91,
22 =
standard deviations in the fm1Oxide output:
5.98882 = 35.866, and
2 = 3.54532 = 12.569. We can assess the variability
in these estimates with the intervals method.
> intervals( fm1Oxide, which = "var-cov" )
Random Effects:
Level: Lot
lower
est. upper
sd((Intercept)) 5.0277 11.398 25.838
Level: Wafer
lower
est. upper
sd((Intercept)) 3.4615 5.9888 10.361
lower
est. upper
2.6719 3.5453 4.7044
All intervals are bounded well away from zero, indicating that the two
random eects should be kept in (4.7). We can test, for example, if the
wafer within lot random eect can be eliminated from the model with
> fm2Oxide <- update( fm1Oxide, random = ~ 1 | Lot)
> anova( fm1Oxide, fm2Oxide )
Model df
AIC
BIC logLik
fm1Oxide
1 4 462.02 471.07 -227.01
fm2Oxide
2 3 497.13 503.92 -245.57 1 vs 2
37.11 <.0001
The very high value of the likelihood ratio test statistic conrms that the
signicance of that term in the model.
As with single-level ts, estimated BLUPs of the individual coecients
are obtained using coef, but, because of the multiple grouping levels, a
level argument is used to specify the desired grouping level. For example,
to get the estimated average oxide layer thicknesses by lot, we use
> coef( fm1Oxide, level = 1 )
(Intercept)
1
1996.7
2
1988.9
170
3
4
5
6
7
8

2001.0
1995.7
2013.6
2019.6
1992.0
1993.8
while the estimated average thicknesses per wafer are obtained with
> coef( fm1Oxide, level = 2 )
(Intercept)
1/1
2003.2
1/2
1984.7
1/3
2001.1
. . .
8/2
1995.2
8/3
1990.7
# default when level is omitted
The level argument is used similarly with the methods fitted, predict,
ranef, and resid, with the dierence that multiple levels can be simultaneously specied. For example, to get the estimated random eects at both
grouping levels we use
> ranef( fm1Oxide, level = 1:2 )
Level: Lot
(Intercept)
1
-3.46347
2
-11.22164
. . .
8
-6.38538
Level: Wafer %in% Lot
(Intercept)
1/1
6.545993
1/2 -11.958939
. . .
8/3
-3.074863
These methods are further illustrated in 4.3, when we describe tools for
assessing the adequacy of tted models.
Manufacturing of Analog MOS Circuits
The Wafer data, introduced in 3.3.4 and shown in Figure 4.15, provide
another example of multilevel data in IC manufacturing and are used here
to illustrate the capabilities in lme when covariates are used in a multilevel
model. As described in Appendix A.30, these data come from an experiment conducted at the Microelectronics Division of Lucent Technologies to
study dierent sources of variability in the manufacturing of analog MOS
circuits. The intensity of current (in mA) at 0.8, 1.2, 1.6, 2.0, and 2.4

1.0
1.5
2.0
1.0
10
1.5
171
2.0
7
15
10
15
10
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
Voltage (V)
FIGURE 4.15. Current versus voltage curves for each site, by wafer.
V was measured on 80m0.6m n-channel devices. Measurements were

made on eight sites in each of ten wafers selected from the same lot. Two
levels of nesting are present in these data: wafer and site within wafer. The
main objective of the experiment was to construct an empirical model for
simulating the behavior of similar circuits.
From Figure 4.15, it appears that current can be modeled as a quadratic
function of voltage. We initially consider a full mixed-eects model, with all
terms having random eects at both the wafer and the site within wafer
levels. The corresponding multilevel model for the intensities of current
yijk at the kth level of voltage vk in the jth site within the ith wafer is
expressed, for i = 1, . . . , 10, j = 1, . . . , 8, and k = 1, . . . , 5 as
yijk = (0 + b0i + b0i,j ) + (1 + b1i + b1i,j ) vk + (2 + b2i + b2i,j ) vk2 + ijk ,
b0i
b0i,j
bi = b1i N (0, 1 ) , bi,j = b1i,j N (0, 2 ) ,
b2i
b2i,j

2
ijk N 0, .
(4.8)
The parameters 0 , 1 , and 2 are the xed eects in the quadratic model,
bi is the wafer-level random-eects vector, bi,j is the site within waferlevel random-eects vector, and ijk is the within-group error. As usual,
the bi are assumed to be independent for dierent i, the bi,j are assumed
to be independent for dierent i, j and independent of the bi , and the ijk
are assumed to be independent for dierent i, j, k and independent of the
random eects.
172
The large number of parameters in (4.8)twelve variancecovariance

components for the random eectsmakes the optimization of the proled
log-restricted-likelihood quite dicult and unstable. To make the optimization more stable during this initial model-building phase, we simplify (4.8)
by assuming that 1 and 2 are diagonal matrices.
> fm1Wafer <- lme( current ~ voltage + voltage^2, data = Wafer,
+
random = list(Wafer = pdDiag(~voltage + voltage^2),
+
Site = pdDiag(~voltage + voltage^2)))
> summary( fm1Wafer )
Data: Wafer
AIC
BIC logLik
-281.51 -241.67 150.75
Random effects:
Formula: ~ voltage + voltage^2 | Wafer
Structure: Diagonal
(Intercept) voltage I(voltage^2)
StdDev: 0.00047025 0.18717
0.025002
Formula: ~ voltage + voltage^2 | Site %in% Wafer
Structure: Diagonal
(Intercept) voltage I(voltage^2) Residual
StdDev: 0.00038085 0.13579
1.5202e-05 0.11539
Fixed effects: current ~ voltage + voltage^2
(Intercept) -4.4612 0.051282 318 -86.992 <.0001
voltage 5.9034 0.092700 318 63.683 <.0001
I(voltage^2) 1.1704 0.022955 318 50.987 <.0001
Correlation:
(Intr) voltag
voltage -0.735
I(voltage^2) 0.884 -0.698
Min
Q1
Med
Q3
Max
-1.8967 -0.53534 0.024858 0.79853 1.7777
Number of Groups:
Wafer Site %in% Wafer
10
80
Because Wafer is a groupedData object and the random-eects model is

identical for both levels of grouping, we could have used
random = pdDiag( ~ voltage + voltage^2 )
173
in the lme call to produce fm1Wafer. In this case, the object specied in
random is repeated for all levels of grouping.
The very small estimated standard deviations for the (Intercept) random eect at both levels of grouping and for the voltage^2 random eect
at the Site %in% Wafer level suggest that these terms could be eliminated
from (4.8). Before pursuing this any further, we should assess the adequacy of the tted model. This is considered in detail in 4.3 and reveals
that important terms are omitted from the xed-eects model in (4.8). We
therefore postpone this discussion until 4.3 and proceed with the analysis
of fm1Wafer in this section to further illustrate the use of lme methods with
multilevel objects.
As with single-level objects, the fitted method is used to extract the
tted values, with the level argument being used to specify the desired
level(s) of grouping. For example, to get the population level tted values,
we use
> fitted( fm1Wafer, level = 0 )
1
1
1
1
1
1
1
1
1
1.0106 4.3083 7.9805 12.027 16.448 1.0106 4.3083 7.9805 12.027
. . .
10
10
10
10
4.3083 7.9805 12.027 16.448
attr(, "label"):
[1] "Fitted values (mA)"
Similarly, residuals are extracted using the resid method. For example, the
Wafer and Site %in% Wafer residuals are obtained with
> resid( fm1Wafer, level = 1:2 )
Wafer
Site
1 0.0615008 0.0680629
2 -0.1898559 -0.1800129
. . .
399 0.0051645 0.1187074
400 -0.2076543 -0.0714028
The predict method is used to obtain predictions for new observations.

For example, to obtain the predicted currents at 1.0 V, 1.5 V, 3.0 V and
3.5 V for Wafer 1, we rst construct a data frame with the relevant information
> newWafer <+
data.frame( Wafer = rep(1, 4), voltage = c(1, 1.5, 3, 3.5) )
and then use

> predict( fm1Wafer, newWafer, level = 0:1 )
Wafer predict.fixed predict.Wafer
1
1
2.6126
2.4014
174
2
3
4

1
1
1
7.0273
23.7826
30.5381
6.7207
23.2314
29.9192
Note that, because no predictions were desired at the Site %in% Wafer level,
Site did not need to be specied in newWafer. If we are interested in getting
predictions for a specic site, say 3, within Wafer 1, we can use
> newWafer2 <- data.frame( Wafer = rep(1, 4), Site = rep(3, 4),
+
voltage = c(1, 1.5, 3, 3.5) )
> predict( fm1Wafer, newWafer2, level = 0:2 )
Wafer Site predict.fixed predict.Wafer predict.Site
1
1 1/3
2.6126
2.4014
2.4319
2
1 1/3
7.0273
6.7207
6.7666
3
1 1/3
23.7826
23.2314
23.3231
4
1 1/3
30.5381
29.9192
30.0261
These methods will be used extensively in the next section, when we

describe methods for assessing the adequacy of the tted models.
4.3 Examining a Fitted Model

Before making inferences about a tted mixed-eects model, we should
check whether the underlying distributional assumptions appear valid for
the data. There are two basic distributional assumptions for the mixedeects models considered in this chapter:
Assumption 1 - the within-group errors are independent and identically normally distributed, with mean zero and variance 2 , and they
are independent of the random eects.
Assumption 2 - the random eects are normally distributed, with
mean zero and covariance matrix (not depending on the group)
and are independent for dierent groups;
The nlme library provides several methods for assessing the validity of these
assumptions. The most useful of these methods are based on plots of the
residuals, the tted values, and the estimated random eects. The validity
of the distributional assumptions may also be formally assessed using hypothesis tests. However, only rarely do the conclusions of a hypothesis test
about some assumption in the model contradict the information displayed
in a diagnostic plot.
4.3.1 Assessing Assumptions on the Within-Group Error

Because the assumptions above break down into several smaller assumptions, several diagnostic plots are needed to properly assess their validity.
We start by considering Assumption 1 on the within-group error term.
175
Dependencies among the within-group errors are usually modeled with

correlation structures, which will be discussed in detail in 5.3, where methods for assessing the assumption of independence among the within-group
errors will also be described. In this section, we concentrate on methods
for assessing the assumption that the within-group errors are normally distributed, are centered at zero, and have constant variance.
The primary quantities used to assess the adequacy of Assumption 1 are
the within-group residuals, dened as the dierence between the observed
response and the within-group tted value. Conditional on the randomeects variancecovariance components, the within-group residuals are the
BLUPs of the within-group errors. In practice, the within-group residuals are only estimated BLUPs, as the random-eects variancecovariance
components need to be replaced with their estimates. Nevertheless, they
generally provide good surrogates for the within-group errors and can be
used to qualitatively assess the validity of Assumption 1. Other quantities
used for assessing Assumption 1 graphically include the within-group tted
values, the observed values, and any covariates of interest.
The plot method for the lme class is the primary tool for obtaining
diagnostic plots for Assumption 1. It takes several optional arguments, but
a typical call is
plot( object, formula )
where object is an lme object and formula is a formula object describing

the components to be used in the plot. The general expression for formula
is y~x|g where y and x dene, respectively, the vertical and horizontal axes
for the plot and g is an optional factor (or a set of factors separated by *
operators) dening the panels of a Trellis display of y~x. Any variables, or
functions of variables, which can be evaluated in the data frame used to
t object, are allowed for y, x, and g, provided the resulting plot is a valid
Trellis plot. The symbol "." is reserved to represent the tted object itself
in the formula denition. For example, resid(.) can be used in formula to
represent resid(object). We illustrate the capabilities of the plot method
through the analysis of the examples described in earlier sections of this
chapter. Details about the arguments to the lmList and lme plot methods
are given in the help les in Appendix B.
The rst residual plot we consider is the boxplot of residuals by group. For
the fm2Orth.lme object of 4.2.1 we use
> plot( fm2Orth.lme, Subject~resid(.), abline = 0 )
# Figure 4.16
The argument abline = 0 indicates that a vertical line at zero should be

added to the plot. This plot is useful for verifying that the errors are
centered at zero (i.e., E [] = 0), have constant variance across groups
Subject
176
F11
F04
F03
F08
F02
F07
F05
F01
F06
F09
F10
M10
M01
M04
M06
M15
M09
M14
M13
M12
M03
M08
M07
M11
M02
M05
M16
-4
-2
Residuals (mm)
FIGURE 4.16. Boxplots of the residuals for fm2Orth.lme by subject.
(Var (ij ) = 2 ), and are independent of the group levels. Figure 4.16 indicates that the residuals are centered at zero, but that the variability
changes with group. Because there are only four observations per subject,
we cannot rely too much on the individual boxplots for inference about
the within-group variances. We observe an outlying observation for subject
M13 and large residuals for subject M09. A pattern suggested by the individual boxplots is that there is more variability among boys (the lower 16
boxplots) than among girls (the upper 11 boxplots). We can get a better
feeling for this pattern by examining the plot of the standardized residuals
versus tted values by gender, shown in Figure 4.17
> plot( fm2Orth.lme, resid(., type = "p") ~ fitted(.) | Sex,
+
id = 0.05, adj = -0.3 )
# Figure 4.17
The type = "p" argument to the resid method species that the standardized residuals should be used. The id argument species a critical value for
identifying observations in the plot (standardized residuals greater than the
1-id/2 standard normal quantile in absolute value are identied in plot).
By default, the group labels are used to identify the observations. The argument adj controls the position of the identifying labels. It is clear from
Figure 4.17 that the variability in the orthodontic distance measurements
is greater among boys than among girls. Within each gender the variability
seems to be constant. The outlying observations for subjects M09 and M13
are evident.

18
20
Male
24
26
28
30
32
Female
22
177
M09
-2
M09
M13
18
20
22
24
26
28
30
32
Fitted values (mm)
FIGURE 4.17. Scatter plots of standardized residuals versus tted values for
fm2Orth.lme by gender.
A more general model to represent the orthodontic growth data allows

dierent variances by gender for the within-group error. The lme function
allow the modeling of heteroscesdasticity of the within-error group via a
weights argument. This topic will be covered in detail in 5.2, but, for now,
it suces to know that the varIdent variance function structure allows
dierent variances for each level of a factor and can be used to t the
heteroscedastic model for the orthodontic growth data as
> fm3Orth.lme <+
update( fm2Orth.lme, weights = varIdent(form = ~ 1 | Sex) )
> fm3Orth.lme
Data: Orthodont
Fixed: distance ~ Sex + I(age - 11) + Sex:I(age - 11)
(Intercept)
Sex I(age - 11) Sex:I(age - 11)
23.808 -1.1605
0.63196
-0.15241
Random effects:
StdDev
Corr
I(age - 11) 0.15652 0.394
Residual 1.62959
Variance function:
Structure: Different standard deviations per stratum
Formula: ~ 1 | Sex
Parameter estimates:
Male Female
1 0.40885
178

18
20
Male
24
26
28
30
Female
M09
22
2
1
0
-1
-2
M09
M13
-3
18
20
22
24
26
28
30
Fitted values (mm)
FIGURE 4.18. Scatter plots of standardized residuals versus tted values for the
heteroscedastic t of fm3Orth.lme by gender.

The parameters for varIdent give the ratio of the stratum standard errors
to the within-group standard error. To allow identiability of the parameters, the within-group standard error is equal to the rst stratum standard
error. For the orthodontic data, the standard error for the girls is about
41% of that for the boys. The remaining estimates are very similar to the
ones in the homoscedastic t fm2Orth.lme. We can assess the adequacy of
the heteroscedastic t by re-examining plots of the standardized residuals
versus the tted values by gender, shown in Figure 4.18. The standardized
residuals in each gender now have about the same variability. We can still
identify the outlying observations, corresponding to subjects M09 and M13.
Overall, the standardized residuals are small, suggesting that the linear
mixed-eects model was successful in explaining the orthodontic growth
curves This is better seen by looking at a plot of the observed responses
versus the within-group tted values.
> plot( fm3Orth.lme, distance ~ fitted(.),
+
id = 0.05, adj = -0.3 )
# Figure 4.19
The fm3Orth.lme tted values are in close agreement with the observed
orthodontic distances, except for the three extreme observations on subjects
M09 and M13.
The need for an heteroscedastic model for the orthodontic growth data
can be formally tested with the anova method.
> anova( fm2Orth.lme, fm3Orth.lme )
Model df
AIC
BIC logLik
fm2Orth.lme
1 8 451.35 472.51 -217.68
fm3Orth.lme
2 9 432.30 456.09 -207.15 1 vs 2 21.059 <.0001
179
M09
30
25
M09
20
M13
18
20
22
24
26
28
30
Fitted values (mm)
FIGURE 4.19. Observed versus tted values plot for fm3Orth.lme.
The very small p-value of the likelihood ratio statistic conrms that the
heteroscedastic model explains the data signicantly better than the homoscedastic model.
The assumption of normality for the within-group errors can be assessed
with the normal probability plot of the residuals, produced by the qqnorm
method. A typical call to qqnorm is of the form
qqnorm( object, formula )
where object is an lme object and formula is a one-sided formula object of

the form ~x|g. As in the plot method, the symbol "." represents the tted
lme object. The x term in formula can be either the residuals (resid(.)),
or the random eects (ranef(.)) associated with the t. In this section, we
consider only the case where x denes a vector of residuals. The randomeects case is considered in 4.3.2. The g term in formula denes an optional
factor (or set of factors joined by *) determining the panels for a Trellis
display.
For example, to obtain the normal plots of the residuals corresponding
to fm3Orth.lme by gender, we use
> qqnorm( fm3Orth.lme, ~resid(.) | Sex )
# Figure 4.20
Once again, we observe the three outlying points, but for the rest of the
observations the normality assumption seems plausible.
We initially consider the plot of the standardized residuals versus tted
values by Lot for the fm2IGF.lme object, obtained with
180

-4
-2
Quantiles of standard normal
Male
Female
-1
-2
-4
-2
Residuals (mm)
FIGURE 4.20. Normal plot of residuals for the fm3Orth.lme lme t.
> plot( fm2IGF.lme, resid(., type = "p") ~ fitted(.) | Lot,

+
layout = c(5,2) )
# Figure 4.21
The residuals are centered around zero and seem to have similar variability
across lots. There are some outliers in the data, most noticeably for Lots 3
and 7.
We assess the normality of the within-group errors with a qqnorm plot of
the residuals.
> qqnorm( fm2IGF.lme, ~ resid(.),
+
id = 0.05, adj = -0.75 )
# Figure 4.22
The normal plot in Figure 4.22 suggests that the distribution of the withingroup errors has heavier tails than expected under normality, but is also
symmetric around zero. Perhaps a mixture of normal distributions or a
t-distribution with a moderate number of degrees of freedom would model
the distribution of the within-group error more adequately. However, as the
heavier tails seem to be distributed symmetrically, the estimates of the xed
eects should not change substantially under either a mixture model or a
t-model. The heavier tails tend to inate the estimate of the within-group
standard error under the Gaussian model, leading to more conservative
tests for the xed eects, but, because the p-value for the hypothesis that
the decay of tracer activity with age is zero is quite high (0.673), the main
conclusion should remain unchanged under either a mixture or a t-model.
The plot of the within-group standardized residuals (level = 2 in this case)
versus the within-group tted values is the default display produced by the
plot method. Therefore,
5.1
5.2
5.3
5.4
5.5
5.1
5.2
5.3
5.4
181
5.5
4
2
0
-2
-4
9
10
4
2
0
-2
-4
5.1
5.2
5.3
5.4
5.5
5.1
5.2
5.3
5.4
5.5
5.1
5.2
5.3
5.4
5.5
Fitted values (ng/ml)
FIGURE 4.21. Scatter plots of standardized residuals versus tted values for the
fm2IGF.lme t, by lot.
2
7
7
3
4
-1
-2
-3
2
1
9
3
-4
-2
Residuals (ng/ml)
FIGURE 4.22. Normal plot of residuals for the fm2IGF.lme t.
182
-1
-2
1990
2000
2010
2020
2030
Fitted values
FIGURE 4.23. Scatter plot of the standardized within-group residuals versus the
within-group tted values for the fm1Oxide t.
> plot( fm1Oxide )
# Figure 4.23
results in the plot shown in Figure 4.23, which does not indicate any departures from the within-group errors assumptions: the residuals are symmetrically distributed around zero, with approximately constant variance.
By default, the qqnorm method produces a normal plot of the withingroup standardized residuals. Hence,
> qqnorm( fm1Oxide )
# Figure 4.24
gives the normal plot in Figure 4.24, which indicates that the assumption
of normality for the within-group errors is plausible.
The plot of the within-group residuals versus voltage by wafer, shown in
Figure 4.25, shows a clear periodic pattern for the residuals.
> plot( fm1Wafer, resid(.) ~ voltage | Wafer )
Figure 4.25
We can enhance the visualization of this pattern by adding a loess smoother

(Cleveland, Grosse and Shyu, 1992) to each panel, using
> plot( fm1Wafer, resid(.) ~ voltage | Wafer,
+
panel = function(x, y, ...) {
+
panel.grid()
+
panel.xyplot(x, y)
+
panel.loess(x, y, lty = 2)
+
panel.abline(0, 0)
+
} )
# Figure 4.26
The panel argument to the plot method overwrites the default panel function, allowing customized displays.
183
-1
-2
-2
-1
FIGURE 4.24. Normal plot of within-group standardized residuals for the

fm1Oxide t.
1.0
6
1.5
2.0
1.0
1.5
2.0
10
0.2
0.1
Residuals (mA)
0.0
-0.1
-0.2
1
0.2
0.1
0.0
-0.1
-0.2
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
Voltage (V)
FIGURE 4.25. Scatter plots of within-group residuals versus voltage by wafer for
the fm1Wafer t.
184

1.0
6
1.5
2.0
1.0
1.5
2.0
10
0.2
0.1
Residuals (mA)
0.0
-0.1
-0.2
1
0.2
0.1
0.0
-0.1
-0.2
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
Voltage (V)
FIGURE 4.26. Scatter plots of within-group residuals versus voltage by wafer for
the fm1Wafer t. A loess smoother has been added to each panel to enhance the
visualization of the residual pattern.
The same periodic pattern appears in all panels of Figure 4.26, with a
period T of approximately 1.5 V. Noting that the residuals are centered
around zero, this periodic pattern can be represented by the cosine wave
3 cos (v) + 4 sin (v) ,
(4.9)

where 3 and 4 determine the amplitude (= 32 + 42 ) and is the frequency of the cosine wave. We can incorporate this pattern into model (4.8)
by rewriting the xed-eects model for the expected value of yijk as
E [yijk ] = 0 + 1 vk + 2 vk2 + 3 cos (vk ) + 4 sin (vk ) .
(4.10)
Because is unknown, (4.10) formally gives an example of a nonlinear

mixed-eects model, discussed later in Chapter 8. To proceed with the
model building analysis, we assume, for now, that is a purely xed eect
(i.e., does not vary with wafer and/or site) and estimate it by tting (4.9) to
the within-group residuals of fm1Wafer, using the nls function in S (Bates
and Chambers, 1992). By replacing in (4.10) with its nonlinear leastsquares estimate
, we are left with a linear mixed-eects model, which
can then be tted with lme.
Initial values for 3 , 4 , and are needed in the iterative algorithm
used in nls. The frequency and the period T are related by the formula
= 2/T , giving the initial estimate 0 = 2/1.5 = 4.19 V 1 for . For a
xed = 0 , (4.9) is a linear function of 3 and 4 and we can use lm to
obtain initial estimates for these parameters.
185
> attach( Wafer )

# making variables in Wafer available
> coef(lm(resid(fm1Wafer) ~ cos(4.19*voltage)+sin(4.19*voltage)-1))
cos(4.19 * voltage) sin(4.19 * voltage)
-0.051872
0.1304
> nls( resid(fm1Wafer) ~ b3*cos(w*voltage) + b4*sin(w*voltage),
+
start = list(b3 = -0.0519, b4 = 0.1304, w = 4.19) )
Residual sum of squares : 0.72932
parameters:
b3
b4
w
-0.11173 0.077682 4.5679
formula: resid(fm1Wafer) ~ b3*cos(w*voltage) + b4*sin(w*voltage)
400 observations
> detach( )
The resulting estimate for the frequency is w

= 4.5679 V 1 .
We ret model (4.8), with the xed-eects model replaced by (4.10) and
held constant at
, with
> fm2Wafer <- update( fm1Wafer,
+
fixed = . ~ . + cos(4.5679*voltage) + sin(4.5679*voltage),
+
random = list(Wafer=pdDiag(~voltage+voltage^2),
+
Site=pdDiag(~voltage+voltage^2)) )
> summary( fm2Wafer )
Data: Wafer
AIC
BIC logLik
-1232.6 -1184.9 628.31
Random effects:
Structure: Diagonal
(Intercept) voltage I(voltage^2)
StdDev:
0.12888 0.34865
0.049074
Structure: Diagonal
StdDev:
0.039675 0.23437
0.047541 0.011325
Fixed effects: current ~ voltage + I(voltage^2) +
cos(4.5679 * voltage) + sin(4.5679 * voltage)
(Intercept) -4.2554
0.04223 316 -100.76 <.0001
voltage 5.6224
0.11416 316
49.25 <.0001
I(voltage^2) 1.2585
0.01696 316
74.21 <.0001
cos(4.5679 * voltage) -0.0956
0.00112 316 -85.05 <.0001
sin(4.5679 * voltage) 0.1043
0.00150 316
69.42 <.0001
. . .
186
The .~. in the fixed formula is an abbreviated form for the xed-eects formula in the original lme object, fm1Wafer. This convention is also available
in other S modeling functions, such as lm and aov. The random argument
was included in the call to update to prevent the estimated random-eects
parameters from fm1Wafer to be used as initial estimates (these give bad
initial estimates in this case and may lead to convergence problems).
The very high t-values for the sine and cosine terms in the summary output
indicate a signicant increase in the quality of the t when these terms are
included in the xed-eects model. The estimated standard deviations for
the random eects are quite dierent from the ones in fm1Wafer and now
suggest that there is signicant wafer-to-wafer and site-to-site variation
in all random eects in the model. The estimated within-group standard
deviation for fm2Wafer is about ten times smaller than that of fm1Wafer,
giving further evidence of the greater adequacy of (4.10). We assess the
variability in the estimates with
> intervals( fm2Wafer )
Fixed effects:
lower
est.
upper
(Intercept) -4.338485 -4.255388 -4.172292
voltage 5.397744 5.622357 5.846969
I(voltage^2) 1.225147 1.258512 1.291878
cos(4.5679 * voltage) -0.097768 -0.095557 -0.093347
sin(4.5679 * voltage) 0.101388 0.104345 0.107303
Random Effects:
Level: Wafer
lower
est.
upper
sd((Intercept)) 0.065853 0.128884 0.25225
sd(voltage) 0.174282 0.348651 0.69747
sd(I(voltage^2)) 0.023345 0.049074 0.10316
Level: Site
lower
est.
upper
sd((Intercept)) 0.017178 0.039675 0.091635
sd(voltage) 0.175311 0.234373 0.313332
sd(I(voltage^2)) 0.035007 0.047541 0.064564
lower
est.
upper
0.0085375 0.011325 0.015023
The plots of the within-group residuals versus voltage, by wafer, shown

in Figure 4.27, indicate that there is still some periodicity left in some of the
wafers, suggesting that random eects may be needed to accommodate the
variation in . We postpone this discussion until Chapter 8, when nonlinear
mixed-eects models are described. Note that the absolute values of the

1.0
6
1.5
2.0
1.0
1.5
187
2.0
10
0.03
0.02
Residuals (mA)
0.01
0.0
-0.01
-0.02
1
0.03
0.02
0.01
0.0
-0.01
-0.02
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
Voltage (V)
FIGURE 4.27. Scatter plots of within-group residuals versus within-group tted

values by wafer for the fm2Wafer t. A loess smoother has been added to each
panel to enhance the visualization of the residual pattern.
within-groups residuals in Figure 4.27 are about an order of magnitude

smaller than the ones in Figure 4.26.
The normal plot of the within-group residuals for fm2Wafer, obtained
with
> qqnorm( fm2Wafer )
# Figure 4.28
and shown in Figure 4.28, does not indicate any violations from the
assumption of normality for the within-group errors.
4.3.2 Assessing Assumptions on the Random Eects

In this section, we describe diagnostic methods for assessing Assumption
2, on the distribution of the random eects. The ranef method is used
to extract the estimated BLUPs of the random eects from lme objects.
These are the primary quantities for assessing the distributional assumptions about the random eects.
Two types of diagnostic plots will be used to investigate departures from
Assumption 2:
qqnormnormal plot of estimated random eects for checking marginal
normality and identifying outliers;
pairsscatter plot matrix of the estimated random eects for identifying outliers and checking the assumption of homogeneity of the
random eects covariance matrix;
188
-1
-2
-3
-2
-1
FIGURE 4.28. Normal probability plot of the within-group standardized residuals

for the fm2Wafer t.
We illustrate the use of these display methods by continuing the analysis

of the examples in 4.3.1.
We rst consider the homoscedastic tted object fm2Orth.lme and investigate the marginal normality of the corresponding random eects using the
normal plots in Figure 4.29.
> qqnorm( fm2Orth.lme, ~ranef(.),
+
id = 0.10, cex = 0.7 )
# Figure 4.29
As in the residual plots of 4.3.1, the id argument species a critical value

for identifying points in the plot (standardized random-eects estimates
greater than the 1-id/2 standard normal quantile in absolute value are
identied). By default, the group labels are used to identify the points.
The assumption of normality seems reasonable for both random eects,
though there is some asymmetry in the distribution of the (Intercept)
random eects. A few outliers appear to be present in both random eects:
F10, F11, and M10 for the intercept and M13 for the slope.
To investigate the homogeneity of the random-eects distribution for
boys and girls, we use the pairs method to obtain scatter plots of the
random-eects estimates by gender, as shown in Figure 4.30.
> pairs( fm2Orth.lme, ~ranef(.) | Sex,
+
id = ~ Subject == "M13", adj = -0.3 )
# Figure 4.30
(Intercept)
189
I(age - 11)
M10
M13
F11
-1
-2
F10
-2
-0.1
0.0
0.1
0.2
0.3
Random effects
FIGURE 4.29. Normal plot of estimated random eects for the homoscedastic
fm2Orth.lme t.
-2
Male
Female
M13
0.3
I(age - 11)
0.2
0.1
0.0
-0.1
-2
(Intercept)
FIGURE 4.30. Scatter plot of estimated random eects for the homoscedastic
fm2Orth.lme t.
190

(Intercept)
I(age - 11)
M10
F11
-1
-2
F10
-4
-2
-0.1
0.0
0.1
0.2
Random effects
FIGURE 4.31. Normal plot of estimated random eects for the heteroscedastic
fm3Orth.lme lme t.
The id argument is given as a formula object, which is used to identify

subject M13 in the plot. Alternatively, it can be given as a numeric value
such that points outside the normal distribution contour of level 1-id/2
are identied in the plot. Except for the pair corresponding to subject
M13, the estimated random eects in the two groups seem to have similar
distributions.
As seen in 4.3.1, the within-group variance appears to be larger among
boys than among girls for the Orthodont data and the heteroscedastic t
fm3Orth.lme gives a better representation of the data. We consider now the
diagnostic plots for the random eects in this model. The normal probability plots of the estimated random eects (Figure 4.31) are similar to
those in Figure 4.29 with one important exception: the estimated random
eect for the slope of subject M13 is no longer identied in the plot as an
outlier. In mixed-eects estimation, there is a trade-o between the withingroup variability and the between-group variability, when accounting for
the overall variability in the data. The use of a common within-group variance in fm2Orth.lme leads to an increase in the estimated between-group
variability, which in turn allows the random-eects estimates to be pulled
away by outliers. The heteroscedastic model in fm3Orth.lme accommodates
the impact of the boys outlying observations in the within-group variances
estimation and reduces the estimated between-group variability, thus increasing the degree of shrinkage in the random-eects estimates. In this
case, the use of dierent within-group variances by gender adds a certain
degree of robustness to the lme t.

(Intercept)
191
age
1.5
1.0
0.5
0.0
-0.5
-1.0
-1.5
10^-7
2*10^-7
-0.004
0.0
0.002
0.004
0.0
Random effects
FIGURE 4.32. Normal plot of estimated random eects for the homoscedastic
fm2IGF.lme t.
The pairs plots by gender for Orthodont.lme3, not included here, do not
suggest any departures from the assumption of homogeneity of the randomeects distribution.
The normal plots of the estimated random eects for the fm2IGF.lme t,
shown in Figure 4.32, do not indicate any departures from normality or any
outlying subjects. They do, however, suggest that there is very little variability in the (Intercept) random eect, as the estimated random eects
are on the order of 107 .
The relative variability of each random eect with respect to the corresponding xed-eect estimates can be calculated as
> fm2IGF.lme
. . .
Fixed: conc ~ age
(Intercept)
age
5.369 -0.0019301
Random effects:
Structure: Diagonal
(Intercept)
age Residual
StdDev: 0.00031074 0.0053722
0.8218
. . .
> c( 0.00031074, 0.0053722 )/abs( fixef(fm2IGF.lme) )
(Intercept)
age
5.7876e-05 2.7833
192

(Intercept)
1.5
1.0
0.5
0.0
-0.5
-1.0
-1.5
-10
-5
10
15
20
Random effects
FIGURE 4.33. Normal plot of estimated Lot random eects for the fm1Oxide t.
The relative variability of the (Intercept) random eect is only 0.006%,

while the relative variability of the age random eect is about 278.3%,
suggesting that the former could be dropped from the model.
> fm3IGF.lme <- update( fm2IGF.lme, random = ~ age - 1 )
> anova( fm2IGF.lme, fm3IGF.lme )
Model df
AIC
BIC logLik
Test
L.Ratio p-value
fm2IGF.lme
1 5 604.8 622.10 -297.4
fm3IGF.lme
2 4 602.8 616.64 -297.4 1 vs 2 1.0881e-05 0.9974
The high p-value for the likelihood ratio test indicates that the random
eect for the intercept does not contribute signicantly to the t of the IGF
data.
The normal plot of the estimated random eects for the reduced model
fm3IGF.lme, not included here, does not suggest any violations of the normality assumption and does not show any outlying values.
Normal probability plots of the estimated random eects must be examined
at each level of grouping when assessing the adequacy of a multilevel model
t. The normal plot of the estimated Lot random eects for the fm1Oxide
t, shown in Figure 4.33, is obtained with
> qqnorm( fm1Oxide, ~ranef(., level = 1), id=0.10 )
# Figure 4.33
The level argument to the ranef method is required in this case, as, by
default, ranef returns a list with the estimated random eects at each level
193
(Intercept)
6/1
-1
-2
1/2
-10
-5
10
Random effects
FIGURE 4.34. Normal plot of estimated Wafer %in% Lot random eects for the
fm1Oxide t.
of grouping, which will cause an error in qqnorm. Because there are only
eight random eects at the Lot level, it is dicult to identify any patterns
in Figure 4.33. Lot 6 is indicated as a potential outlier, which is consistent
with the plot of the data in Figure 4.14, where this lot is shown to have
the thickest oxide layers.
The normal plot of the Wafer %in% Lot random eects, shown in Figure 4.34, and obtained with
> qqnorm( fm1Oxide, ~ranef(., level = 2), id=0.10 )
# Figure 4.34
does not indicate any departures from normality. There is some mild evidence that Wafers 1/2 and 6/1 may be outliers.
The pairs plot of the estimated Wafer random eects for the fm2Wafer t,
shown in Figure 4.35, suggests that the random eects at that level are
correlated.
The fm2Wafer t uses a diagonal structure for the variancecovariance
matrix of the Wafer random eects, which is equivalent to assuming that
these random eects are independent. We test the independence assumption by tting the model with a general positive-denite structure for the
variancecovariance matrix of the Wafer random eects and comparing it
to the fm2Wafer t using the anova method.
194

0.02
0.06
0.04
0.06
0.04
0.02
I(voltage^2)
0.00
-0.02
-0.04
-0.04
-0.02
0.00
0.4
0.0
0.2
0.4
0.2
voltage
0.0
0.0
-0.2
-0.4
0.00
0.10
0.05
-0.2
0.0
-0.4
0.10
0.05
0.00
(Intercept)
-0.05
-0.10
-0.15
-0.10
-0.05
-0.15
FIGURE 4.35. Scatter plot of estimated Wafer random eects for the fm2Wafer
t.
+
+
random = list(Wafer = ~voltage+voltage^2,

Site = pdDiag(~voltage+voltage^2)))
> fm3Wafer
. . .
Random effects:
StdDev
Corr
(Intercept) 0.131622 (Intr) voltag
voltage 0.359244 -0.967
I(voltage^2) 0.051323 0.822 -0.940
Structure: Diagonal
StdDev:
0.033511 0.21831
0.045125 0.011832
. . .
> anova( fm2Wafer, fm3Wafer )
Model df
AIC
BIC logLik
fm2Wafer
1 12 -1232.6 -1184.9 628.31
fm3Wafer
2 15 -1267.0 -1207.3 648.50 1 vs 2 40.378 <.0001
There is a very signicant increase in the log-restricted-likelihood, as evidenced by the large value for the likelihood ratio test, indicating that the
more general model represented by fm3Wafer gives a better t.

0.05
0.00
195
0.05
0.00
I(voltage^2)
-0.05
-0.10
-0.10
0.2
0.6
0.4
-0.05
0.6
0.4
0.2
voltage
0.2
0.0
-0.2
-0.2
0.00
0.02
0.0
0.2
0.04
0.04
0.02
0.00
(Intercept)
0.00
-0.02
-0.04
-0.02
0.00
-0.04
FIGURE 4.36. Scatter plot of estimated random eects at the Site %in% Wafer
level for the fm3Wafer t.
The pairs plot for the estimated Site %in% Wafer random eects corresponding to fm3Wafer in Figure 4.36 indicate that there is a strong negative correlation between the voltage and voltage^2 random eects, but
no substantial correlation between either of these random eects and the
(Intercept) random eects. A block-diagonal matrix can be used to represent such covariance structure, with the (Intercept) random eect corresponding to one block and the voltage and voltage^2 random eects
corresponding to another block.
+
random = list(Wafer = ~ voltage + voltage^2,
+
Site = pdBlocked(list(~1, ~voltage+voltage^2 - 1))))
> fm4Wafer
. . .
Random effects:
StdDev
Corr
(Intercept) 0.131807 (Intr) voltag
voltage 0.354746 -0.967
I(voltage^2) 0.049957 0.814 -0.935
Formula: ~ 1 | Site %in% Wafer
196
StdDev:
(Intercept)
0.066562
Block 2: voltage, I(voltage^2)

Formula: ~ voltage + voltage^2 - 1 | Site %in% Wafer
StdDev
Corr
voltage 0.2674061 voltag
I(voltage^2) 0.0556441 -0.973
Residual 0.0091086
. . .
> anova( fm3Wafer, fm4Wafer )
Model df
AIC
BIC logLik
fm3Wafer
1 15 -1267.0 -1207.3 648.50
fm4Wafer
2 16 -1461.6 -1398.0 746.82 1 vs 2 196.65 <.0001
The small p-value for the likelihood ratio test indicates that the fmWafer4
model ts the data signicantly better than the model represented by
fm3Wafer. This will be the nal model consider in this chapter for the Wafer
data. Later in 8 we revisit this example, tting a nonlinear mixed-eects
model to the data.
The normal plot of the estimated Site %in% Wafer random eects corresponding to fm4Wafer, shown in Figure 4.37, does not suggest any signicant
departure from the assumption of normality for these random eects. There
is some moderate evidence that Sites 1/6, 7/3 and 8/1 may be outliers.
> qqnorm( fm4Wafer, ~ranef(., level = 2), id = 0.05,
+
cex = 0.7, layout = c(3, 1) )
# Figure 4.37
4.4 Chapter Summary

This chapter describes the capabilities available in the nlme library for
tting and analyzing linear mixed-eects models with uncorrelated, homoscedastic within-group errors. The lme function, for tting linear mixedeects models, is described in detail and its various capabilities and associated methods are illustrated through the analyses of several real data
examples, covering single-level models, multilevel nested models, and models with crossed random eects.
The model-building approach developed in this chapter follows an insideout strategy, using individual lm ts, obtained with the lmList function,
to construct more sophisticated linear mixed-eects models. A rich, integrated suite of diagnostic plots to assess model assumptions is described
and illustrated through examples.
The class of mixed-eects models which can be t with lme is greatly extended by the availability of patterned random-eects variancecovariance
Exercises
(Intercept)
voltage
I(voltage^2)
1/6
197
7/3
8/1
-1
-2
8/1
-0.10
7/3
0.0
0.10
-0.4
0.0
0.2
0.4
0.6
0.8 -0.15
-0.05
0.0
0.05
Random effects
FIGURE 4.37. Normal plot of estimated Site %in% Wafer random eects for the
fm4Wafer t.
structures. These are implemented in S through pdMat classes, which can

be extended with user dened classes.
The linear mixed-eects model considered in this chapter is extended in
two dierent ways later in the book. In Chapter 5, the assumption of uncorrelated, homoscedastic within-group errors is relaxed, and variance functions and correlation structures are introduced the model heteroscedasticity
and within-group dependence. The assumption of linearity for E [y i |bi ] is
relaxed in Chapter 8, when nonlinear mixed-eects models are described.
Exercises
1. In 1.3.3 (p. 27) we t the model
> fm3Machine <- update(fm1Machine, random = ~Machine-1|Worker)
The estimated correlations of the components of the random-eects

vectors bi in fm3Machine, 0.803, 0.623, and 0.771, are similar in magnitude. The estimated standard deviations of these components, 4.08,
8.63, and 4.39 are comparable to each other although not as close
as the estimated correlations. Together, these suggest a compound
symmetry structure for as described in 4.2.2.
(a) Fit a model like that of fm3Machine, but with a compound symmetry structure. This is similar to the model fm4OatsB in 4.2.2.
Examine numerical summaries and residual plots for this model.
198
(b) Compare this tted model to fm3Machine with anova. Is the

greater number of parameters in fm3Machine producing a signicantly better t?
(c) Compare the pdCompSymm model to fm2Machine from 1.3.3 with
anova. Note that the likelihoods and the numbers of parameters
are identical.
(d) The pdCompSymm model is equivalent to fm2Machine. Use VarCorr
to extract the estimates of the variance components from each
model. What is the relationship between the estimated variance
components of the two models?
2. There are some unusual residuals for subjects M09 and M13 in several of
the ts of the Orthodont data. See, for example, Figure 4.17 (p. 177).
An examination of the original data, say in Figure 4.9 (p. 153), shows
some suspicious observations on these subjects. The observation at
age 8 for subject M13 is unusually low. The four observations for subject M09 decrease, then increase dramatically, then decrease again.
The two observations at ages 10 and 12 appear to be incorrect.
(a) Repeat the stages of tting a linear mixed-eects models to the
Orthodont data with the suspicious observations for subjects M13
and M09 removed. Do you arrive at dierent conclusions regarding the models?
(b) There are other subjects like M09 in the Orthodont data for whom
the observed distance decreases with increasing age. Because
it is unlikely that this measurement on the same child would
get smaller over time, these are probably misrecorded data or
unusually large measurement errors.
i. Find the number of pairs of measurements that represent decreases within a subject. You can begin with gapply(Orthodont, "distance", diff) which returns a list of successive
dierences of the distance by Subject.
ii. What should be done with these aberrant data points?
iii. If you handle the suspect data by deleting some observations, repeat as much of this chapters analysis of Orthodont
as is feasible with the modied data. (Note that if you decide to delete some of the suspect data, the resulting data
will be unbalanced. Is it still possible to t mixed-eects
models to such unbalanced data?)
3. When analyzing the Assay data (Figure 4.13, p. 164), we noted that
the interaction term in the xed eects for the tted model fm1Assay
is not signicant.
Exercises
199
(a) Ret the model as, say, fm2Assay with the interaction term removed from the xed eects. Are the estimates of the other
parameters changed noticeably? Can you compare this tted
model to fm1Assay with anova and assess the signicance of the
interaction term?
(b) Use VarCorr to extract the estimates of the variance components
from fm2Assay. Notice that the estimated variance component for
the columns on the plate, indexed by dilut within Block, is very
small. Ret the model as, say, fm3Assay with this variance component eliminated. You can express the random argument for this
reduced model in at least three ways: (i) using pdBlocked and
pdIdent as in fm1Assay, (ii) using pdCompSymm as in fm4OatsB, or
(iii) using nested random eects for Block and for sample within
Block. Experiment with these dierent representations. Demonstrate that that the tted models for these three representations
are indeed equivalent. For which of the three representations
does lme converge most easily? Compare one of these tted models to fm2Assay using anova. Is the variance component for the
columns signicant?
(c) Extract the estimates of the variance components from the tted model fm3Assay and examine the condence intervals on
the standard deviations of the random eects. Do the variance
components appear to be signicantly greater than zero? Fit a
model, say fm4Assay, with a single variance component for the
Block. Compare it to fm3Assay with anova.
(d) Can the random eects be eliminated entirely? Fit a model,
say fm5Assay, using just the xed-eects terms from fm4Assay.
Because this model does not have any random-eects terms, you
will need to use lm or gls to t it.
(e) Compare models fm2Assay, fm3Assay, fm4Assay, and fm5Assay
with anova. Which model provides the simplest adequate representation for these data?
(f) Notice that the dilut factor represents serial dilutions that will
be equally spaced on the logarithm scale. Does converting dilut
to an ordered factor, so the contrasts used for this factor will
be polynomial contrasts, suggest further simplications of the
model?
5
Extending the Basic Linear
Mixed-Eects Model
The linear mixed-eects model formulation used in Chapters 2 and 4 allows

considerable exibility in the specication of the random-eects structure,
but restricts the within-group errors to be independent, identically distributed random variables with mean zero and constant variance. As illustrated in Chapters 2 and 4, this basic linear mixed-eects model provides
an adequate model for many dierent types of grouped data observed in
practice. However, there are many applications involving grouped data for
which the within-group errors are heteroscedastic (i.e., have unequal variances) or are correlated or are both heteroscedastic and correlated.
In this chapter, we extend the basic linear mixed-eects model to allow
heteroscedastic, correlated within-group errors. We describe how the lme
function can be used to t the extended linear mixed-eects model, illustrating its various capabilities through examples. We also introduce a new
modeling function, glsfor generalized least squares, to t models with
heteroscedastic and correlated within-group errors, but with no random
eects.
In 5.1, we show how the estimation and computational methods of
Chapter 2 can be adapted to the extended model formulation and describe
how the variancecovariance structure of the within-group errors can be
decomposed into two independent components: a variance structure and a
correlation structure. Variance function models to represent the variance
structure component of the within-group errors are described in 5.2 and
their use with lme is illustrated through examples. Classes of correlation
models to represent the correlation structure of the within-group errors are
202
5. Extending the Basic Linear Mixed-Eects Model
presented and have their use illustrated in 5.3. In 5.4, the gls function is
described and illustrated through examples.
5.1 General Formulation of the Extended Model

As described in 2.1.1, the basic single-level linear mixed-eects
model (2.1)
assumes that the within-group errors i are independent N 0, 2 I random vectors. The extended single-level linear mixed-eects model relaxes
this assumption by allowing heteroscedastic and correlated within-group
errors, being expressed as
y i = X i + Z i bi + i , i = 1, . . . , M,

bi N (0, ), i N 0, 2 i , i = 1, . . . , M,
(5.1)
where the i are positive-denite matrices parametrized by a xed, generally small, set of parameters . As in the basic linear mixed-eects model of
2.1, the within-group errors i are assumed to be independent for dierent
i and independent of the random eects bi . The 2 is factored out of the
i for computational reasons (it can then be eliminated from the proled
likelihood function).
Similarly, the extended two-level linear mixed-eects model generalizes
the basic two-level model (2.2) described in 2.1.2 by letting

ij N 0, 2 ij , i = 1, . . . , M, j = 1, . . . , Mi ,
where the ij are positive-denite matrices parametrized by a xed vector. This readily generalizes to a multilevel model with Q levels of random
eects. For simplicity, we concentrate for the remainder of this section on
the extended single-level model (5.1), but the results we obtain are easily
generalizable to multilevel models with an arbitrary number of levels of
random eects.

1/2
Because i is positive-denite, it admits an invertible square-root i

1/2
(Thisted, 1988, 3), with inverse i
, such that
T

T

1/2
1/2
1/2
1/2
i = i
i
and 1
=
.
i
i
i
Letting

T
1/2
y i = i
yi ,
T

1/2
X i,
X i = i

T
1/2
i = i
i ,
T

1/2
Z i = i
Z i,
(5.2)
203
and noting that

T

T

1/2
1/2
1/2
2
i N i
0, i
i i
= N 0, 2 I ,
we can rewrite (5.1) as
y i = X i + Z i bi + i , i = 1, . . . , M,

bi N (0, ), i N 0, 2 I , i = 1, . . . , M.
(5.3)
That is, y i is described by a basic linear mixed-eects model.

1/2 T
Because the dierential
of the linear transformation y i = (i
) y i is

1/2
simply dy i = i
dy i , the likelihood function L corresponding to the
extended linear mixed-eects model (5.1) is expressed as
M

p y i |, , 2 ,
L , , 2 , |y =
i=1
M
M

1/2
1/2
2
2
=
p y i |, , , i
= L , , , |y
i
,
i=1
(5.4)
i=1
where p() denotes a probability

density function.

The likelihood function L , , 2 , |y corresponds to a basic linear
mixed-eects model and, therefore, all results presented in 2.2 also apply to it. The compact representation of the proled log-likelihood based
on orthogonal-triangular decompositions,
described in 2.2.3, can also be

used with the likelihood L , , 2 , |y , leading to numerically ecient
algorithms for maximum likelihood estimation.
The restricted likelihood corresponding to the extended model (5.1) is
dened, as in 2.2.5, by integrating out the xed eects from the likelihood.

LR , 2 , |y =
M

1/2
L , , 2 , |y d = LR , 2 , |y
i
.
i=1

The function LR , , 2 , |y corresponds to a restricted likelihood function of a basic linear mixed-eects model. Hence, the results in 2.2.5 can
be used to obtain a numerically ecient representation of the proled logrestricted-likelihood.
5.1.2 An Extended Linear Model with No Random Eects

The variancecovariance matrix of the response vector y i in the extended
linear mixed-eects model (5.1),

Var (y i ) = i = 2 Z i DZ Ti + i
204
has two components that can be used to model heteroscedasticity and correlation: a random-eects component, given by Z i DZ Ti , and a within-group
component, given by i . In practice, these two components may compete with each other in the model specication, in the sense that similar
i matrices may result from a more complex random-eects component
being added to a simpler within-group component (say i = I), or a simpler random-eects component (say Z i DZ Ti = b2 11T ) being added to a
more complex within-group component. There will generally be a trade-o
between the complexity of the two components of i and some care must
be exercised to prevent nonidentiability, or near nonidentiability, of the
parameters in the model.
In some applications, one may wish to avoid incorporating random eects
in the model to account for dependence among observations, choosing to
use the within-group component i to directly model variancecovariance
structure of the response. This results in the simplied version of the extended linear mixed-eects model (5.1)

(5.5)
y i = X i + i , i N 0, 2 i , i = 1, . . . , M.
Estimation under this model has been studied extensively in the linear
regression literature (Draper and Smith, 1998; Thisted, 1988), usually assuming that the i are known, being referred to as the generalized least
squares problem.
Using the same transformations as in (5.2), we can re-express (5.5) as a
classic linear regression model.

(5.6)
y i = X i + i , i N 0, 2 I , i = 1, . . . , M.
Hence, for xed , the maximum likelihood estimators of and 2 are
obtained by solving an ordinary least-squares problem. Letting X denote
the matrix obtained by stacking up the X i matrices, the conditional MLEs
of and 2 are
1

T
T

()
= (X ) X
(X ) y ,

2
(5.7)

()
y X

&
2
.
() =
N
The proled log-likelihood corresponding to (5.5), which is a function of
only, is obtained by replacing and 2 in the full log-likelihood by their
conditional MLEs (5.7), giving
M

1"

(|y) = const N log y X ()
log |i | .

2 i=1
(5.8)
Maximum likelihood estimates are obtained by rst optimizing the proled

to
log-likelihood (5.8) over and then replacing in (5.7) with its MLE ,
205
get the MLEs for and 2 . Orthogonal-triangular decomposition methods

similar to the ones described in 2.2.3 can be used to obtain more compact
and numerically ecient representations of the proled log-likelihood (5.8)
and the conditional MLEs (5.7), being left to the reader as an exercise.
The restricted likelihood for the extended linear model (5.5) is dened as
in the extended linear mixed-eects model (5.1), by integrating the parameter vector out of the full likelihood. Using the results in Harville (1977),
we write the proled log-restricted-likelihood corresponding to (5.5) as

R (|y) = const (N p) log y X ()

M

1"
1

T
log (X ) X
log |i | ,
2
2 i=1
with p denoting the dimension of . The REML estimator of 2 is dened

as in (5.7), but with the N in the denominator replaced by N p.
The modeling function gls in the nlme library is used to t the extended
linear model (5.5), using either maximum likelihood or restricted maximum
likelihood. We describe the use of this function in 5.4, after presenting the
capabilities available in the nlme library for modeling heteroscedasticity
and correlation.
5.1.3 Decomposing the Within-Group VarianceCovariance

Structure
The i matrices can always be decomposed into a product of simpler
matrices
i = V i C i V i ,
(5.9)
where V i is diagonal and C i is a correlation matrix, that is, a positivedenite matrix with all diagonal elements equal to one. The matrix V i
in (5.9) is not uniquely dened, as we can multiply any number of its rows
by 1 and still get the same decomposition. To ensure uniqueness, we
require that all the diagonal elements of V i be positive.
It is easy to verify that
2
Var (ij ) = 2 [V i ]jj ,
cor (ij , jk ) = [C i ]jk ,
so that V i describes the variance and C i describes the correlation of the

within-group errors i . This decomposition of i into a variance structure
component and a correlation structure component is convenient both theoretically and computationally. It allows us to model and develop code for
the two structures separately and to combine them into a exible family of
models for the within-group variancecovariance. In 5.2 we describe variance function structures to represent the variance component V i and in
5.3 we present correlation structures for the correlation component C i .
206
5.2 Variance Functions for Modeling

Heteroscedasticity
Variance functions are used to model the variance structure of the withingroup errors using covariates. They have been studied in detail in the context of mixed-eects models by Davidian and Giltinan (1995) and in the
context of the extended linear model (5.5) by Carroll and Ruppert (1988).
Following Davidian and Giltinan (1995, Ch. 4), we dene the general
variance function model for the within-group errors in the extended singlelevel linear mixed-eects model (5.1) as
Var (ij |bi ) = 2 g 2 (ij , v ij , ) ,
i = 1, . . . , M,
j = 1, . . . , ni ,
(5.10)
where ij = E [yij |bi ], v ij is a vector of variance covariates, is a vector of

variance parameters and g() is the variance function, assumed continuous
in . For example, if the within-group variability is believed to increase
with some power of the absolute value of a covariate vij , we can write the
variance model as
2
Var (ij |bi ) = 2 |vij |
The variance function in this case is g(x, y) = |x|y and the covariate vij
can be the expected value ij .
The single-level variance function model (5.10) can be generalized to
multilevel models. For example, the variance function model for a two-level
model is
Var (ijk |bi,j , bij ) = 2 g 2 (ijk , v ijk , ) ,
i = 1, . . . , M, j = 1, . . . , Mi ,
k = 1, . . . , nij ,
where ijk = E [yijk |bi,j , bij ]. We concentrate, for the remainder of this
section, in the single-level model (5.1), but all results presented here easily
generalize to multilevel models.
The variance function formulation (5.10) is very exible and intuitive,
because it allows the within-group variance to depend on the xed eects,
, and the random eects, bi , through the expected values, ij . However, as
discussed in Davidian and Giltinan (1995, Ch. 4), it poses some theoretical
and computational diculties, as the within-group errors and the random
eects can no longer assumed to be independent. Under the assumption
that E [i |bi ] = 0, it is easy to verify that Var (ij ) = E [Var (ij |bi )], so
that the dependence on the unobserved random eects can be avoided by
integrating them out of the variance model. Because the variance function
g is generally nonlinear in bi , integrating the random eects out of the variance model (5.10) does not lead to a computationally feasible optimization
5.2 Variance Functions for Modeling Heteroscedasticity
207
procedure. Instead, we proceed as in Davidian and Giltinan (1995, Ch. 6),

and use an approximate variance model in which the expected values ij
are replaced by their BLUPs
ij = xTij + z Tij
bi , where xij and z ij denote,
respectively, the jth rows of X i and Z i ,
ij , v ij , ) ,
Var (ij ) 2 g 2 (
i = 1, . . . , M,
j = 1, . . . , ni .
(5.11)
Under this approximation, the within-group errors are assumed independent of the random eects, as in (5.1), and the results in 5.1.1 can still be
used. Note that, if the conditional variance model (5.10) does not depend
on ij , (5.11) gives the exact marginal variance and no approximation is
required.
When the conditional variance model (5.10) depends on ij , the optimization algorithm follows an iteratively reweighted scheme: for given
(t)
(t) , (t) , (t) , the corresponding BLUPs
ij are obtained and held xed
while the objective function is optimized to produce new estimates (t+1) ,
(t+1)
(t+1) , (t+1) which, in turn, give updated BLUPs
ij , with the process
iterating until convergence. The resulting estimates approximate the (restricted) maximum likelihood estimates. When the variance model does not
involve ij , the (restricted) likelihood can be directly optimized, producing
the exact (restricted) maximum likelihood estimates.
Variance functions for the extended linear model (5.5) are similarly dened, but, because no random eects are present, the model for the marginal
variance does not involve any approximations, being expressed as
Var (ij ) = 2 g 2 (ij , v ij , ) ,
i = 1, . . . , M,
j = 1, . . . , ni ,
(5.12)
where ij = E [yij ] = xTij and v ij and are dened as in (5.10). This

coincides with the variance function model proposed in Carroll and Ruppert (1988, 3). When the variance model (5.12) depends on ij , the optimization algorithm parallels the linear mixed-eects model iteratively
(t)
reweighted scheme described before: for given (t) and (t) estimated ij
are obtained and held xed while the objective function is optimized to
produce new estimates (t+1) and (t+1) , which, in turn, give updated
(t+1)
ij , with the process iterating until convergence. This estimation procedure is known in the literature (Carroll and Ruppert, 1988, 3.2) as
pseudo-likelihood estimation (when the objective function corresponds to
a log-likelihood), or pseudo-restricted-likelihood estimation (when the objective function corresponds to a log-restricted-likelihood). If the variance
model (5.12) does not involve ij , the (restricted) likelihood can be directly optimized and the exact (restricted) maximum likelihood estimates
obtained.
208
TABLE 5.1. Standard varFunc classes.

varFixed
varIdent
varPower
varExp
varConstPower
varComb
xed variance
dierent variances per stratum
power of covariate
exponential of covariate
constant plus power of covariate
combination of variance functions
5.2.1 Variance Functions in nlme: The varFunc Classes

The nlme library provides a set of classes of variance functions, the varFunc classes, that are used to specify within-group variance models in
either the extended linear mixed-eects model (5.1), or the extended linear model (5.5). Table 5.1 lists the standard varFunc classes included in
the nlme library. The varFunc constructors have the same name as their
corresponding classes.
The two main arguments for most of the varFunc constructors are value
and form. The rst species the values of the variance parameters and
the second is a one-sided formula specifying the variance covariate v and,
optionally, a stratication variable for the variance parametersdierent
parameters are used for each level of the stratication variable. For example, to specify age as a variance covariate and to have dierent variance
parameters for each level of Sex, we would use
form = ~ age | Sex
The tted object may be referenced in form by the symbol ., so, for example,
form = ~ fitted(.)
species the tted values as the variance covariate.

Several methods are available for each varFunc class, including initialize, which initializes the variance covariates and stratication variables,
and varWeights, which extracts the variance weights, dened as the inverse
of the variance function values. We now describe and illustrate each of the
standard varFunc classes.
varFixed
This class represents a variance function with no parameters and a single
variance covariate, being used when the within-group variance is known
up to a proportionality constant. The varFixed constructor takes a single
argument, value, which is a one-sided formula that evaluates to the desired variance model. No stratication variables or expressions involving
the symbol . are allowed in value. For example, suppose it is known that
209
the within-group variance increases linearly with age

Var (ij ) = 2 ageij ,
corresponding to a variance function

g ageij = ageij
and being represented as
> vf1Fixed <- varFixed( ~ age )
The variance covariate is calculated in the initialize method, which,

besides the varFixed object, also requires a data argument naming a data
frame in which to evaluate the variance covariate formula. Initialization of
a varFunc object is generally done inside the modeling function (e.g., lme,
gls) which uses it.
> vf1Fixed <- initialize( vf1Fixed, data = Orthodont )
> varWeights( vf1Fixed )
[1] 0.35355 0.31623 0.28868 0.26726 0.35355 0.31623 0.28868
. . .
The variance weights in this case are given by 1/ ageij .

varIdent
This class represents a variance model with dierent variances for each level
of a stratication variable s, taking values in the set {1, 2, . . . , S},
Var (ij ) = 2 s2ij ,
(5.13)
corresponding to the variance function

g (sij , ) = sij .
The variance model (5.13) uses S +1 parameters to represent S variances
and, therefore, is not identiable. To achieve identiability, we need to
impose some restriction on the variance parameters . We use 1 = 1, so
that l , l = 2, . . . , S represent the ratio between the standard deviations of
the lth stratum and the rst stratum. By denition, l > 0, l = 2, . . . , S.
The main arguments to the varIdent constructor are value, a named
numeric vector or a named list specifying the values and the strata of the
parameters that are allowed to vary in the optimization, and form, a onesided formula of the form ~1|s specifying the stratication variable s. Some,
or all, of the variance parameters may be set to xed values (which do not
vary during the optimization) using the argument fixed. This may be given
either as named numeric vector, or a named list, specifying the values and
the strata of the that are to remain xed during the optimization.
For example, to specify and initialize a variance function with dierent
variances per gender for the Orthodont data described in 1.4.1, with the
initial value of the parameter corresponding to Female set to 0.5, we use
210
> vf1Ident <- varIdent( c(Female = 0.5), ~ 1 | Sex )

> vf1Ident <- initialize( vf1Ident, Orthodont )
> varWeights( vf1Ident )
Male Male Male Male Male Male Male Male Male Male Male Male
1
1
1
1
1
1
1
1
1
1
1
1
. . .
Female Female
2
2
If the ratio between the Female and Male standard deviations, given by ,
is to be kept xed at 0.5 and not allowed to vary during the optimization,
we can use instead
> vf2Ident <- varIdent( form = ~ 1 | Sex, fixed = c(Female = 0.5))
Male Male Male Male Male Male Male Male Male Male Male Male
1
1
1
1
1
1
1
1
1
1
1
1
. . .
Female Female
2
2
It is possible to specify several stratication variables simultaneously in

form, separated by the * operator. The levels of the dierent stratication
variables are pasted together and a dierent is used for each combination
of levels. For example, to specify a variance function with dierent variances
for each age and Sex combination we can use
> vf3Ident <- varIdent( form = ~ 1 | Sex * age )
Male*8 Male*10 Male*12 Male*14 Male*8 Male*10 Male*12 Male*14
1
1
1
1
1
1
1
1
. . .
Female*12 Female*14
1
1
By default, all variance parameters are initialized to 1, corresponding to

equal variance weights of 1 being assigned to all strata.
varPower
The variance model represented by this class is
2
Var (ij ) = 2 |vij |

g (vij , ) = |vij | ,
(5.14)
211
which is a power of the absolute value of the variance covariate. The parameter is unrestricted (i.e., may take any value in the real line) so (5.14)
can model cases where the variance increases or decreases with the absolute
value of the variance covariate. Note that, when vij = 0 and > 0, the
variance function is 0 and the variance weight is undened. Therefore, this
class of variance functions should not be used with variance covariates that
may assume the value 0.
The main arguments to the varPower constructor are value and form,
which specify, respectively, an initial value for , when this is allowed to vary
in the optimization, and a one-sided formula with the variance covariate. By
default, value = 0, corresponding to equal variance weights of 1, and form =
~fitted(.), corresponding to a variance covariate given by the tted values.
For example, to specify a variance model with the parameter initially set
to 1, and allowed to vary in the optimization, and the tted values as the
variance covariate, we use
> vf1Power <- varPower( 1 )
> formula( vf1Power )
~ fitted(.)
The fixed argument can be used to set to a xed value, which does
not change in the optimization. For example,
> vf2Power <- varPower( fixed = 0.5 )
species a model in which the variance increases linearly with the tted
values.
An optional stratication variable, or several stratication variables separated by *, may be included in the form argument, with a dierent being used for each stratum. This corresponds to the following generalization
of (5.14)
2sij
Var (ij ) = 2 |vij |
sij
g (vij , sij , ) = |vij |
When a stratication variable is included in form, the arguments value

and fixed must be either named vectors, or named lists. For example, to
specify a model for the Orthodont data in which the variance increases
linearly with the tted values for Males and stays constant for Females,
with both parameters held xed during the optimization, we use
> vf3Power <- varPower( form = ~ fitted(.) | Sex,
+
fixed = list(Male = 0.5, Female = 0) )
varExp
Var (ij ) = 2 exp (2vij ) ,
(5.15)
212

g (vij , ) = exp (vij ) ,
which is an exponential function of the variance covariate. The parameter
is unrestricted, so (5.15) can model cases where the variance increases
or decreases with the variance covariate. There are no restrictions on the
variance covariate, which, in particular, may take the value 0.
The arguments value, form, and fixed to the varExp constructor are
dened and declared as in varPower, assuming also the same default values.
An optional stratication variable, or stratication variables separated by
*, may be specied in form, corresponding to the following generalization
of (5.15)

Var (ij ) = 2 exp 2sij vij , g (vij , sij , ) = exp sij vij .
As with varPower, when a stratication variable is included in form, the
arguments value and fixed must be either named vectors, or named lists.
For example, a variance model in which the Male variance increases exponentially with age, but the Female variance is held at a constant value is
expressed as
> vf1Exp <- varExp( form = ~ age | Sex, fixed = c(Female = 0) )
varConstPower

2
Var (ij ) = 2 1 + |vij |2 ,
(5.16)
g (vij , ) = 1 + |vij | 2 ,
which is a constant plus a power of the absolute value of the variance

covariate. 1 is restricted to be positive and 2 is unrestricted. If 2 > 0,
which will generally be the case in practice, the varConstPower variance
function is approximately constant and equal to 1 , when the variance
covariate is close to 0, and increases with the absolute value of the variance
covariate as it gets away from 0. This generally gives a more realistic model
than the varPower model, in cases when the variance covariate takes values
close or equal to 0.
Initial values for parameters that are allowed to vary during the optimization are specied through the arguments const and power. The former
is used for 1 and the latter for 2 . By default, const = 1 and power = 0,
corresponding to constant variance weights equal to 1. The form argument
is dened as in varPower and varExp, assuming the same default value. The
213
argument fixed is given as a list with components const and power and
may be used to set either, or both, of the variance parameters to a xed
value. For example, to specify a variance function with 1 xed at the value
1, with 2 allowed to vary but initialized to 0.5, and the tted values as the
variance covariate, we use
> vf1ConstPower <- varConstPower( power = 0.5,
+
fixed = list(const = 1) )
An optional stratication variable, or several stratication variables separated by *, may be included in the form argument, with dierent 1 and
2 being used for each stratum. This corresponds to the following generalization of (5.16)

2
Var (ij ) = 2 1,sij + |vij | 2,sij ,
2,sij
g (vij , sij , ) = 2,sij + |vij |
When a stratication variable is included in form, the arguments const,

power and the components of fixed must be either named vectors, or named
lists.
varComb
This class allows the combination of two, or more, variance models, by
multiplying together the corresponding variance functions. For example,
a variance model in which the variance is proportional to an exponential
function of a variance covariate, but in which the proportionality constant
varies according to the levels of an stratication variable s can be expressed
as a product of a varIdent variance function and a varExp variance function.
2
exp (22 vij ) = 2 g12 (sij , 1 ) g22 (vij , 2 ) ,
Var (ij ) = 2 1,s
ij
g1 (sij , 1 ) = 1,sij ,
g2 (vij , 2 ) = exp (2 vij ) .
(5.17)
The varComb constructor can take any number of varFunc objects as arguments. For example, to represent (5.17), with Sex as the stratication
variable and age as the variance covariate, we use
> vf1Comb <- varComb( varIdent(c(Female = 0.5), ~ 1 | Sex),
+
varExp(1, ~ age) )
> vf1Comb <- initialize( vf1Comb, Orthodont )
> varWeights( vf1Comb )
1
2
3
4
5
6
7
8
9
0.125 0.1 0.083333 0.071429 0.125 0.1 0.083333 0.071429 0.125
. . .
98
99
100 101 102
103
104 105 106
107
0.2 0.16667 0.14286 0.25 0.2 0.16667 0.14286 0.25 0.2 0.16667
108
0.14286
214
The varComb variance weights correspond to the product of the individual

variance weights of each of its varFunc objects. In the case of vf1Comb, these
are given by 1/ ageij for Male (observations 164) and 2/ ageij for Female
(observations 65108).
New varFunc classes, representing user-dened variance functions, can be
added to the set of standard classes in Table 5.1 and used with the modeling functions in the nlme library. For this, one must specify a constructor
function, generally with the same name as the class, and, at a minimum,
methods for the functions coef, coef<-, and initialize. The varPower constructor and methods can serve as templates for these.
5.2.2 Using Variance Functions with lme

Variance functions are specied in the lme function using the weights argument. By default, weights = NULL, corresponding to a homoscedastic variance model for the within-group errors. Variance models can be specied
in weights either as a one-sided formula, in which case it is passed as the
single argument to the varFixed constructor, or as a varFunc object, created using the standard constructors described in 5.2.1, or a user-dened
constructor. In this section, we describe the use of variance models in lme
through the analysis of two examples of grouped data with heteroscedastic
within-group errors.
High-Flux Hemodialyzer Ultraltration Rates
Vonesh and Carter (1992) describe and analyze data measured on highux hemodialyzers to assess their in vivo ultraltration characteristics. The
ultraltration rates (in ml/hr) of 20 high-ux dialyzers were measured at
7 ascending transmembrane pressures (in dmHg). The in vitro evaluation
of the dialyzers used bovine blood at ow rates of either 200 dl/min or
300 dl/min. These data are also described in Appendix A.6 and are included
in the nlme library as the groupedData object Dialyzer.
The plots of the ultraltration rates versus transmembrane pressure by
bovine blood ow rate, displayed in Figure 5.1, reveal that the ultraltration rate increases with transmembrane pressure and that higher ultraltration rates are attained with the 300 dl/min blood ow dialyzers. These
plots also indicate that the variability in the ultraltration rates increases
with transmembrane pressure.
Vonesh and Carter (1992) use a nonlinear model to represent the relationship between ultraltration rate and transmembrane pressure. An
alternative analysis is presented in Littell et al. (1996), who compare several linear mixed-eects models and extended linear models to represent
the ultraltration rate yij at the jth transmembrane pressure xij for the
ith subject. The best linear mixed-eects model indicated by their analysis

0.5
1.0
1.5
Ultrafiltration rate (ml/hr)
200
2.0
2.5
215
3.0
300
60
50
40
30
20
10
0
0.5
1.0
1.5
2.0
2.5
3.0
Transmembrane pressure (dmHg)
FIGURE 5.1. Hemodialyzer ultraltration rates (in ml/hr) measured at 7 dierent transmembrane pressures (in dmHg) on 20 high-ux dialyzers. In vitro evaluation of dialyzers based on bovine blood ow rates of 200 dl/min and 300 dl/min.
is
yij = (0 + 0 Qi + b0i ) + (1 + 1 Qi + b1i ) xij
+ (2 + 2 Qi + b2i ) x2ij + (3 + 3 Qi ) x3ij + (4 + 4 Qi ) x4ij + ij ,
(5.18)
b0i

bi = b1i N (0, ) , ij N 0, 2 ,
b2i
where Qi is a binary variable taking values 1 for 200 dl/min hemodialyzers
and 1 for 300 dl/min hemodialyzers; 0 , 1 , 2 , 3 , and 4 are, respectively,
the intercept, linear, quadratic, cubic, and quartic xed eects averaged
over the levels of Q; i is the blood ow eect associated with the xed eect
i ; bi is the vector of random eects, assumed independent for dierent i;
and ij is the within-group error, assumed independent for dierent i, j and
independent of the random eects.
We t the homoscedastic linear mixed-eects model (5.18) with
> fm1Dial.lme <+
lme(rate ~(pressure + pressure^2 + pressure^3 + pressure^4)*QB,
+
Dialyzer, ~ pressure + pressure^2)
> fm1Dial.lme
Data: Dialyzer
Fixed: rate ~(pressure + pressure^2 + pressure^3 + pressure^4)*QB
216
(Intercept) pressure I(pressure^2) I(pressure^3) I(pressure^4)

-16.598
88.673
-42.732
9.2165
-0.77561
QB pressure:QB I(pressure^2):QB I(pressure^3):QB
-0.63174
0.31044
1.5741
0.050928
I(pressure^4):QB
-0.085967
Random effects:
Formula: ~ pressure + pressure^2 | Subject
StdDev
Corr
(Intercept) 1.4989 (Intr) pressr
pressure 4.9074 -0.507
I(pressure^2) 1.4739 0.311 -0.944
Residual 1.8214
The primary tool for investigating within-group heteroscedasticity is the

plots of residuals against the tted values and other candidate variance
covariates. In the hemodialyzer example, the transmembrane pressure is a
natural candidate for the variance covariate. The corresponding residuals
plot, obtained with
> plot( fm1Dial.lme, resid(.) ~ pressure, abline = 0 ) # Figure 5.2
and displayed in Figure 5.2, conrms that the within-group variability increases with transmembrane pressure.
Because of its exibility, the varPower variance function is a common
choice for modeling monotonic heteroscedasticity, when the variance covariate is bounded away from zero (transmembrane pressure varies in the
data between 0.235 dmHg and 3.030 dmHg). The corresponding model
only diers from (5.18) in that the within-group errors are allowed to be
heteroscedastic with variance model
Var (ij ) = 2 x2
ij
(5.19)
and we t it with
> fm2Dial.lme <- update( fm1Dial.lme,
+
weights = varPower(form = ~ pressure) )
> fm2Dial.lme
Data: Dialyzer
Fixed: rate ~(pressure + pressure^2 + pressure^3 + pressure^4)*QB
-17.68
93.711
-49.186
12.245
-1.2426
217
Residuals (ml/hr)
-2
-4
0.5
1.0
1.5
2.0
2.5
3.0
FIGURE 5.2. Plot of residuals versus transmembrane pressure for the homoscedastic tted object fm1Dial.lme.

-0.92072
1.3528
0.48071
0.49118
I(pressure^4):QB
-0.14624
Random effects:
Formula: ~ pressure + pressure^2 | Subject
StdDev
Corr
(Intercept) 1.8569 (Intr) pressr
pressure 5.3282 -0.522
I(pressure^2) 1.6483 0.362 -0.954
Residual 1.2627
Variance function:
Structure: Power of variance covariate
Formula: ~ pressure
power
0.74923
The anova method can be used to test the signicance of the heteroscedastic model (5.19).
218
-1
-2
0.5
1.0
1.5
2.0
2.5
3.0
FIGURE 5.3. Plot of standardized residuals versus transmembrane pressure for

the varPower tted object fm2Dial.lme.
> anova( fm1Dial.lme, fm2Dial.lme )

Model df
AIC
BIC logLik
fm1Dial.lme
1 17 686.78 735.53 -326.39
fm2Dial.lme
2 18 655.01 706.63 -309.51 1 vs 2
33.77 <.0001
As expected, there is a highly signicant increase in the log-likelihood associated with the inclusion of the varPower variance function. The plot of
the standardized residuals, dened as rij = (yij yij ) /(
xij ) in this case,
versus the variance covariate is used to graphically assess the adequacy of
the variance model.
> plot( fm2Dial.lme, resid(., type = "p") ~ pressure,
+
abline = 0 )
# Figure 5.3
The resulting plot, displayed in Figure 5.3, reveals a reasonably homogeneous pattern of variability for the standardized residuals, indicating that
the varPower model successfully describes the within-group variance.
We assess the variability of the variance parameter estimate with the
intervals method.
> intervals( fm2Dial.lme )
. . .
Variance function:
lower
est. upper
power 0.4079 0.74923 1.0906
. . .
The resulting condence interval is based on a normal approximation for

the distribution of the (restricted) maximum likelihood estimators, with

0.5
1.0
200
1.5
2.0
2.5
219
3.0
300
Residuals (ml/hr)
4
2
0
-2
-4
-6
0.5
1.0
1.5
2.0
2.5
3.0
FIGURE 5.4. Plot of residuals versus transmembrane pressure by bovine blood

ow level for the varPower tted object fm2Dial.lme.
an approximate variancecovariance matrix given by the inverse of the

observed information matrix evaluated at the converged values.
An interesting question about the hemodialyzer data is whether the
within-group variability depends on the bovine blood ow level. We can
investigate this graphically with the plots of the raw residuals versus transmembrane pressure by blood ow level, obtained with
> plot(fm2Dial.lme, resid(.) ~ pressure|QB, abline = 0)
# Fig. 5.4
and displayed in Figure 5.4. Note that the pattern of increasing variability
is still present in the raw residuals, as we did not transform the data, but,
instead, incorporated the within-group heteroscedasticity in the model. The
heteroscedastic patterns seem the same for both blood ow levels. We can
test this formally using
> fm3Dial.lme <- update(fm2Dial.lme,
+
weights=varPower(form = ~ pressure | QB))
> fm3Dial.lme
. . .
Variance function:
Structure: Power of variance covariate, different strata
Formula: ~ pressure | QB
200
300
0.64775 0.83777
. . .
Model df
AIC
BIC logLik
fm2Dial.lme
1 18 655.01 706.63 -309.51
fm3Dial.lme
2 19 656.30 710.78 -309.15 1 vs 2 0.71091 0.3991
220

0.5
1.5
2.5
0.5
1.5
2.5
0.5
1.5
16
13
14
18
15
19
17
20
11
2.5
60
Ultrafiltration rate (ml/hr)
40
20
12
60
40
20
10
60
40
20
0
0.5
1.5
2.5
0.5
1.5
2.5
0.5
1.5
2.5
0.5
1.5
2.5
FIGURE 5.5. Plot of predicted ultraltration rates versus transmembrane pressure by subject corresponding to the tted object fm2Dial.lme.
As expected, there is no evidence of a signicant eect of blood ow in the

within-group variability.
Because the transmembrane pressure takes values relatively close to zero,
we may wish to investigate if a varConstPower variance model

2
Var (ij ) = 2 1 + xij2
would give a more appropriate representation of the within-group variance.
> fm4Dial.lme <- update(fm2Dial.lme,
+
weights = varConstPower(form = ~ pressure))
Model df
AIC
BIC logLik
fm2Dial.lme
1 18 655.01 706.63 -309.51
fm4Dial.lme
2 19 656.85 711.34 -309.43 1 vs 2 0.15915 0.6899
The nearly identical log-likelihood values indicate that there is no need for
the extra parameter associated with varConstPower and that the simpler
varPower model should be maintained.
A nal assessment of the heteroscedastic version of the linear mixedeects model (5.18) with within-group variance model (5.19) is provided
by the plot of the augmented predictions by subject, obtained with
> plot( augPred(fm2Dial.lme), grid = T )
# Figure 5.5
and displayed in Figure 5.5. The predicted values closely match the observed ultraltration rates, attesting the adequacy of the model.
221
One of the questions of interest for the hemodialyzer data is whether the
ultraltration characteristics dier with the evaluation blood ow rates,
which is suggested by the plots in Figure 5.1. The anova method can be
used to test the signicance of the terms associated with the evaluation
blood ow rates, in the order they were entered in the model (a sequential
type of test).
> anova( fm2Dial.lme )
. . .
(Intercept)
1
112
552.9 <.0001
pressure
1
112 2328.6 <.0001
I(pressure^2)
1
112 1174.6 <.0001
I(pressure^3)
1
112
359.9 <.0001
I(pressure^4)
1
112
12.5 0.0006
QB
1
18
4.8 0.0414
pressure:QB
1
112
80.1 <.0001
I(pressure^2):QB
1
112
1.4 0.2477
I(pressure^3):QB
1
112
2.2 0.1370
I(pressure^4):QB
1
112
0.2 0.6840
. . .
The large p-values associated with terms of degree greater than or equal to
2 involving the variable QB suggest that they are not needed in the model.
We can verify their joint signicance with
> anova( fm2Dial.lme, Terms = 8:10 )
F-test for: I(pressure^2):QB, I(pressure^3):QB, I(pressure^4):QB
1
3
112 1.2536 0.2939
The large p-value for the F-test conrms that these terms could be eliminated from the model.
Body Weight Growth in Rats
As a second example to illustrate the use of variance functions with lme, we
revisit the BodyWeight data introduced in 3.2.1 and described in Hand and
Crowder (1996, Table A.1), on the body weights of rats measured over 64
days. The body weights of the rats (in grams) are measured on day 1 and
every seven days thereafter, until day 64, with an extra measurement on
day 44. There are three groups of rats, each on a dierent diet. These data
are also described in Appendix A.3 and are included in the nlme library as
the groupedData object BodyWeight.
The plots of the body weights versus time by diet, shown in Figure 5.6,
indicate strong dierences among the three diet groups. There is also evidence of a rat in diet group 2 with an unusually high initial body weight.
The body weights appear to grow linearly with time, possibly with different intercepts and slopes for each diet, and with intercept and slope
222

0
1
20
40
60
Body weight (g)
600
500
400
300
20
40
60
20
40
60
Time (days)
FIGURE 5.6. Body weights of rats measured over a period of 64 days. The rats
are divided into three groups on dierent diets.
random eects to account for rat-to-rat variation. We express this as the

linear mixed-eects model
yij = (0 + 02 D2i + 03 D3i + b0i ) + (1 + 12 D2i + 13 D3i + b1i ) tij + ij ,

b0i
(5.20)
N (0, ) , ij N 0, 2 ,
bi =
b1i
where D2i is a binary variable taking the value 1 if the ith rat receives Diet
2; D3i is a binary indicator variable for Diet 3; 0 and 1 are, respectively,
the average intercept and the average slope for rats under Diet 1; 0k is the
average dierence in intercept between rats under Diet k and rats under
Diet 1; 1k is the average dierence in slope between rats under Diet k and
rats under Diet 1; bi is the vector of random eects, assumed independent
for dierent i; and ij is the within-group error, assumed independent for
dierent i, j and independent of the random eects.
To t (5.20), we rst need to reset the contrasts option to use the
contr.treatment parameterization for factors, as described in 1.2.1,
and then use

> fm1BW.lme <- lme( weight ~ Time * Diet, BodyWeight,
+
random = ~ Time )
223
2
1
0
-1
-2
-3
300
400
500
600
Fitted values (g)
FIGURE 5.7. Plot of standardized residuals versus tted values for the homoscedastic tted object fm1BW.lme.
> fm1BW.lme
Data: BodyWeight
Fixed: weight ~ Time * Diet
(Intercept)
Time Diet2 Diet3 TimeDiet2 TimeDiet3
251.65 0.35964 200.67 252.07
0.60584
0.29834
Random effects:
Formula: ~ Time | Rat
StdDev
Corr
Time 0.24841 -0.149
Residual 4.44361
The plot of the standardized residuals versus the tted values, displayed
in Figure 5.7, gives clear indication of within-group heteroscedasticity.
Because the tted values are bounded away from zero, we can use the
varPower variance function to model the heteroscedasticity.
> fm2BW.lme <- update( fm1BW.lme, weights = varPower() )
> fm2BW.lme
Data: BodyWeight
224
Fixed: weight ~ Time * Diet
(Intercept)
Time Diet2 Diet3 TimeDiet2 TimeDiet3
251.6 0.36109 200.78 252.17
0.60182
0.29523
Random effects:
StdDev
Corr
Time 0.24373 -0.147
Residual 0.17536
Variance function:
Formula: ~ fitted(.)
power
0.54266
Note that the form argument did not need to be specied in the call to
varPower, because its default value, ~fitted(.), corresponds to the desired
variance covariate.
The plot of the standardized residuals versus tted values for the heteroscedastic t corresponding to fm2BW.lme, displayed in Figure 5.8, indicates that the varPower variance function adequately represents the withingroup heteroscedasticity.
We can test the signicance of the variance parameter in the varPower
model using the anova method, which, as expected, strongly rejects the
assumption of homoscedasticity (i.e., = 0).
> anova( fm1BW.lme, fm2BW.lme )
Model df
AIC
BIC logLik
fm1BW.lme
1 10 1171.7 1203.1 -575.86
fm2BW.lme
2 11 1163.9 1198.4 -570.96 1 vs 2 9.7984 0.0017
The primary question of interest for the BodyWeight data is whether the
growth rates dier signicantly among diets. Because of the parametrization used in (5.20), the summary method only provides tests for dierences
between Diets 1 and 2 (12 in (5.20)) and between Diets 1 and 3 (13
in (5.20)).
> summary( fm2BW.lme )
. . .
Fixed effects: weight ~ Time * Diet
(Intercept) 251.60
13.068 157 19.254 <.0001
225
-1
-2
-3
300
400
500
600
Fitted values (g)
FIGURE 5.8. Plot of standardized residuals versus tted values for the varPower
tted object fm2BW.lme.
Time
0.36
Diet2 200.78
Diet3 252.17
TimeDiet2
0.60
TimeDiet3
0.30
. . .
0.088 157
22.657 13
22.662 13
0.155 157
0.156 157
4.084
8.862
11.127
3.871
1.893
0.0001
<.0001
<.0001
0.0002
0.0602
There appears to be a signicant increase in growth rate associated with

Diet 2 (TimeDiet2) and a borderline signicant increase in growth rate for
Diet 3 (TimeDiet3). We can test the dierence in growth rates between Diets
2 and 3 using the anova method.
> anova( fm2BW.lme, L = c(TimeDiet2 = 1, TimeDiet3 = -1) )
F-test for linear combination(s)
TimeDiet2 TimeDiet3
1
-1
1
1
157 2.8608 0.0927
The argument L is used to specify contrasts of coecients to be tested as

equal to zero. The names of the elements in L must correspond to coecients
in the model. There does not seem to be a signicant dierence in growth
rate between Diets 2 and 3.
226
5.3 Correlation Structures for Modeling

Dependence
Correlation structures are used to model dependence among observations.
In the context of mixed-eects models and extended linear models, they
are used to model dependence among the within-group errors. Historically,
correlation structures have been developed for two main classes of data:
time-series data and spatial data. The former is generally associated with
observations indexed by an integer-valued time variable, while the latter
refers primarily to observations indexed by a two-dimensional spatial location vector, taking values in the real plane.
To establish a general framework for correlation structures, we assume
that the within-group errors ij are associated with position vectors pij .
For time series data, the pij are typically integer scalars, while for spatial
data they are generally two-dimensional coordinate vectors. The correlation structures considered in this book are assumed to be isotropic (Cressie,
1993, 2.3.1); that is, the correlation between two within-group errors ij , ij
is assumed to depend on the corresponding position vectors pij , pij only

through some distance between them, say d pij , pij , and not on the particular values they assume. The general within-group correlation structure
for single-level grouping is expressed, for i = 1, . . . , M and j, j = 1, . . . , ni ,
as
$
%
(5.21)
cor (ij , ij ) = h d(pij , pij ), ,
where is a vector of correlation parameters and h() is a correlation
function taking values between 1 and 1, assumed continuous in , and
such that h (0, ) = 1, that is, if two observations have identical position
vectors, they are the same observation and therefore have correlation 1.
The single-level correlation model (5.21) can be easily generalized to
multilevel grouping. For example, the correlation model for two nested
levels of grouping is
$
%
cor (ijk , ijk ) = h d(pijk , pijk ), ,
i = 1, . . . , M,
j = 1, . . . , Mi ,
k, k = 1, . . . , nij .
Note that the correlation model applies to within-group errors within the
same innermost level of grouping. We concentrate, for the remainder of
this section, in the single-level correlation model (5.21), but all correlation
structures presented here can be easily extended to multilevel models.
5.3.1 Serial Correlation Structures

Serial correlation structures are used to model dependence in time-series
data, that is, data observed sequentially over time and indexed by a onedimensional position vector. Serial correlation structures for linear models
5.3 Correlation Structures for Modeling Dependence
227
without random eects have been extensively studied by Box, Jenkins and
Reinsel (1994). In the context of linear mixed-eects models, they are described in detail in Jones (1993).
We simplify the isotropy assumption and assume that the serial correlation model depends on the one-dimensional positions pij , pij only through
their absolute dierence. The general serial correlation model is then dened as
cor (ij , ij ) = h (|pij pij | , ) .
In the context of time-series data, the correlation function h () is referred to as the autocorrelation function. The empirical autocorrelation
function (Box et al., 1994, 3), a nonparametric estimate of the autocorrelation function, provides a useful tool for investigating serial correlation in
ij , denote the standardized residtime-series data. Let rij = (yij yij ) /
2
= Var (ij ). The empirical
uals from a tted mixed-eects model, with ij
autocorrelation at lag l is dened as
M ni l
i=1
j=1 rij ri(j+l) /N (l)
,
(5.22)
(l) =
M ni 2
j=1 rij /N (0)
i=1
where N (l) is the number of residual pairs used in the summation dening
the numerator of (l).
Serial correlation structures typically require that the data be observed
at integer time points and do not easily generalize to continuous position
vectors. We describe below some of the most common serial correlation
structures used in practice, all of which are implemented in the nlme library.
Compound Symmetry
This is the simplest serial correlation structure, which assumes equal correlation among all within-group errors pertaining to the same group. The
corresponding correlation model is
cor (ij , ij ) = ,
j = j ,
h(k, ) = ,
k = 1, 2, . . . ,
(5.23)
where the single correlation parameter is generally referred to as the

intraclass correlation coecient.
The variancecovariance matrix for the ith response vector in a singlelevel linear mixed-eects model with independent and identically distributed
within-group errors, with variance 2 , and a single intercept random eect,
T
with variance b2 , is 2 I + b2 11
,Tcorresponding to the correlation matrix
2
2
2
2
2
2
/ b + I+b / b + 11 . This is equivalent to a compound
sym
metry structure with intraclass correlation = b2 / b2 + 2 , indicating
that the correlation structure dened by this linear mixed-eects model
is a particular case of (5.23). The two correlation models are not equivalent, however, because the intraclass correlation in the linear mixed-eects
228
model can only take values between 0 and 1, while (5.23) allows to take
negative values (to have a positive-denite compound symmetry correlation
structure, it is only required that > 1/ [maxiM (ni ) 1]).
The compound symmetry correlation model tends to be too simplistic
for practical applications involving time-series data, as, in general, it is
more realistic to assume a model in which the correlation between two
observations decreases, in absolute value, with their distance. It is a useful
model for applications involving short time series per group, or when all
observations within a group are collected at the same time, as in split-plot
experiments.
General
This structure represents the other extreme in complexity to the compound
symmetry structure. Each correlation in the data is represented by a different parameter, corresponding to the correlation function
h(k, ) = k ,
k = 1, 2, . . . .
(5.24)
Because the number of parameters in (5.24) increases quadratically with

the maximum number of observations within a group, this correlation structure will often lead to overparameterized models. When there are relatively
few observations per group, the general correlation structure is useful as
an exploratory tool to determine a more parsimonious correlation model.
AutoregressiveMoving Average
This family of correlation structures, described in detail in Box et al. (1994),
includes dierent classes of linear stationary models: autoregressive models, moving average models, and mixture of autoregressivemoving average
models. These are also called Box and Jenkins models.
Autoregressivemoving average correlation models assume that the data
are observed at integer time points and, for simplicity, we use the notation
t to refer to an observation taken at time t. The distance, or lag, between
two observations t and s is given by |t s|. So lag-1 refers to observations
one time unit apart and so on.
Autoregressive models express the current observation as a linear function of previous observations plus a homoscedastic noise term, at , centered
at 0 (E [at ] = 0) and assumed independent of the previous observations.
t = 1 t1 + + p tp + at .
(5.25)
The number of past observations included in the linear model (5.25), p, is

called the order of the autoregressive model, which is denoted by AR(p).
There are p correlation parameters in an AR(p) model, given by =
(1 , . . . , p ).
229
The AR(1) model is the simplest (and one of the most useful) autoregressive model. Its correlation function decreases in absolute value exponentially with lag.
h(k, ) = k ,
k = 0, 1, . . . .
(5.26)
The single correlation parameter, , represents the lag-1 correlation and

takes values between 1 and 1. The AR(1) model is one of the few serial
correlation structures that can be generalized to continuous time measurements. We dene the continuous time AR(1) correlation function (Jones,
1993, 3.3), denoted CAR(1), as
h(s, ) = s ,
s 0,
0.
(5.27)
Note that the single correlation parameter in (5.27) must be non-negative.

For autoregressive models of order greater than 1, the correlation function does not admit a simple representation as in (5.26), being dened
recursively through the dierence equation (Box et al., 1994, 3.2.2)
h(k, ) = 1 h(|k 1|, ) + + p h(|k p|, ),
k = 1, 2, . . . .
Moving average correlation models assume that the current observation

is a linear function of independent and identically distributed noise terms.
t = 1 at1 + + q atq + at .
(5.28)
The number of noise terms included in the linear model (5.28), q, is called
the order of the moving average model, which is denoted by MA(q). There
are q correlation parameters in an MA(q) model, given by = (1 , . . . , q ).
The correlation function for an MA(q) model is
'
k +1 k1 ++kq q
, k = 1, . . . , q,
1+12 ++q2
h(k, ) =
0,
k = q + 1, q + 2, . . . .
Observations more than q time units apart are uncorrelated, as they do not
share any common noise terms at .
Strategies for estimating the order of autoregressive and moving average
models in time series applications are discussed in Box et al. (1994, 3).
Mixed autoregressivemoving average models, called ARMA models, are
obtained by combining together an autoregressive model and a moving
average model.
t =
p
"
i=1
i ti +
q
"
j atj + at .
j=1
There are p + q correlation parameters in an ARMA(p, q) model, corresponding to the combination of the p autoregressive parameters =
230
(1 , . . . , p ) and the q moving average parameters = (1 , . . . , q ). By

convention, ARMA(p, 0) = AR(p) and ARMA(0, q) = M A(q), so that both
autoregressive and moving average models are particular examples of the
general ARMA model.
The correlation function for an ARMA(p, q) model behaves like the correlation function of an AR(p) model for lags greater than q and like an AR(p)
correlation function plus a term related to the moving average part of the
model, between lags 1 and q. It is obtained using the recursive relations
1 h(|k 1|, ) + + p h(|k p|, )+

1 (k 1, ) + + q (k q, ), k = 1, . . . , q,
h(k, ) =
1 h(|k 1|, ) + + p h(|k p|, ), k = q + 1, q + 2, . . . ,

where (k, , ) = E [tk at ] /Var (t ). Note that (k, , ) = 0, k =
1, 2, . . . , as, in this case, tk and at are independent and E [at ] = 0.
5.3.2 Spatial Correlation Structures

Spatial correlation structures were originally proposed to model dependence
in data indexed by continuous two-dimensional position vectors, such as
geostatistical data, lattice data, and point patterns. Because the isotropic
spatial correlation structures we consider are continuous functions of some
distance between position vectors, they are easily generalized to any nite
number of position dimensions. In particular, they can be used with time
series data. The basic reference for spatial correlation structures used with
linear models with no random eects is Cressie (1993). Spatial correlation
structures in the context of mixed-eects models are described at length in
Diggle et al. (1994).
For simplicity of notation, we denote by x the observation taken at
position x = (x1 , . . . , xr )T . Any distance metric may be used with isotropic
spatial correlation structures, the
most common being the Euclidean, or L2 ,
r
2
distance, dened as dE (x , y ) =
i=1 (xi yi ) . Other popular choices
r
are the Manhattan, or L1 , distance dMan (x , y ) = i=1 |xi yi | and the
maximum distance dMax (x , y ) = maxi=1,... ,r |xi yi |.
Semivariogram
Spatial correlation structures are generally represented by their semivariogram, instead of their correlation function (Cressie, 1993, 2.3.1). The
semivariogram of an isotropic spatial correlation structure with a distance
function d() is dened as
2
[d (x , y ) , ] = 12 Var (x y ) = 12 E [x y ] ,
(5.29)
with the last equality following from E [x ] = E [y ] = 0. The withingroup errors can be standardized to have unit variance, without changing
231
their correlation structure. So, without loss of generality, we assume that

Var (x ) = 1, x. In this case, () will depend only on the correlation
parameters and it is easy to verify that
(s, ) = 1 h(s, ).
It follows from h(0, ) = 1 that (0, ) = 0. To account for abrupt changes
at very small distances, it is desirable, in some applications, to allow a discontinuity in () at 0, so that (s, ) c0 , when s 0, with 0 < c0 < 1.
This is called a nugget eect in the spatial statistics literature (Cressie,
1993, 2.3.1). In terms of the correlation function, the nugget eect translates into h(s, ) 1 c0 as s 0. It is easy to obtain a correlation
function that incorporates a nugget eect from a correlation function that
is continuous in s.
+
(1 c0 ) hcont (s, ) , s > 0,
hnugg (s, c0 , ) =
0,
s = 0.
2
The standardized residuals rij = (yij yij ) /
ij , with ij
= Var (ij ), are
the primary quantities used for estimating the semivariogram. The classical
estimator of the semivariogram (Matheron, 1962) is
1 "
2N (s) i=1
"
(s) =
(rij rij ) ,
(5.30)
d(pij ,pij )=s
where N (s) denotes the number of residual pairs at a distance s of each

other. Because
(s) uses the squared dierences between residual pairs, it
can be quite sensitive to outliers. Furthermore, because each residual rij
appears in ni 1 squared dierences in (5.30), a single outlier can aect the
estimation of the semivariogram at several distances. A robust estimator
of the semivariogram, proposed by Cressie and Hawkins (1980), uses the
square-root dierences to reduce the inuence of outliers.
4
(s) =
1
2N (s)
M
"
"
1/2
|rij rij |
/ (0.457 + 0.494/N (s)) .
i=1 d(p ,p )=s

ij
ij
(5.31)
Some Isotropic Variogram Models
Cressie (1993, 2.3.1) describes an extensive collection of isotropic variogram models and give conditions for their validity. The single-parameter
models in Table 5.2 are a subset of the collection in Cressie (1993), with
the Linear variogram model modied so that it is bounded in s. I(s < )
denotes a binary variable taking value 1 when s < and 0 otherwise. Most
232
TABLE 5.2. Some isotropic variogram models for spatial correlation structures.
Exponential
Gaussian
Linear
Rational quadratic
Spherical
(s, ) = 1 exp (s/)

2
(s, ) = 1 exp (s/)
(s, ) = 1 (1 s/) I(s < )
2
(s, ) = (s/) / 1 + (s/)

3
(s, ) = 1 1 1.5 (s/) + 0.5 (s/) I(s < )
of the models in Table 5.2 are also described in Littell et al. (1996, 9.3.1).
The correlation parameter is generally referred to as the range in the
spatial statistics literature (Littell et al., 1996, 9.3).
For one-dimensional position vectors, the exponential spatial correlation
structure is equivalent to the CAR(1) structure (5.27). This is easily veried
by dening = exp(1/) and noting the correlation function associated
with the exponential structure is expressed as h(s, ) = s . The exponential
correlation model can be regarded as a multivariate generalization of the
the CAR(1) model.
Correlation functions for the structures in Table 5.2 may be obtained
using the relation h(s, ) = 1 (s, ). A nugget eect c0 may be added to
any of the variogram models, using
+
c0 + (1 c0 ) (s, ), s > 0,
nugg (s, co , ) =
0,
s = 0.
Figure 5.9 displays plots of the semivariograms models in Table 5.2,
corresponding to a range of = 1 and a nugget eect of c0 = 0.1 The
semivariograms increase monotonically with distance and vary between 0
and 1, corresponding to non-negative correlation functions that decrease
monotonically with distance. All of the spatial correlation models in Table 5.2 are implemented in the nlme library as corStruct classes, described
in the next section.
5.3.3 Correlation Structures in nlme: The corStruct Classes

The nlme library provides a set of classes of correlation structures, the
corStruct classes, which are used to specify within-group correlation models in either the extended linear mixed-eects model (5.1), or the extended
linear model (5.5). Table 5.3 lists the standard corStruct classes in the nlme
library. The corStruct constructors have the same name as their corresponding classes.
The two main arguments to most of the corStruct constructors are value
and form. The rst species the values of the correlation parameters and

0.0
Rational quadratic
0.5
1.0
1.5
2.0
2.5
233
3.0
Spherical
1.0
0.8
0.6
Semivariogram
0.4
0.2
0.0
Exponential
Gaussian
Linear
1.0
0.8
0.6
0.4
0.2
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Distance
FIGURE 5.9. Plots of semivariogram versus distance for the isotropic spatial
correlation models in Table 5.2 with range = 1 and nugget eect = 0.1.
the second is a one-sided formula specifying the position vector and, optionally, a grouping variable for the dataobservations in dierent groups
are assumed independent. For example, to specify age as a position variable
and to have Subject dening the grouping of the data, we use
form = ~ age | Subject
A two-dimensional position vector with coordinates x and y is specied

with
form = ~ x + y
The argument fixed, available in all corStruct constructors, may be used

to specify xed correlation structures, which coecients are not allowed
to change during the numerical optimization in the modeling functions. If
fixed = TRUE, the coecients in the structure are xed. Default is fixed =
FALSE.
Several methods are available for each corStruct class, including initialize, which initializes position vectors and grouping variables, and corMatrix,
which extracts the within-group correlation matrices. We now describe and
illustrate the standard corStruct classes.
corCompSymm
This class implements the compound symmetry correlation structure (5.23.
The argument value is used to initialize the intraclass correlation coefcient, assuming a default value of 0. Because the compound symmetry
234
TABLE 5.3. Standard corStruct classes.

corCompSymm
corSymm
corAR1
corCAR1
corARMA
corExp
corGaus
corLin
corRatio
corSpher
compound symmetry
general
autoregressive of order 1
continuous-time AR(1)
autoregressive-moving average
exponential
Gaussian
linear
rational quadratic
spherical
correlation model does not depend on the position of the observation, but
just on the group to which it belongs, the form argument is used only to
specify the grouping structure. For example,
> cs1CompSymm <- corCompSymm( value = 0.3, form = ~ 1 | Subject )
species a compound symmetry structure with intraclass correlation of 0.3

and grouping dened by Subject. The variable used on the left hand side
of the | operator in form is ignored, so
> cs2CompSymm <- corCompSymm( value = 0.3, form = ~ age | Subject )
gives a corCompSymm object identical to cs1CompSymm. By default, form = ~1,

implying that all observations belong to the same group.
The initialize method is used to initialize the grouping factor and the
correlation matrices per group. It takes an argument data, naming a data
frame in which to evaluate the variables in form.
> cs1CompSymm <- initialize( cs1CompSymm, data = Orthodont )
> corMatrix( cs1CompSymm )
$M01:
[,1] [,2] [,3] [,4]
[1,] 1.0 0.3 0.3 0.3
[2,] 0.3 1.0 0.3 0.3
[3,] 0.3 0.3 1.0 0.3
[4,] 0.3 0.3 0.3 1.0
. . .
Typically, initialize is only called from within the modeling function using
the corStruct object.
corSymm
This class implements the general correlation structure (5.24). The argument value is used to initialize the correlation parameters, being given as
235
a numeric vector with the lower diagonal elements of the the largest correlation matrix represented by the corSymm object stacked columnwise. For
example, to represent the correlation matrix
1.0 0.2 0.1 0.1

0.2 1.0 0.0 0.2
(5.32)
0.1 0.0 1.0 0.0
0.1 0.2 0.0 1.0
we use
value = c( 0.2, 0.1, -0.1, 0, 0.2, 0 )
The correlations specied in value must dene a positive-denite correlation matrix. By default, value = numeric(0), which leads to initial values
of 0 being assigned to all correlations in the initialize method.
The argument form species a one-sided formula with the position variable and, optionally, a grouping variable. The position variable denes the
indices of the correlation parameters for each observation and must evaluate to an integer vector, with nonrepeated values per group, such that
its unique values, when sorted, form a sequence of consecutive integers. By
default, the position variable in form is 1, in which case the order of the
observations within the group is used to index the correlation parameters.
For example, to specify a general correlation correlation structure with
initial correlation matrix as in (5.32), observation order within the group
as the position variable, and grouping variable Subject, we use
> cs1Symm <- corSymm( value = c(0.2, 0.1, -0.1, 0, 0.2, 0),
+
form = ~ 1 | Subject )
> cs1Symm <- initialize( cs1Symm, data = Orthodont )
> corMatrix( cs1Symm )
$M01:
[,1] [,2] [,3] [,4]
[1,] 1.0 0.2 0.1 -0.1
[2,] 0.2 1.0 0.0 0.2
[3,] 0.1 0.0 1.0 0.0
[4,] -0.1 0.2 0.0 1.0
corAR1
This class implements an autoregressive correlation structure of order 1, for
integer position vectors. The argument value initializes the single correlation parameter , which takes values between 1 and 1, and, by default, is
set to 0. The argument form is a one-sided formula specifying the position
variable and, optionally, a grouping variable. The position variable must
evaluate to an integer vector, with nonrepeated values per group, but its
values are not required to be consecutive, so that missing time points are
naturally accommodated. By default, form = ~1, implying that the order
of the observations within the group be used as the position variable.
236
For example, to specify an AR(1) correlation structure with = 0.8,

position variable given by observation order within-group, and grouping
variable Subject, we use
> cs1AR1 <- corAR1( 0.8, form = ~ 1 | Subject )
> cs1AR1 <- initialize( cs1AR1, data = Orthodont )
> corMatrix( cs1AR1 )
$M01:
[,1] [,2] [,3] [,4]
[1,] 1.000 0.80 0.64 0.512
[2,] 0.800 1.00 0.80 0.640
[3,] 0.640 0.80 1.00 0.800
[4,] 0.512 0.64 0.80 1.000
As described in 5.3.1, the AR(1) model is equivalent to an ARMA(1, 0)

model, so that the corARMA class can also be used to represent an corAR1
object. However, the corAR1 methods are designed to take advantage of
the particular structure of the AR(1) model, and are substantially more
ecient than the corresponding corARMA methods.
corCAR1
This class implements the continuous time AR(1) structure (5.27). Its arguments are dened as in corAR1, but the position variable can be any
continuous variable with non-repeated values per group and the correlation parameter can only take positive values.
corARMA
This corStruct class is used to specify autoregressivemoving average models
for the within-group errors. The argument value species the values of
the autoregressive and moving average parameters. If both autocorrelation
and moving average parameters are present, the former should precede
the latter in value. By default, all correlation parameters are set to 0,
corresponding to uncorrelated within-group errors. The form argument is a
one-sided formula dening the position variable and, optionally, a grouping
variable. The position variable must evaluate to an integer vector, with
non-repeated elements per group. Its values do not need to be consecutive,
so that missing time points are allowed. By default, form = ~1, implying
that the order of the observations within the group be used as the position
variable.
Two additional arguments, p and q, are used, respectively, to specify
the order of the autoregressive model and the order of the moving average
model. Using p > 0 and q = 0 species an AR(p) model, while p = 0 and
q > 0 species an MA(q) model. By default, p = 0 and q = 0.
For example, to specify an MA(1) model with parameter = 0.4, position variable given by the observation order within the group, and groups
dened by Subject, we use

> cs1ARMA <> cs1ARMA <> corMatrix(
$M01:
[,1]
[1,] 1.00000
[2,] 0.34483
[3,] 0.00000
[4,] 0.00000
237
corARMA( 0.4, form = ~ 1 | Subject, q = 1 )

initialize( cs1ARMA, data = Orthodont )
cs1ARMA )
[,2]
0.34483
1.00000
0.34483
0.00000
[,3]
0.00000
0.34483
1.00000
0.34483
[,4]
0.00000
0.00000
0.34483
1.00000
An ARMA(1, 1) model with parameters = 0.8 and = 0.4 is specied

with
> cs2ARMA <- corARMA( c(0.8, 0.4), form = ~ 1 | Subject, p=1, q=1 )
> cs2ARMA <- initialize( cs2ARMA, data = Orthodont )
> corMatrix( cs2ARMA )
$M01:
[,1] [,2] [,3]
[,4]
[1,] 1.0000 0.880 0.704 0.5632
[2,] 0.8800 1.000 0.880 0.7040
[3,] 0.7040 0.880 1.000 0.8800
[4,] 0.5632 0.704 0.880 1.0000
Spatial corStruct Classes

The corStruct classes representing spatial correlation structures are corExp,
corGaus, corLin, corRatio, and corSpher. As the corresponding constructors
all have the same syntax, we will show examples for corExp only.
The argument value is used to specify values for the range and the
nugget eect c0 , in this order. The range only takes positive values and
the nugget eect can only vary between 0 and 1. By default, value =
numeric(0), in which case the range is initialized to 90% of the minimum
between-pairs distance and the nugget eect is initialized to 0.1.
The argument form is a one-sided formula specifying a position vector
and, optionally, a grouping variable. The coordinates of the position vector must be numeric variables, but are otherwise unrestricted. By default,
form = ~1, translating into a one-dimensional position vector given by the
observation order within the group.
The argument nugget determines whether a nugget eect should be included in the correlation model. If TRUE, a nugget eect is included. Its
default value is FALSE, corresponding to no nugget eect in the model. The
argument metric is a character string specifying a metric to be used for calculating the between-pairs distances. Possible values include "euclidean",
"maximum", and "manhattan", corresponding to the metrics described in
5.3.2.
To illustrate the use of the spatial corStruct classes, we consider an articial data frame spatDat, with columns x and y.
238
> spatDat
x
y
1 0.00 0.00
2 0.25 0.25
3 0.50 0.50
4 0.75 0.75
5 1.00 1.00
An exponential spatial correlation structure based on the Euclidean distance between x and y, with range equal to 1, and no nugget eect is
constructed and initialized with
> cs1Exp <- corExp( 1, form = ~ x + y )
> cs1Exp <- initialize( cs1Exp, spatDat )
> corMatrix( cs1Exp )
[,1]
[,2]
[,3]
[,4]
[,5]
[1,] 1.00000 0.70219 0.49307 0.34623 0.24312
[2,] 0.70219 1.00000 0.70219 0.49307 0.34623
[3,] 0.49307 0.70219 1.00000 0.70219 0.49307
[4,] 0.34623 0.49307 0.70219 1.00000 0.70219
[5,] 0.24312 0.34623 0.49307 0.70219 1.00000
To calculate the distances in the Manhattan (L1 ) metric, we use

> cs2Exp <- corExp( 1, form = ~ x + y, metric = "man" )
[,1]
[,2]
[,3]
[,4]
[,5]
[1,] 1.00000 0.60653 0.36788 0.22313 0.13534
[2,] 0.60653 1.00000 0.60653 0.36788 0.22313
[3,] 0.36788 0.60653 1.00000 0.60653 0.36788
[4,] 0.22313 0.36788 0.60653 1.00000 0.60653
[5,] 0.13534 0.22313 0.36788 0.60653 1.00000
Note that because partial matches are used on the value of the metric
argument, we only gave the rst three characters of "manhattan" in the
call.
A nugget eect of 0.2 is added to the correlation structure using
> cs3Exp <- corExp( c(1, 0.2), form = ~ x + y, nugget = T )
[,1]
[,2]
[,3]
[,4]
[,5]
[1,] 1.00000 0.56175 0.39445 0.27698 0.19449
[2,] 0.56175 1.00000 0.56175 0.39445 0.27698
[3,] 0.39445 0.56175 1.00000 0.56175 0.39445
[4,] 0.27698 0.39445 0.56175 1.00000 0.56175
[5,] 0.19449 0.27698 0.39445 0.56175 1.00000
New corStruct classes, representing user-dened correlation structures,

can be added to the set of standard classes in Table 5.3 and used with
239
the modeling functions in the nlme library. For this, one must specify a
constructor function, generally with the same name as the class, and, at a
minimum, methods for the functions coef, corMatrix, and initialize. The
corAR1 constructor and methods can serve as templates for these.
5.3.4 Using Correlation Structures with lme

Correlation structures are specied in lme through the correlation argument. By default, correlation = NULL, corresponding to uncorrelated
within-group errors. Correlation structures are specied as corStruct objects, created using the standard constructors described in 5.3.3, or a
user-dened corStruct constructor. In this section, we describe the use of
correlation models in lme through the analysis of two examples of grouped
data with correlated within-group errors.
When assessing the adequacy of a correlation model, it is often useful to consider diagnostic plots of the normalized residuals, dened as
1/2 )T (y i y
i denotes the estimated variance
i ), where
1 (
2
ri =
i
covariance matrix for the i within-group errors. If the within-group variance
covariance model is correct, the normalized residuals should be approximately distributed as independent N (0, I) random vectors.
Counts of Ovarian Follicles
Pierson and Ginther (1987) report on a study of the number of ovarian follicles larger than 10 mm in diameter detected in eleven dierent
mares at several times in their estrus cycles. The data were recorded daily
from three days before ovulation until three days after the next ovulation.
The measurement times for each mare are scaled so that the ovulations
for each mare occur at times 0 and 1. These data are also described in
Appendix A.18 and are included in the nlme library as the groupedData
object Ovary.
The plots of the number of follicles versus time per mare, shown in Figure 5.10, suggest a periodic behavior for the number of follicles over time.
Preliminary analyses indicate that the following linear mixed-eects model
provides a reasonable representation for the number of follicles yij for the
ith mare at time tij .
yij = (0 + b0i ) + (1 + b1i ) sin (2tij ) + 2 cos (2tij ) + ij ,

b0i
N 0, diag 02 , 12 , ij N 0, 2 ,
bi =
b1i
(5.33)
where 0 , 1 , and 2 are the xed eects, bi is the random eects vector,
assumed independent for dierent mares, and ij is the within-group error,
assumed independent for dierent i, j and independent of the random effects. The random eects b0i and b1i are assumed to be independent with
variances 02 and 12 , respectively.
240
Number of ovarian follicles > 10 mm. diameter
25
20
15
10
5
25
20
15
10
5
25
20
15
10
5
-0.2
0.0
0.2
0.4
25
20
15
10
5
10
11
0.6
0.8
1.0
25
20
15
10
5
25
20
15
10
5
1.2
Time in estrus cycle

FIGURE 5.10. Number of ovarian follicles greater than 10 mm in diameter detected in mares at various times in their estrus cycles. The times have been scaled
so the ovulations occur at times 0 and 1.
We t the linear mixed-eects model (5.33) with

> fm1Ovar.lme <- lme( follicles ~ sin(2*pi*Time) + cos(2*pi*Time),
+
data = Ovary, random = pdDiag(~sin(2*pi*Time)) )
> fm1Ovar.lme
Data: Ovary
Fixed: follicles ~ sin(2 * pi * Time) + cos(2 * pi * Time)
(Intercept) sin(2 * pi * Time) cos(2 * pi * Time)
12.182
-3.2985
-0.86237
Random effects:
Formula: ~ sin(2 * pi * Time) | Mare
Structure: Diagonal
(Intercept) sin(2 * pi * Time) Residual
StdDev:
3.0521
2.0793
3.1129
241
The observations in the Ovary data were collected at equally spaced calendar times. When the calendar time was converted to the ovulation cycle scale, the intervals between observations remained very similar, but no
longer identical. Therefore, when considered in the scale of the within-group
observation order, the Ovary data provides an example of time-series data.
We use it here to illustrate the modeling of serial correlation structures in
lme.
The ACF method for the lme class obtains the empirical autocorrelation
function (5.22) from the residuals of an lme object.
> ACF(
lag
1
0
2
1
3
2
4
3
5
4
6
5
7
6
8
7
9
8
10
9
11 10
12 11
13 12
14 13
15 14
fm1Ovar.lme )
ACF
1.000000
0.379480
0.179722
0.035693
0.059779
0.002097
0.064327
0.071635
0.048578
0.027782
-0.034276
-0.077204
-0.161132
-0.196030
-0.289337
Empirical autocorrelations at larger lags tend to be less reliable, because

they are estimated with fewer residual pairs. We control the number of lags
for which to calculate the empirical autocorrelations in ACF with the argument maxLag. A plot of the empirical autocorrelation function, displayed in
Figure 5.11, is obtained with
> plot(ACF(fm1Ovar.lme,
maxLag = 10), alpha = 0.01)
# Figure 5.11
The argument alpha species the signicance level for approximate twosided critical bounds
for the autocorrelations (Box et al., 1994), given by

z(1 /2)/ N (l), with z(p) denoting the standard normal quantile of
order p and N (l) dened as in (5.22).
The empirical autocorrelations in Figure 5.11 are signicantly dierent
from 0 at the rst two lags, decrease approximately exponentially for the
rst four lags, and stabilize at nonsignicant levels for larger lags. This suggests that an AR(1) model may be suitable for the within-group correlation
and we t it with
> fm2Ovar.lme <- update( fm1Ovar.lme, correlation = corAR1() )
Data: Ovary
242
Autocorrelation
1.0
0.8
0.6
0.4
0.2
0.0
-0.2
0
10
Lag
FIGURE 5.11. Empirical autocorrelation function corresponding to the standardized residuals of the fm1Ovar.lme object.
Fixed: follicles ~ sin(2 * pi * Time) + cos(2 * pi * Time)
(Intercept) sin(2 * pi * Time) cos(2 * pi * Time)
12.188
-2.9853
-0.87776
Random effects:
Formula: ~ sin(2 * pi * Time) | Mare
Structure: Diagonal
(Intercept) sin(2 * pi * Time) Residual
StdDev:
2.8585
1.258
3.5071
Correlation Structure: AR(1)
Formula: ~ 1 | Mare
Parameter estimate(s):
Phi
0.5722
Note that no arguments need to be passed to corAR1 in this case, as its

default formula ~1 species the position variable as the within-group order
of the observations, which is what is desired for the fm2Ovar.lme model.
Because the fm1Ovar.lme model is nested within the fm2Ovar.lme model
(corresponding to = 0) we can compare them using a likelihood ratio
test.
> anova( fm1Ovar.lme, fm2Ovar.lme )
Model df
AIC
BIC logLik
fm1Ovar.lme
1 6 1638.1 1660.4 -813.04
fm2Ovar.lme
2 7 1563.4 1589.5 -774.72 1 vs 2 76.634 <.0001
The very signicant p-value for the likelihood ratio test indicates that the
AR(1) provides a substantially better t of the data than the independent errors model (5.33), suggesting that within-group serial correlation is
243
present in Ovary. We can assess the precision of the correlation parameter

estimate in fm2Ovar.lme with the intervals method.
> intervals( fm2Ovar.lme )
. . .
Correlation structure:
lower
est.
upper
Phi 0.36607 0.5722 0.72481
. . .
Consistently with the likelihood ratio test results, the condence interval
on indicates that it is signicantly dierent from 0.
The autocorrelation pattern in Figure 5.11 is also consistent with that of
an MA(2) model, in which only the rst two lags have nonzero correlations.
We t this model with
> fm3Ovar.lme <- update(fm1Ovar.lme, correlation = corARMA(q = 2))
> fm3Ovar.lme
. . .
Correlation Structure: ARMA(0,2)
Formula: ~ 1 | Mare
Theta1 Theta2
0.47524 0.25701
. . .
The AR(1) and MA(2) models are not nested and, therefore, cannot be
compared through a likelihood ratio test. They can, however, be compared
via their information criterion statistics.
> anova( fm2Ovar.lme, fm3Ovar.lme, test = F )
Model df
AIC
BIC logLik
fm2Ovar.lme
1 7 1563.4 1589.5 -774.72
fm3Ovar.lme
2 8 1571.2 1601.0 -777.62
Even though it has one fewer parameter than the MA(2) model, the AR(1)
model is associated with a larger log-restricted-likelihood, which translates
into smaller AIC and BIC, making it the preferred model of the two.
Because the xed- and random-eects models in (5.33) use a continuous
time scale, we investigate if a continuous time AR(1) model would provide
a better representation of the within-group correlation, using the corCAR1
class.
> fm4Ovar.lme <- update( fm1Ovar.lme,
+
correlation = corCAR1(form = ~Time) )
> anova( fm2Ovar.lme, fm4Ovar.lme, test = F )
Model df
AIC
BIC logLik
fm2Ovar.lme
1 7 1563.4 1589.5 -774.72
fm4Ovar.lme
2 7 1565.5 1591.6 -775.77
244
No grouping variable needs to be specied in the call to corCAR1, as the

innermost grouping variable in the lme object is used by default. As indicated by the AIC and the BIC in the anova output, the AR(1) model
provides a better representation of the within-group correlation than its
continuous-time version.
An intermediate model between the AR(1) and the MA(2) models is
the ARMA(1, 1) model, which has an exponentially decaying autocorrelation function for lags 2, but allows more exibility in the rst autocorrelation. We t it with
> fm5Ovar.lme <- update(fm1Ovar.lme, corr = corARMA(p = 1, q = 1))
> fm5Ovar.lme
. . .
Correlation Structure: ARMA(1,1)
Formula: ~ 1 | Mare
Phi1
Theta1
0.78716 -0.27957
. . .
The AR(1) model is nested within the ARMA(1, 1) model (corresponding

to 1 = 0) and we can use anova to compare the two ts through a likelihood
ratio test.
> anova( fm2Ovar.lme, fm5Ovar.lme )
Model df
AIC
BIC logLik
fm2Ovar.lme
1 7 1563.4 1589.5 -774.72
fm5Ovar.lme
2 8 1559.9 1589.7 -771.95 1 vs 2 5.5537 0.0184
The low p-value for the likelihood ratio test indicates that the ARMA(1, 1)
model provides a better t of the data.
We can assess the adequacy of the ARMA(1, 1) model using the empirical
autocorrelation function of the normalized residuals.
> plot( ACF(fm5Ovar.lme, maxLag = 10, resType = "n"),
+
alpha = 0.01 )
# Figure 5.12
No signicant autocorrelations are observed in Figure 5.12, indicating that

the normalized residuals behave like uncorrelated noise, as expected under
the appropriate correlation model.
Body Weight Growth in Rats
We revisit the BodyWeight example of 5.2.2 to illustrate the use of corStruct
classes in lme in combination with variance functions. As described in 5.2.2,
the observations in the BodyWeight data are not equally spaced in time, as
an extra observation is taken at 44 days. We use the spatial correlation
corStruct classes to use t continuous-time within-group correlation models,
which naturally accommodate the imbalance in the data.
245
Autocorrelation
1.0
0.8
0.6
0.4
0.2
0.0
-0.2
0
10
Lag
FIGURE 5.12. Empirical autocorrelation function corresponding to the normalized residuals of the fm5Ovar.lme object.
The Variogram method for the lme class estimates the sample semivariogram from the residuals of the lme object. The arguments resType
and robust control, respectively, what type of residuals should be used
("pearson" or "response") and whether the robust algorithm (5.31) or the
classical algorithm (5.30) should be used to estimate the semivariogram.
The defaults are resType = "pearson" and robust = FALSE, so that classical
estimates of the semivariogram are obtained from the standardized residuals. The argument form is a one-sided formula specifying the position vector
to be used for the semivariogram calculations.
> Variogram( fm2BW.lme, form = ~ Time )
variog dist n.pairs
1 0.34508
1
16
2 0.99328
6
16
3 0.76201
7
144
4 0.68496
8
16
5 0.68190
13
16
6 0.95118
14
128
7 0.89959
15
16
8 1.69458
20
16
9 1.12512
21
112
10 1.08820
22
16
11 0.89693
28
96
12 0.93230
29
16
13 0.85144
35
80
14 0.75448
36
16
15 1.08220
42
64
16 1.56652
43
16
17 0.64378
49
48
18 0.67350
56
32
19 0.58663
63
16
246
Semivariogram
1.5
1.0
0.5
0.0
0
10
20
30
40
Distance
FIGURE 5.13. Sample semivariogram estimates corresponding to the standardized residuals of the fm2BW.lme object. A loess smoother is added to the plot to
enhance the visualization of patterns in the semivariogram.
The columns in the data frame returned by Variogram represent, respectively, the sample semivariogram, the distance, and the number of residual
pairs used in the estimation. Because of the imbalance in the time measurements, the number of residual pairs used at each distance varies considerably, making some semivariogram estimates more reliable than others. In
general, the number of residual pairs used in the semivariogram estimation
decreases with distance, making the values at large distances unreliable.
We can control the maximum distance for which semivariogram estimates
should be calculated using the argument maxDist.
A graphical representation of the sample semivariogram is obtained with
the plot method for class Variogram.
> plot( Variogram(fm2BW.lme, form = ~ Time,
+
maxDist = 42) )
# Figure 5.13
The resulting plot, shown in Figure 5.13, includes a loess smoother (Cleveland
et al., 1992) to enhance the visualization of semivariogram patterns. The
semivariogram seems to increase with distance up to 20 days and then stabilizes around 1. We initially use an exponential spatial correlation model
for the within-group errors, tting it with
> fm3BW.lme <- update( fm2BW.lme, corr = corExp(form = ~ Time) )
> fm3BW.lme
. . .
Correlation Structure: Exponential spatial correlation
247

range
4.8862
Variance function:
power
0.59436
. . .
Note that the model corresponding to fm3BW.lme includes both a variance

function and a correlation structure. We assess the variability in the spatial
correlation parameter estimate with the intervals method.
> intervals( fm3BW.lme )
. . .
lower
est. upper
range 1.852 4.8862 12.891
. . .
The condence intervals is bounded away from zero, suggesting that the
spatial correlation model produced a signicantly better t. We can also
test this using the anova method.
Model df
AIC
BIC logLik
fm2BW.lme
1 11 1163.9 1198.4 -570.96
fm3BW.lme
2 12 1145.1 1182.8 -560.57 1 vs 2 20.781 <.0001
The likelihood ratio test also indicates that the corExp model ts the data
signicantly better than the independent errors model corresponding to
fm2BW.lme.
The semivariogram plot in Figure 5.13 gives some indication that a
nugget eect may be present in the data. We can test it with
> fm4BW.lme <- update( fm3BW.lme,
+
corr = corExp(form = ~ Time, nugget = T) )
Model df
AIC
BIC logLik
Test
L.Ratio
fm3BW.lme
1 12 1145.1 1182.8 -560.57
fm4BW.lme
2 13 1147.1 1187.9 -560.57 1 vs 2 0.00043111
p-value
fm3BW.lme
fm4BW.lme 0.9834
The nearly identical log-likelihood values indicate that a nugget eect is,
in fact, not needed.
248
Semivariogram
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0
10
20
30
40
Distance
FIGURE 5.14. Sample semivariogram estimates corresponding to the standardized residuals of the fm2BW.lme object. The tted semivariogram corresponding
to fm3BW.lme is added to plot.
When an lme object includes a spatial corStruct object, we can further

assess the adequacy of the correlation model with the plot method. In this
case, instead of a loess smoother, the tted semivariogram corresponding
to the corStruct object is displayed in the plot, along with the sample
variogram estimates.
> plot( Variogram(fm3BW.lme, form = ~ Time,
+
maxDist = 42) )
# Figure 5.14
The tted semivariogram agrees reasonably well with the sample variogram
estimates.
We can also assess the adequacy of the exponential spatial correlation
model by investigating the sample semivariogram for the normalized residuals.
> plot( Variogram(fm3BW.lme, form = ~ Time, maxDist = 42,
+
resType = "n", robust = T) )
# Figure 5.15
The robust semivariogram estimator is used to reduce the inuence of an

outlying value at distance 1 on the loess smoother. The sample semivariogram estimates in Figure 5.15 appear to vary randomly around the y = 1
line, suggesting that the normalized residuals are approximately uncorrelated and, hence, the corExp model is adequate.
We may compare the corExp model to other spatial correlation models,
using the update method and the anova method. As the models are not
5.4 Fitting Extended Linear Models with gls
249
Semivariogram
1.5
1.0
0.5
0.0
0
10
20
30
40
Distance
FIGURE 5.15. Sample semivariogram estimates corresponding to the normalized

residuals of the fm3BW.lme object. A loess smoother is added to the plot to
nested, they have to be compared based on the information criteria AIC

and BIC.
>
>
>
>
>
fm5BW.lme <- update( fm3BW.lme, corr = corRatio(form = ~ Time) )

fm6BW.lme <- update( fm3BW.lme, corr = corSpher(form = ~ Time) )
fm7BW.lme <- update( fm3BW.lme, corr = corLin(form = ~ Time) )
fm8BW.lme <- update( fm3BW.lme, corr = corGaus(form = ~ Time) )
anova( fm3BW.lme, fm5BW.lme, fm6BW.lme, fm7BW.lme, fm8BW.lme )
Model df
AIC
BIC logLik
fm3BW.lme
1 12 1145.1 1182.8 -560.57
fm5BW.lme
2 12 1148.8 1186.4 -562.38
fm6BW.lme
3 12 1150.8 1188.4 -563.39
fm7BW.lme
4 12 1150.8 1188.4 -563.39
fm8BW.lme
5 12 1150.8 1188.4 -563.39
The corExp t has the smallest AIC and BIC and seems the most adequate
within-group correlation model for the BodyWeight data, among the spatial
correlation models considered.

The general formulation of the extended linear model, as well as the estimation methods used to t it, have been described in 5.1.2. In this section,
250
TABLE 5.4. Main gls methods.

ACF
anova
augPred
coef
fitted
intervals
logLik
plot
predict
print
qqnorm
resid
summary
update
Variogram
empirical autocorrelation function of residuals

likelihood ratio or conditional tests
estimated coecients for expected response model
tted values
predicted values
residuals
update the gls t
semivariogram of residuals
we concentrate on the capabilities available in the nlme library for tting

such models.
The gls function is used to t the extended linear model (5.5), using
either maximum likelihood, or restricted maximum likelihood. It can be
viewed as an lme function without the argument random. Several arguments
are available in gls, but typical calls are of the form
gls( model, data, correlation )
# correlated errors
gls( model, data, weights )
# heteroscedastic errors
gls( model, data, correlation, weights ) # both
The rst argument, model, is a two-sided linear formula specifying the model
for the expected value of the response. Correlation and weights are used,
as in lme, to dene, respectively, the correlation model and the variance
function model for the error term. Data species a data frame in which the
variables named in model, correlation, and weights can be evaluated.
The tted object returned by gls inherits from class gls, for which several
methods are available to display, plot, update, and further explore the
estimation results. Table 5.4 lists the most important methods for class gls.
The use of the gls function and its associated methods is described and
illustrated through the examples in the next sections.
The Orthodont data were analyzed in 4.2 and 4.3 using a linear mixedeects model. We describe here an alternative analysis based on the extended linear model (5.5).
251
Because of the small number of observations per subject, it is feasible for

these data to t a linear model with a general variancecovariance structure
for the errors. The corresponding extended linear model for the ith subject
orthodontic distance at age j, i = 1, . . . , 27 and j = 1, . . . , 4, is written as

distanceij = 0 + 1 Sex + 2 agej 11 + 3 agej 11 Sex + ij ,
2
i1
1 12 13 14
i2
12 22 23 24
N (0, i ) ,
i =
=
i
i3
13 23 32 34 , (5.34)
i4
14 24 34 42
with Sex representing a binary variable taking values 1 for boys and 1
for girls. The parameters 1 and 3 represent, respectively, the intercept
and slope gender eects. The variancecovariance matrix i is assumed the
same for all subjects.
We t (5.34) with gls using a combination of the corSymm correlation
class and the varIdent variance function class.
> fm1Orth.gls <- gls( distance ~ Sex * I(age - 11), Orthodont,
+
correlation = corSymm(form = ~ 1 | Subject),
+
weights = varIdent(form = ~ 1 | age) )
In this case, because Orthodont is a groupedData object with grouping variable Subject, the argument form could be omitted in the call to corSymm.
The print method gives some basic information about the t.
> fm1Orth.gls
Generalized least squares fit by REML
Model: distance ~ Sex * I(age - 11)
Data: Orthodont
Coefficients:
(Intercept)
Sex I(age - 11) Sex:I(age - 11)
23.801 -1.136
0.6516
-0.17524
Correlation Structure: General
Formula: ~ 1 | age
Correlation:
1
2
3
2 0.568
3 0.659 0.581
4 0.522 0.725 0.740
Variance function:
Structure: Different standard deviations per stratum
Formula: ~ 1 | age
252
8
10
12
14
1 0.8792 1.0747 0.95872
The correlation estimates are similar, suggesting that a compound symmetry structure may be a suitable correlation model. We explore this further
with the intervals method.
> intervals( fm1Orth.gls )
Coefficients:
(Intercept)
Sex
I(age - 11)
Sex:I(age - 11)
lower
23.06672
-1.87071
0.52392
-0.30292
lower
est.
cor(1,2) 0.098855 0.56841
cor(1,3) 0.242122 0.65878
cor(1,4) 0.021146 0.52222
cor(2,3) 0.114219 0.58063
cor(2,4) 0.343127 0.72510
cor(3,4) 0.382248 0.73967
est.
23.80139
-1.13605
0.65160
-0.17524
upper
24.536055
-0.401380
0.779289
-0.047552
upper
0.83094
0.87030
0.81361
0.83731
0.90128
0.90457
Variance function:
lower
est. upper
10 0.55728 0.87920 1.3871
12 0.71758 1.07468 1.6095
14 0.61253 0.95872 1.5005
Residual standard error:
lower est. upper
1.5985 2.329 3.3933
All condence intervals for the correlation parameters overlap, corroborating the compound symmetry assumption. We can test it formally by
updating the tted object and using the anova method.
> fm2Orth.gls <+
update(fm1Orth.gls, corr = corCompSymm(form = ~ 1 | Subject))
> anova( fm1Orth.gls, fm2Orth.gls )
Model df
AIC
BIC logLik
fm1Orth.gls
1 14 455.32 492.34 -213.66
fm2Orth.gls
2 9 452.74 476.54 -217.37 1 vs 2 7.4256 0.1909
253
The large p-value for the likelihood ratio statistics reported in the anova
output conrms the compound symmetry model.
The condence intervals for the variance function parameters corresponding to fm2Orth.gls,
> intervals( fm2Orth.gls )
. . .
Variance function:
lower
est. upper
10 0.56377 0.86241 1.3193
12 0.68132 1.03402 1.5693
14 0.60395 0.92045 1.4028
. . .
all include the value 1, suggesting that the variability does not change with
age. The high p-value for the associated likelihood ratio test conrms this
assumption.
> fm3Orth.gls <- update( fm2Orth.gls, weights = NULL )
Model df
AIC
BIC logLik
fm2Orth.gls
1 9 452.74 476.54 -217.37
fm3Orth.gls
2 6 448.53 464.40 -218.26 1 vs 2 1.7849 0.6182
As with other modeling functions, the plot method is used to assess the
assumptions in the model. Its syntax is identical to the plot method for
class lme. For example, to examine if the error variance is the same for boys
and girls we may examine the plot of the normalized residuals versus age
by gender, obtained with
> plot( fm3Orth.gls, resid(., type = "n") ~ age | Sex ) # Fig. 5.16
and displayed in Figure 5.16. It is clear that there is more variability among
boys than girls, which we can represent in the model with the varIdent
variance function class.
> fm4Orth.gls <- update( fm3Orth.gls,
+
weights = varIdent(form = ~ 1 | Sex) )
Model df
AIC
BIC logLik
fm3Orth.gls
1 6 448.53 464.40 -218.26
fm4Orth.gls
2 7 438.96 457.47 -212.48 1 vs 2 11.569
7e-04
As expected, the likelihood ratio test gives strong evidence in favor of the
heteroscedastic model.
The qqnorm method is used to assess the assumption of normality for the
errors. Its syntax is identical to the corresponding lme method. The normal
plots of the normalized residuals by gender, obtained with
> qqnorm( fm4Orth.gls, ~resid(., type = "n") )
# Figure 5.17
254

8
Male
10
11
12
13
14
Female
Normalized residuals
4
3
2
1
0
-1
-2
8
10
11
12
13
14
Age (yr)
FIGURE 5.16. Scatter plots of normalized residuals versus age by gender for the
fm3Orth.gls tted object.
and displayed in Figure 5.17, do not indicate serious departures from normality and conrm that the variance function model included in fm4Orth.gls
was successful in accommodating the error heteroscedasticity.
It is interesting, at this point, to compare the gls t corresponding to
fm4Orth.gls to the lme t corresponding to fm3Orth.lme, obtained in 4.3.1.
Because the corresponding models are not nested, a likelihood ratio test
is nonsensical. However, the information criteria can be compared, as the
xed eects models are identical for the two ts. The anova method can be
used to compare gls and lme objects.
> anova( fm3Orth.lme, fm4Orth.gls, test = F )
Model df
AIC
BIC logLik
fm3Orth.lme
1 9 432.30 456.09 -207.15
fm4Orth.gls
2 7 438.96 457.47 -212.48
The lme t has the smallest AIC and BIC and, therefore, seems to give a
better representation of the Orthodont data.
The choice between an lme model and a gls model should take into account more than just information criteria and likelihood tests. A mixedeects model has a hierarchical structure which, in many applications, provides a more intuitive way of accounting for within-group dependency than
the direct modeling of the marginal variancecovariance structure of the response in the gls approach. Furthermore, the mixed-eects estimation gives,
as a byproduct, estimates for the random eects, which may be of interest
in themselves. The gls model focuses on marginal inference and is more
appealing when a hierarchical structure for the data is not believed to be

-2
-1
Male
255
3
Female
-1
-2
-2
-1
FIGURE 5.17. Normal plots plots of normalized residuals by gender for the
fm4Orth.gls tted object.
present, or is not relevant in the analysis, and one is more interested in

parameters associated with the error variancecovariance structure, as in
time-series analysis and spatial statistics.
The hemodialyzer data were analyzed in 5.2.2 to illustrate the use of
variance functions with linear mixed-eects models. An alternative analysis,
using the gls function, is presented here.
We initially consider a linear model with the same expected response
as the linear mixed-eects model (5.18) and independent, homoscedastic
errors.
yij = (0 + 0 Qi ) + (1 + 1 Qi ) xij + (2 + 2 Qi +) x2ij +

(3 + 3 Qi ) x3ij + (4 + 4 Qi ) x4ij + ij , ij N 0, 2 ,
(5.35)
where Qi is a binary variable taking values 1 for 200 dl/min hemodialyzers

and 1 for 300 dl/min hemodialyzers; 0 , 1 , 2 , 3 , and 4 are, respectively,
the intercept, linear, quadratic, cubic, and quartic coecients averaged over
the levels of Q; i is the blood ow eect associated with the coecient i ;
and ij is the error term.
We may t the linear model (5.35) using the gls function
> fm1Dial.gls <+
gls(rate ~(pressure + pressure^2 + pressure^3 + pressure^4)*QB,
+
Dialyzer)
256
Residuals (ml/hr)
-5
-10
0.5
1.0
1.5
2.0
2.5
3.0
FIGURE 5.18. Plot of residuals versus transmembrane pressure for the homoscedastic tted object fm1Dial.gls.
Because no variance functions and correlation structures are used, the gls
t is equivalent to an lm t, in this case.
The plot of the residuals versus transmembrane pressure, obtained with
> plot( fm1Dial.gls, resid(.) ~ pressure,
+
abline = 0 )
# Figure 5.18
and shown in Figure 5.18, displays the same pattern of heteroscedasticity observed for the within-group residuals of the lme object fm1Dial.lme,
presented in Figure 5.2.
As in the lme analysis of 5.2.2, we choose the exible variance function
class varPower to model the heteroscedasticity in the response, and test its
signicance using the anova method.
> fm2Dial.gls <- update( fm1Dial.gls,
+
> anova( fm1Dial.gls, fm2Dial.gls)
Model df
AIC
BIC logLik
fm1Dial.gls
1 11 768.10 799.65 -373.05
fm2Dial.gls
2 12 738.22 772.63 -357.11 1 vs 2 31.882 <.0001
As expected, the likelihood ratio test strongly rejects the assumption of

homoscedasticity. The plot of the standard residuals corresponding to
fm2Dial.gls versus pressure, shown in Figure 5.19, indicates that the power
variance function adequately represents the heteroscedasticity in the data.
Because the hemodialyzer ultraltration rates are measured sequentially
over time on the same subjects, the within-subject observations are likely
257
-1
-2
0.5
1.0
1.5
2.0
2.5
3.0

the heteroscedastic tted object fm2Dial.gls.
to be correlated. In 5.2.2, random eects were used to account for the

within-group correlation. We may, alternatively, use a correlation structure
to directly model the association among the within-subject errors. Because
the measurements in this example are equally spaced in time, the empirical autocorrelation function can be used to investigate within-subject
correlation. The ACF method for the gls class has a syntax similar to the
corresponding lme method, but includes a form argument which allows the
specication of a time covariate, to dene the lags between observations,
and a grouping variable, to specify a partition for the residuals.
> ACF( fm2Dial.gls, form = ~ 1 | Subject )
lag
ACF
1
0 1.000000
2
1 0.770851
3
2 0.632301
4
3 0.408306
5
4 0.200737
6
5 0.073116
7
6 0.077802
The empirical ACF values indicate that the within-group observations are
correlated, and that the correlation decreases with lag. As usual, it is more
informative to look at a plot of the empirical ACF, displayed in Figure 5.20
and obtained with
> plot( ACF( fm2Dial.gls, form = ~ 1 | Subject),
+
alpha = 0.01 )
# Figure 5.20
258
Autocorrelation
1.0
0.5
0.0
-0.5
0
Lag
FIGURE 5.20. Empirical autocorrelation function corresponding to the standardized residuals of the fm2Dial.gls object.
The ACF pattern observed in Figure 5.20 suggests that an AR(1) model
may be appropriate to describe it. The corAR1 class is used to represent it.
> fm3Dial.gls <- update( fm2Dial.gls,
+
corr = corAR1(0.771, form = ~ 1 | Subject) )
> fm3Dial.gls
Model: rate ~(pressure + pressure^2 + pressure^3 + pressure^4)*QB
Data: Dialyzer
Coefficients:
-16.818
92.334
-49.265
11.4
-1.0196
-1.5942
1.7054
2.1268
0.47972
I(pressure^4):QB
-0.22064
Phi
0.75261
Variance function:
Formula: ~ pressure
power
0.51824
259
The initial value for the AR(1) parameter is given by the lag-1 empirical
autocorrelation. The form argument is used to specify the grouping variable
Subject.
The intervals method is used to assess the variability in the estimates.
> intervals( fm3Dial.gls )
Coefficients:
(Intercept)
pressure
I(pressure^2)
I(pressure^3)
I(pressure^4)
QB
pressure:QB
I(pressure^2):QB
I(pressure^3):QB
I(pressure^4):QB
lower
est.
-18.8968 -16.81845
81.9144 92.33423
-63.1040 -49.26515
4.5648 11.39967
-2.1248 -1.01964
-4.7565 -1.59419
-13.6410
1.70544
-17.9484
2.12678
-9.3503
0.47972
-1.8021 -0.22064
upper
-14.740092
102.754073
-35.426279
18.234526
0.085557
1.568141
17.051847
22.201939
10.309700
1.360829
lower
est.
upper
Phi 0.56443 0.75261 0.86643
Variance function:
lower
est.
upper
power 0.32359 0.51824 0.71289
Residual standard error:
lower
est. upper
2.3051 3.0463 4.0259
The condence interval on the correlation parameter is bounded away

from zero, indicating that the AR(1) model provides a signicantly better
t. We can conrm it with the likelihood ratio test.
> anova( fm2Dial.gls, fm3Dial.gls )
Model df
AIC
BIC logLik
fm2Dial.gls
1 12 738.22 772.63 -357.11
fm3Dial.gls
2 13 642.67 679.95 -308.34 1 vs 2 97.546 <.0001
No signicant correlations are observed in the plot of the empirical autocorrelation function for the normalized residuals of fm3Dial.gls, displayed
in Figure 5.21, indicating that the AR(1) adequately represents the withinsubject dependence.
The gls model corresponding to fm3Dial.gls may be compared to the best
lme model for the Dialyzer data in 5.2.2, corresponding to the fm2Dial.lme
object. As the models are not nested, only the information criterion statistics can be compared.
260
Autocorrelation
1.0
0.5
0.0
-0.5
0
Lag
FIGURE 5.21. Empirical autocorrelation function for the normalized residuals of

the fm3Dial.gls object.
> anova( fm3Dial.gls, fm2Dial.lme, test = F )

Model df
AIC
BIC logLik
fm3Dial.gls
1 13 642.67 679.95 -308.34
fm2Dial.lme
2 18 655.01 706.63 -309.51
The two log-likelihoods are very similar, suggesting that the models give
equivalent representations of the data. Because the gls model has ve fewer
parameters than the lme model, its information criterion statistics take
smaller values, suggesting it is a better model. However, as pointed out
in the previous example, the choice between a gls and an lme should take
other factors in consideration, besides the information criteria.
Wheat Yield Trials
Stroup and Baenziger (1994) describe an agronomic experiment to compare
the yield of 56 dierent varieties of wheat planted in four blocks arranged
according to a randomized complete complete block design. All 56 varieties
of wheat were used in each block. The latitude and longitude of each experimental unit in the trial were also recorded. These data are described in
greater detail in Appendix A.31, being included in the nlme library as the
groupedData object Wheat2.
The plot of the wheat yields for each variety by block, shown in Figure 5.22, suggests that a block eect is present in the data. As pointed out
by Littell et al. (1996, 9.6.2), the large number of plots within each block
makes the assumption of within-block homogeneity unrealistic. A better
representation of the dependence among the experimental units may be
obtained via spatial correlation structures that use the information on their
latitude and longitude. The corresponding extended linear model for the ith
wheat variety yield in the jth block, yij , for i = 1, . . . , 56, j = 1, . . . , 4,
Block
s + # # w{
>
1
# w{
s > > #w
{+ > #
s
2
+ {+ +{
s{ + sw{+
10
#{{{
+
+{>+ ## s
>w
wssw
s{#>+
{{>ww
#>+ >
#ss {++
#> {ww
+
>{> + {w>s#
s #w+ >w
sws
#{ + s> s#{ w
# s+{+
261
+#s#+w
>s>{{ >{>s>{+ w >
w{ s#+
w
w
#+{w
#>#+s{ w
#{+
>ss
+sw >s> > >#w

>
sw
s
#{ #
#>##www >>#+w
20
30
40
Yield
FIGURE 5.22. Yields of 56 dierent varieties of wheat for each block of a randomized complete block design.
is given by
yij = i + ij ,

= N 0, 2 ,
(5.36)
where i denotes the average yield for variety i and ij denotes the error
term, assumed to be normally distributed with mean 0 and with variance
covariance matrix 2 .
To explore the structure of , we initially t the linear model (5.36) with
the errors assumed independent and homoscedastic, i.e., = I.
> fm1Wheat2 <- gls( yield ~ variety - 1, Wheat2 )
As described in 5.3.2, the sample semivariogram of the standardized residuals is the primary tool for investigating spatial correlation in the errors.
The variogram method for class gls is used to obtain the sample semivariogram for the residuals of a gls object. Its syntax is identical to that of the
Variogram method for lme objects.
> Variogram( fm1Wheat2, form = ~ latitude + longitude )
variog
dist n.pairs
1 0.36308 4.3000
1212
2 0.40696 5.6080
1273
3 0.45366 8.3863
1256
4 0.51639 9.3231
1245
5 0.57271 10.5190
1254
6 0.58427 12.7472
1285
7 0.63854 13.3929
1175
8 0.65123 14.7635
1288
9 0.73590 16.1818
1290
10 0.73797 17.3666
1187
11 0.75081 18.4567
1298
262

1.0
Semivariogram
0.8
0.6
0.4
0.2
0.0
0
10
15
20
25
30
Distance
FIGURE 5.23. Sample semivariogram estimates corresponding to the standardized residuals of the fm1Wheat2 object. A loess smoother is added to the plot to
12
13
14
15
16
17
18
19
20
0.88098
0.81019
0.86199
0.86987
0.85818
0.97145
0.98778
1.09617
1.34146
20.2428
21.6335
22.6736
24.6221
26.2427
28.4542
30.7877
34.5879
39.3641
1226
1281
1181
1272
1223
1263
1228
1263
1234
The sample semivariogram increases with distance, indicating, as expected,

that the observations are spatially correlated. The graphical representation
of the sample semivariogram produced by the plot method, displayed in
Figure 5.23, allows easier interpretation of the spatial correlation pattern.
> plot( Variogram(fm1Wheat2, form = ~ latitude + longitude,
+
maxDist = 32), xlim = c(0,32) )
# Figure 5.23
The distance is censored at 32, using the maxDist argument, to avoid

the less reliable estimates associated with distant plots. A nugget eect of
about 0.2 seems to be present and the semivariogram appears to approach
1 around distance 28.
Littell et al. (1996, 9.6.2) use the spherical correlation structure to model
the spatial correlation in these data. We can t it using the corSpher class.
Initial values for the range and the nugget eect are obtained from Figure 5.23, noting that, in corSpher, the range is the distance at which the
semivariogram rst equals 1.
> fm2Wheat2 <- update( fm1Wheat2, corr = corSpher(c(28, 0.2),
+
form = ~ latitude + longitude, nugget = T) )
263
> fm2Wheat2
Model: yield ~ variety - 1
Data: Wheat2
Coefficients:
varietyARAPAHOE varietyBRULE varietyBUCKSKIN varietyCENTURA
26.659
25.85
34.848
25.095
. . .
varietySIOUXLAND varietyTAM107 varietyTAM200 varietyVONA
25.656
22.77
18.764
24.782
Correlation Structure: Spherical spatial correlation
Formula: ~ latitude + longitude
range nugget
27.457 0.20931
An alternative model suggested by the slow increase of the semivariogram

with distance in Figure 5.23 is the rational quadratic model of 5.3.2. An
initial value for the range in this model can also be obtained from the
semivariogram plot, noting that, when distance = range the semivariogram
in the rational quadratic model is equal to (1 + nugget)/2. For a nugget
eect of 0.2, this gives an initial estimate for the range of about 12.5 (the
approximate distance in Figure 5.23 at which the semivariogram is 0.6).
The corRatio class is used to t the rational quadratic model.
> fm3Wheat2 <- update( fm1Wheat2,
+
corr = corRatio(c(12.5, 0.2),
+
form = ~ latitude + longitude, nugget = T) )
> fm3Wheat2
Model: yield ~ variety - 1
Data: Wheat2
Coefficients:
varietyARAPAHOE varietyBRULE varietyBUCKSKIN varietyCENTURA
26.546
26.284
35.037
24.867
. . .
varietySIOUXLAND varietyTAM107 varietyTAM200 varietyVONA
25.74
22.476
18.693
25.046
Correlation Structure: Rational quadratic spatial correlation
Formula: ~ latitude + longitude
264
Semivariogram
0.8
0.6
0.4
0.2
0.0
10
20
30
40
Distance
FIGURE 5.24. Sample semivariogram estimates corresponding to the normalized

residuals of the fm3Wheat2 object. A loess smoother is added to the plot to
range nugget
13.461 0.1936
> anova( fm2Wheat2, fm3Wheat2 )
Model df
AIC
BIC logLik
fm2Wheat2
1 59 1185.9 1370.2 -533.93
fm3Wheat2
2 59 1183.3 1367.6 -532.64
The smaller AIC and BIC values for the rational quadratic model indicate that it gives a better representation of the correlation in the data than
the spherical model. We can test the signicance of the spatial correlation
parameters comparing the fm3Wheat2 t to the t with independent errors
corresponding to fm1Wheat2.
> anova( fm1Wheat2, fm3Wheat2 )
Model df
AIC
BIC logLik
fm1Wheat2
1 57 1354.7 1532.8 -620.37
fm3Wheat2
2 59 1183.3 1367.6 -532.64 1 vs 2 175.46 <.0001
The large value of the likelihood ratio test statistics gives strong evidence
against the assumption of independence.
We can verify the adequacy of the corRatio model by examining the plot
of the sample semivariogram for the normalized residuals of fm3Wheat2.
> plot( Variogram(fm3Wheat2, resType = "n") )
# Figure 5.24
No patterns are observed in the plot of the sample semivariogram, suggesting that the rational quadratic model is adequate.
265
-1
-2
-3
20
25
30
35
Fitted values
FIGURE 5.25. Scatter plot of normalized residuals versus tted values for the
tted object fm3Wheat2.
The normalized residuals are also useful for investigating heteroscedasticity and departures from normality. For example, the plot of the normalized
residuals versus the tted values, displayed in Figure 5.25 and obtained
with
> plot( fm3Wheat2, resid(., type = "n") ~ fitted(.),
+
abline = 0 )
# Figure 5.25
does not indicate any heteroscedastic patterns. The normal plot of the
normalized residuals in Figure 5.26 is obtained with
> qqnorm( fm3Wheat2, ~ resid(., type = "n") )
# Figure 5.26
No signicant departures from the assumption of normality are observed

in this plot.
The anova method can be used to assess dierences between the wheat
varieties. For example, to test if there are any dierences between varieties,
we rst have to reparametrize (5.36) in terms of an intercept plus contrasts
between the varieties and then use anova.
> fm4Wheat2 <- update( fm3Wheat2, model = yield ~ variety )
> anova( fm4Wheat2 )
Denom. DF: 168
numDF F-value p-value
(Intercept)
1 30.405 <.0001
variety
55
1.851 0.0015
The small p-value of the F-test for variety indicates that there are signicant dierences between varieties. We can test specic contrasts using the
266

3
-1
-2
-3
-3
-2
-1
FIGURE 5.26. Normal plots plots of normalized residuals for the tted object
fm3Wheat2.
argument L to anova, with the original cell means parametrization. For example, to test the dierence between the rst and the third wheat varieties
we use
> anova( fm3Wheat2, L = c(-1, 0, 1) )
Denom. DF: 168
F-test for linear combination(s)
varietyARAPAHOE varietyBUCKSKIN
-1
1
numDF F-value p-value
1
1 7.6966 0.0062
The small p-value for the F-test, combined with the coecient estimates
displayed previously, indicates that the BUCKSKIN variety has signicant
higher yields than the ARAPAHOE variety. Similar analyses can be obtained
for other linear contrasts of the model coecients.
5.5 Chapter Summary

In this chapter the linear mixed-eects model of Chapters 2 and 4 is extended to include heteroscedastic, correlated within-group errors. We show
how the estimation and computational methods of Chapter 2 can be extended to this more general linear mixed-eects model. We introduce several classes of variance functions to characterize heteroscedasticity and
Exercises
267
several classes of correlation structures to represent serial and spatial correlation, and describe how variance functions and correlations structures
can be combined to exibly model the within-group variancecovariance
structure.
We illustrate, through several examples, how the lme function is used to
t the extended linear mixed-eects model and describe a suite of S classes
and methods to implement variance functions (varFunc) and correlation
structures (corStruct). Any of these classes, or others dened by users, can
be used with lme to t extended linear mixed-eects models.
An extended linear model with heteroscedastic, correlated errors is introduced and a new modeling function to t it, gls, is described. This extended
linear model can be thought of as an extended linear mixed-eects model
with no random eects, and any of the varFunc and corStruct classes available with lme can also be used with gls. Several examples are used the
illustrate the use of gls and its associated methods.
Exercises
1. The within-group heteroscedasticity observed for the fm1BW.lme t of
the BodyWeight data in 5.2.2 was modeled using the power variance
function (varPower) with the tted values as the variance covariate.
An alternative approach, which is explored in this exercise, is to allow
dierent variances for each Diet.
(a) Plot the residuals of fm1BW.lme versus Diet (use plot(fm1BW.lme,
resid(.) ~ as.integer(Diet), abline = 0)). Note that the variability for Diets 1 and 2 are similar, but the residuals for Diet 3
have larger variability.
(b) Update the fm1BW.lme t allowing dierent variances per Diet
(use weights = varIdent(form = ~1|Diet)). To get a t that can
be compared to fm1BW.lme, remember to set the contrasts parameterization to "contr.helmert". Obtain condence intervals
on the variance function coecients using intervals. Do they
agree with the conclusions from the plot of the residuals versus
Diet? Explain.
(c) Compare the t with the varIdent variance function to fm1BW.lme
using anova. Compare it also to fm2BW.lme, the t with the
varPower variance function. Which variance function model is
preferable? Why?
(d) Use the gls function described in 5.4 to t a model with a
varPower variance function in the tted values and a corCAR1
correlation structure in Time, but with no random eects. Com-
268
pare the resulting t to the fm3BW.lme t of 5.3.4 using anova.

Are the two models nested? Why?
2. A multilevel LME analysis of the Oats data was presented in 1.6 and
an equivalent single-level analysis using a block-diagonal matrix
(pdBlocked class) was discussed in 4.2.2. This exercise illustrates yet
another possible approach for analyzing split-plot experiments such
as the Oats data.
(a) Fit an LME model to the Oats data using the same xed-eects
model as in the fm4Oats t of 1.6, a single random intercept at
the Block level, and a general correlation structure (corSymm) at
the Variety within Block level (use random = ~1|Block and corr
= corSymm(form = ~1|Block/Variety)).
(b) Obtain the condence intervals on the within-group correlations
associated with the corSymm structure. Note that all intervals
overlap and the estimates are of similar magnitude. This suggests a compound symmetry correlation structure. Update the
previous t using corr = corCompSymm(form=~1|Block/Variety).
Assess the validity of the corCompSymm model using anova.
(c) Compare the t obtained in the previous item to the fm4Oats
t. Note that the log-likelihoods are nearly identical and so are
the estimated xed eects and the estimated standard error
1 ). The two models are actually identical: verify
for Blocks (
that the estimated within-group variance for the lme t with
the corCompSymm structure is equal to the sum of the estimated
Variety within Block variance (
22 ) and the estimated within2
group variance (
) in the fm4Oats t and that the estimated
correlation for the corCompSymm structure is equal to
22 /(
22 +
2 ).
(d) Use gls to t a model with corCompSymm correlation structure at
the Variety within Block level, but with no random eects and
use it to assess the signicance of the Block eect in the Oats
experiment.
3. Obtain a gls t of the Ovary data using the same xed-eects model
and correlation structure as in the fm5Ovar.lme t of 5.3.4 but with
no random eects (use corr = corARMA(form=~1|Mare, p=1, q=1) to
dene the grouping structure). Compare the t with fm5Ovar.lme using anova. Is there signicant evidence for keeping the random eects
in the model? Can you give an explanation for what you found?
4. As mentioned in 5.2, new varFunc classes representing user-dened
variance functions can be incorporated into the nlme library and used
with any of its modeling function. This exercise illustrates how a new
varFunc class can be created.
Exercises
269
We use a variance function based on a linear combination of covariates, which we denote varReg. Letting v denote a vector of variance
covariates and the variance parameters, the varReg variance model
and variance function are dened as

g (v, ) = exp v T ,
Var () = 2 exp 2v T ,
where, as in 5.2, represents the random variable whose variance
is being modeled. Note that the variance parameters are unconstrained and, for identiability, the linear model in the variance function can not have an intercept (it would be confounded with ). The
varReg class extends the varExp class dened in 5.2 by allowing more
than one variance covariate to be used. The varReg class is frequently
used in the analysis of dispersion eects in robust designs (Wolnger
and Tobias, 1998).
(a) Write a constructor for the varReg class. It should contain at
least two arguments: value with initial values for the variance
parameters (set the default to numeric(0) to indicate noninitialized structures) and form with a linear formula dening the
variance covariates. To simplify things, assume that the tted
object can not be used to dene a variance covariate for this
class, that no stratication of parameters will be available and
that no parameters can be made xed in the estimation. You
can use the varExp constructor as a template.
(b) Next you need to write an initialize method. This takes two
required arguments object, the varReg object, and data, a data
frame in which to evaluate the variance covariates. For consistency with the generic function, include a ... at the end of the
argument list. The initialize method should obtain the model
matrix corresponding to formula(object), evaluated on data,
and save it as an attribute for later calculations. As mentioned
before, the model matrix should not have an (Intercept) column; check if one is present and remove it if necessary. You
should make sure that the parameters are initialized (if no starting values are given in the constructor, initialize them to 0, corresponding to an homoscedastic variance model). Additionally, the
"logLik" and "weights" attributes of the returned object need to
be initialized. Note that the weights are simply exp(-modMat %*%
coefs), where modMat represents the model matrix for the variance covariates and coefs the initialized coecients. You can
use the initialize.varExp method as a template.
(c) The coef method for the varReg class is very simple: for consistency with other varFunc coef methods, it takes three arguments:
object (the varReg object), unconstrained, and allCoef. Because
270
is unconstrained and we are not allowing any parameters to

be xed, the method will always return as.vector(object).
(d) The coef<- method takes two arguments: object, the varReg object, and value, the new values for the coecients. This method
is also used to update the value of the "logLik" and "weights"
attributes, after the coecients in object have been updated.
You can use coef<-.varExp as a template.
(e) Write a summary.varReg method to provide a description of the
varReg variance function when it is the varReg object is printed
(usually as part of the output of some modeling function). Use
summary.varExp as an example.
(f) Test your class by tting an LME model to the BodyWeight data
with a dierent variances for each Diet (you can use weights =
varReg(form = ~Diet), because the initialize method will remove the (Intercept) column of the model matrix).
Part II
Nonlinear Mixed-Eects
Models
6
Nonlinear Mixed-Eects Models: Basic
Concepts and Motivating Examples
This chapter gives an overview of the nonlinear mixed-eects (NLME)

model, introducing its main concepts and ideas through the analysis of
real-data examples. The emphasis is on presenting the motivation for using
NLME models when analyzing grouped data, while introducing some of
the key features in the nlme library for tting and analyzing such models.
This chapter serves as an appetizer for the material covered in the last two
chapters of the book: the theoretical foundations and computational methods for NLME models described in Chapter 7 and the nonlinear modeling
facilities available in the nlme library, described in detail in Chapter 8.
6.1 LME Models vs. NLME Models

The rst and possibly most important question about NLME models is
why would one want to use them? This question, of course, also applies to
nonlinear regression models in general as does the answer: interpretability,
parsimony, and validity beyond the observed range of the data.
When choosing a regression model to describe how a response variable
varies with covariates, one always has the option of using models, such as
polynomial models, that are linear in the parameters. By increasing the
order of a polynomial model, one can get increasingly accurate approximations to the true, usually nonlinear, regression function, within the observed
range of the data. These empirical models are based only on the observed
relationship between the response and the covariates and do not include
274
6. NLME Models: Basic Concepts and MotivatingExamples
any theoretical considerations about the underlying mechanism producing

the data.
Nonlinear models, on the other hand, are often mechanistic, i.e., based on
a model for the mechanism producing the response. As a consequence, the
model parameters in a nonlinear model generally have a natural physical
interpretation. Even when derived empirically, nonlinear models usually incorporate known, theoretical characteristics of the data, such as asymptotes
and monotonicity, and in these cases, can be considered as semi-mechanistic
models. A nonlinear model generally uses fewer parameters than a competitor linear model, such as a polynomial, giving a more parsimonious
description of the data. Nonlinear models also provide more reliable predictions for the response variable outside the observed range of the data
than, say, polynomial models would.
To illustrate these dierences between linear and nonlinear models, let
us consider a simple example in which the expected height ht of a tree at
time t follows a three-parameter logistic growth model.
ht = 1 / {1 + exp [ (t 2 ) /3 ]} .
(6.1)
As described in Appendix C.7, the parameters in (6.1) have a physical

interpretation: 1 is the asymptotic height; 2 is the time at which the tree
reaches half of its asymptotic height; and 3 is the time elapsed between
the tree reaching half and 1/(1 + e1 ) 3/4 of its asymptotic height. The
logistic model (6.1) is linear in one parameter, 1 , but nonlinear in 2 and
3 .
To make the example more concrete, suppose that 1 = 3, 2 = 1, and
3 = 1.2 and that we initially want to model the tree growth for 0.4 t
1.6. The logistic curve, shown as a solid line in Figure 6.1, is approximated
very well in the interval [0.4, 1.6] by the fth-degree polynomial
ht 2.2911 + 16.591t 44.411t2 + 56.822t3 31.514t4 + 6.3028t5
obtained as a least-squares t to equally spaced t values in the interval
[0.4, 1.6]. The polynomial t, shown as a dashed line in Figure 6.1, is virtually indistinguishable from the logistic curve within this interval.
Unlike the coecients in the logistic model, the coecients in the polynomial approximation do not have any physical interpretation. Also, the linear
polynomial model uses twice as many parameters as the logistic model to
give comparable tted values. Finally, the polynomial approximation is unreliable outside the interval [0.4, 1.6]. Figure 6.2, displaying the two curves
over the extended interval [0, 2], shows the dramatic dierences between the
curves outside the original range. We would expect growth curves to follow
a pattern more like the logistic model than like the polynomial model.
Nonlinear mixed-eects models extend linear mixed-eects models by
allowing the regression function to depend nonlinearly on xed and random
eects. Because of its greater exibility, an NLME model is generally more
6.1 LME Models vs. NLME Models
logistic
275
polynomial
3.0
2.5
h(t)
2.0
1.5
1.0
0.5
0.0
0.4
0.6
0.8
1.0
1.2
1.4
1.6
FIGURE 6.1. Logistic curve with parameters 1 = 3, 2 = 1, and 3 = 1.2 and

its fth-order polynomial approximation over the interval [0.4, 1.6].
logistic
polynomial
h(t)
-1
0.0
0.5
1.0
1.5
2.0
FIGURE 6.2. Logistic curve with parameters 1 = 3, 2 = 1, and 3 = 1.2 and

its fth-order polynomial approximation over the interval [0.4, 1.6], plotted over
the interval [0, 2].
276
interpretable and parsimonious than a competitor empirical LME model

based, say, on a polynomial function. Also, the predictions obtained from
an NLME model extend more reliably outside the observed range of the
data.
The greater exibility of NLME models does not come without cost,
however. Because the random eects are allowed to enter the model nonlinearly, the marginal likelihood function, obtained by integrating the joint
density of the response and the random eects with respect to the random
eects, does not have a closed-form expression, as in the LME model. As
a consequence, an approximate likelihood function needs to be used for
the estimation of parameters, leading to more computationally intensive
estimation algorithms and to less reliable inference results. These issues
are described and discussed in detail in Chapter 7.
An important practical dierence between NLME and LME models is
that the former require starting estimates for the xed-eects coecients.
Determining reasonable starting estimates for the parameters in a nonlinear model is somewhat of an art, although some general recommendations
are available (Bates and Watts, 1988, 3.2). In many applications, the same
nonlinear model is to be used several times with similar datasets. In these
cases, it is worthwhile to program the steps used to obtain starting estimates into a function, which can then be used to produce starting estimates
from many datasets. Such self-starting nonlinear models are described and
illustrated in 8.1.2.
There are far more similarities than dierences between LME and NLME
models. Both models are used with grouped data and serve the same purpose: to describe a response variable as a function of covariates, taking
into account the correlation among observations in the same group. Random eects are used to represent within-group dependence in both LME
and NLME models, and the assumptions about the random eects and the
within-group errors are identical in the two models.
The same inside-out model building strategy used for LME models
in Part I is used here with NLME models: whenever feasible, we begin
by getting separate ts of a model for each group, then examining these
individual ts to see which coecients appear to be common to all groups
and which coecients seem to vary among groups. We then proceed to t
an overall mixed-eects model to the data, using random-eects terms if
they seem warranted. Diagnostics plots are then used to assess the models
assumptions and to decide on renements of the initial model.
The S methods for displaying, plotting, comparing, and updating nlme
tted objects will look very familiar to the readers of Part I. Because of
the similarities between LME and NLME models, most of the lme methods
described in 4.2.1 can be used, without changes, with nlme objects. The
nonlinear mixed-eects models modeling functions and methods included
in the nlme library are described and illustrated in detail in Chapter 8. The
examples in the next sections serve to illustrate some of the basic concepts
6.2 Indomethicin Kinetics
277
of NLME models and to introduce the capabilities provided by the nlme

library for analyzing them.

The data for our rst example come from a laboratory study on the pharmacokinetics of the drug indomethicin (Kwan, Breault, Umbenhauer, McMahon and Duggan, 1976). Six human volunteers received bolus intravenous
injections of the same dose of indomethicin and had their plasma concentrations of the drug (in mcg/ml) measured 11 times between 15 minutes
and 8 hours postinjection. These data are described in more detail in Appendix A.12 and are also analyzed in Davidian and Giltinan (1995, 2.1).
The indomethicin data are included in the nlme library as the groupedData
object Indometh.
The plot of the Indometh data, obtained with
> plot( Indometh )
# Figure 6.3
and displayed in Figure 6.3, reveals a familiar pattern in plots of grouped

data: the concentration curves have a similar shape, but dier among individuals. As indicated in Figure 6.3, these data are balanced, as the serum
concentrations were measured at the same time points for all six individuals.
0
5
Indomethicin concentration (mcg/ml)
2.5
2.0
1.5
1.0
0.5
0.0
1
2.5
2.0
1.5
1.0
0.5
0.0
0
Time since drug administration (hr)
FIGURE 6.3. Concentration of indomethicin over time for six subjects following
intravenous injection.
278
A common method of modeling pharmacokinetic data is to represent the

human body as a system of compartments in which the drug is transferred
according to rst-order kinetics (Gibaldi and Perrier, 1982). In such a compartment model the concentration of the drug over time in the dierent
compartments is determined by a linear system of dierential equations,
whose solution can be expressed as a linear combination of exponential
terms. The mechanistic model for the indomethicin concentration is from
a two-compartment model. It expresses the expected concentration E(yt )
at time t as a linear combination of two exponentials
E (yt ) = 1 exp (2 t) + 3 exp (4 t) ,
2 > 0, 4 > 0.
(6.2)
The biexponential model (6.2) is not identiable, in the sense of having

a unique vector of parameters associated with a given set of predictions,
because the parameters in the two exponential terms may be exchanged
without changing the predictions. Identiability is ensured by requiring
that 2 > 4 so that the rst exponential term determines the initial
elimination phase of the drug (the sharp decreases in the individual curves
of Figure 6.3). The terminal elimination phase is primarily determined by
the second exponential term.
Model (6.2) is linear in multipliers 1 and 3 , but nonlinear in the rate
constants 2 and 4 . Because the rate constants must be positive to be
physically meaningful, we reparameterize (6.2) in terms of the log-rate constants 2 = log 2 and 4 = log 4 .
When introducing LME models in 1.1 we demonstrated that the LME
model can be considered as a compromise between a single linear model
that ignores the grouping in the data and a linear model that provides
separate ts for each of the groups. These linear models were t using lm
and lmList, respectively. To illustrate why NLME models are useful for
datasets like the indomethicin data, we will repeat that development using
the nonlinear model-tting functions nls and nlsList.
If we ignore the grouping of the concentration measurements according
to individual and t a single nonlinear model to all the data we express the
indomethicin concentration yij in individual i at time tj is
yij = 1 exp [ exp (2 ) tj ] + 3 exp [ exp (4 ) tj ] + ij ,
(6.3)
where the error terms ij are assumed to be independently distributed

as N (0, 2 ). Nonlinear regression models with independent, identically distributed Gaussian errors, like (6.3), are t by nonlinear least squares (Bates
and Watts, 1988; Seber and Wild, 1989), as implemented in the S function
nls (Bates and Chambers, 1992).
Starting values for the parameters in the biexponential model are usually
obtained through the method of peeling, as described in Appendix C.4.1.
However, we need not be concerned with obtaining initial estimates for
the biexponential model parameters, as the self-starting model function
279
Subject
6
5
2
4
1
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
Residuals
FIGURE 6.4. Boxplots of residuals by subject for fm1Indom.nls, a nonlinear least

squares t of a two-compartment model to the indomethicin data ignoring the
subject dependence.
SSbiexp, described in Appendix C.4, produces them automatically from the

data. A call to nls to t the biexponential model (6.3) using a self-starting
function is almost as simple as a call to lm to t a linear model.
> fm1Indom.nls <- nls( conc ~ SSbiexp(time, A1, lrc1, A2, lrc2),
+
data = Indometh )
> summary(fm1Indom.nls)
Formula: conc ~ SSbiexp(time, A1, lrc1, A2, lrc2)
Parameters:
Value Std. Error
A1 2.77342
0.25323
lrc1 0.88627
0.22222
A2 0.60663
0.26708
lrc2 -1.09209
0.40894
t value
10.9522
3.9882
2.2713
-2.6705

. . .
> plot( fm1Indom.nls, Subject ~ resid(.), abline = 0 ) # Figure 6.4
The correspondence between the parameters in fm1Indom.nls and in (6.3)

is 1 = A1, 2 = lrc1, 3 = A2, 4 = lrc2.
The boxplots of the residuals by individual in Figure 6.4 are similar to
those from the lm t to the Rails data (Figure 1.2). In Figure 6.4 the residuals tend to be mostly negative for some subjects and mostly positive for
others although the pattern is not as pronounced as that in Figure 1.2.
Because a single concentration curve is used for all subjects, the individual dierences noticed in Figure 6.3 are incorporated in the residuals, thus
inating the residual standard error. Probably the most important drawback of using an nls model with grouped data is that it prevents us from
280
understanding the true structure of the data and from considering dierent sources of variability that are of interest in themselves. For example,
in the indomethicin study, an important consideration in determining an
adequate therapeutic regime for the drug is knowing how the concentration
proles vary among individuals.
To t a separate biexponential model to each subject, thus allowing the
individual eects to be incorporated in the parameter estimates, we express
the model as
yij = 1i exp [ exp (2i ) tj ] + 3i exp [ exp (4i ) tj ] + ij ,
(6.4)
where, as before, the ij are independent N (0, 2 ) errors and use the
nlsList function
> fm1Indom.lis <- nlsList(conc ~ SSbiexp(time, A1, lrc1, A2, lrc2),
+
data = Indometh )
> fm1Indom.lis
Call:
Model: conc ~ SSbiexp(time, A1, lrc1, A2, lrc2) | Subject
Data: Indometh
Coefficients:
A1
lrc1
1 2.0293 0.57938
4 2.1979 0.24249
2 2.8277 0.80143
5 3.5663 1.04095
6 3.0022 1.08811
3 5.4677 1.74968
A2
0.19154
0.25481
0.49903
0.29170
0.96840
1.67564
lrc2
-1.78783
-1.60153
-1.63508
-1.50594
-0.87324
-0.41226

We can see that there is considerable variability in the individual parameter estimates and that the residual standard error is less than one-half
that from the nls t. The boxplots of the residuals by subject, shown in
Figure 6.5, indicate that the individual eects have been accounted for in
the tted nlsList model.
The nlsList model is at the other extreme of the exibility spectrum
compared to the nls model: it uses 24 coecients to represent the individual concentration proles and does not take into account the obvious similarities among the individual curves, indicated in Figure 6.3. The nlsList
model is useful when one is interested in modeling the behavior of a particular, xed set of individuals, but it is not adequate when the observed
individuals are to be treated as a sample from a population of similar individuals, which constitutes the majority of applications involving grouped
data. In this case, the interest is in estimating the average behavior of an
281
Subject
6
5
2
4
1
-0.2
-0.1
0.0
0.1
0.2
Residuals (mcg/ml)
FIGURE 6.5. Boxplots of residuals by subject for fm1Indom.lis, a set of nonlinear

regression ts of a two-compartment model to the indomethicin data where each
subjects data is t separately.
A1
3
| | |
| | |
|||
2
|
|
|
|
0.0
0.5
1.0
|||
| | |
2.0
-0.5
|
|
0.5
|
1.5
|
|
|
|
1.5
| | |
lrc2
| | |
|||
A2
| | |
Subject
lrc1
|
-4
-3
-2
-1
FIGURE 6.6. Ninety-ve percent condence intervals on the biexponential model

parameters for each individual in the indomethicin data.
individual in the population and the variability among and within individuals, which is precisely what mixed-eects models are designed to do.
The plot of the individual condence intervals for the coecients in the
nlsList model, shown in Figure 6.6, gives a better idea about their variability among subjects.
> plot( intervals(fm1Indom.lis) )
# Figure 6.6
The terminal phase log-rate constants, 4i , do not seem to vary substantially among individuals, but the remaining parameters do.
Recall that in lmList ts to balanced data, the lengths of the condence
intervals on a parameter were the same for all the groups (see Figure 1.12,
p. 33, or Figure 1.13, p. 34). This does not occur in an nlsList t because
the approximate standard errors used to produce the condence intervals
in a nonlinear least squares t depend on the parameter estimates (Seber
and Wild, 1989, 5.1).
282
To introduce the concepts of xed and random eects in a nonlinear

mixed-eects model, it is useful to re-express the model (6.4) as
%
,
$
% $

yij = 1 + 1i 1 exp exp 2 + 2i 2 tj
$
%
,
$
%

(6.5)
+ 3 + 3i 3 exp exp + tj + ij ,
4
4i
where denotes the average of the individual parameters. The nlsList

model treats the deviations of the individual coecients from their mean
as parameters to be estimated. Mixed-eects models, on the other hand,
represent these deviations from the mean value of the coecients (the xed
eects) as random eects, treating the individuals as a sample from a population. The nonlinear mixed-eects version of (6.5) is
yij = (1 + b1i ) exp [ exp (2 + b2i ) tj ]
+ (3 + b3i ) exp [ exp (4 + b4i )) tj ] + ij . (6.6)
The xed eects 1 , 2 , 3 , and 4 represent the mean values of the parameters in the population of individuals. The individual deviations are
represented by the random eects b1i , b2i , b3i , and b4i , which are assumed
to be distributed normally with mean 0 and variancecovariance matrix .
Random eects corresponding to dierent individuals are assumed to be
independent. The within-group errors ij are assumed to be independently
distributed as N (0, 2 ) and to be independent of the random eects.
The nonlinear mixed-eects model (6.6) gives a compromise between the
rigid nls model (6.3) and the overparameterized nlsList model (6.4). It
accommodates individual variations through the random eects, but ties
the dierent individuals together through the xed eects and the variance
covariance matrix .
A crucial step in the model-building of mixed-eects models is deciding
which of the coecients in the model need random eects to account for
their between-subject variation and which can be treated as purely xed
eects. Plots of individual condence intervals obtained from an nlsList
t, like the one shown in Figure 6.6, are often useful for that purpose.
In the case of the indomethicin data, the individual condence intervals
suggest that the b4i random eect for the terminal elimination phase lograte constant in (6.6) is not needed.
An alternative model-building strategy is to start with a model with
random eects for all parameters and then examine the tted object to
decide which, if any, of the random eects can be eliminated from the
model. One problem with this approach is that, when a general positivedenite structure is assumed for the random eects variancecovariance
matrix , the number of parameters to estimate increases with the square
of the number of random eects. In cases where the number of random
eects is large relative to the number of individuals, as in the indomethicin
data, it is generally recommended to use a diagonal initially, to prevent
283
convergence problems with an overparameterized model. We apply this

approach to the indomethicin data.
> fm1Indom.nlme <- nlme( fm1Indom.lis,
+
random = pdDiag(A1 + lrc1 + A2 + lrc2 ~ 1) )
> fm1Indom.nlme
Nonlinear mixed-effects model fit by maximum likelihood
Model: conc ~ SSbiexp(time, A1, lrc1, A2, lrc2)
Data: Indometh
Log-likelihood: 54.592
Fixed: list(A1 ~ 1, lrc1 ~ 1, A2 ~ 1, lrc2 ~ 1)
A1
lrc1
A2
lrc2
2.8276 0.7733 0.46104 -1.345
Random effects:
Formula: list(A1 ~ 1, lrc1 ~ 1, A2 ~ 1, lrc2 ~ 1)
Level: Subject
Structure: Diagonal
A1
lrc1
A2
lrc2 Residual
StdDev: 0.57136 0.15811 0.11154 7.2051e-11 0.081496
Number of Groups: 6
The nlme function extracts the information about the model to t, the
parameters to estimate, and the starting estimates for the xed eects
from the fm1Indom.lis object.
The near-zero estimate for the standard deviation of the lrc2 random
eect suggests that this term could be dropped from the model. The remaining estimated standard deviations suggest that the other random effects should be kept in the model. We can test if the lrc2 random eect
can be removed from the model by updating the t and using anova.
> fm2Indom.nlme <- update( fm1Indom.nlme,
+
random = pdDiag(A1 + lrc1 + A2 ~ 1) )
> anova( fm1Indom.nlme, fm2Indom.nlme )
Model df
AIC
BIC logLik
Test
L.Ratio
fm1Indom.nlme
1 9 -91.185 -71.478 54.592
fm2Indom.nlme
2 8 -93.185 -75.668 54.592 1 vs 2 6.2637e-06
p-value
fm1Indom.nlme
fm2Indom.nlme
0.998
The two ts give nearly identical log-likelihoods, conrming that lrc2 can
be treated as a purely xed eect.
To further explore the variancecovariance structure of the random effects that are left in fm2Indom.nlme, we update the t using a general
positive-denite matrix.
> fm3Indom.nlme <- update( fm2Indom.nlme, random = A1+lrc1+A2 ~ 1 )
284
> fm3Indom.nlme
. . .
Random effects:
Formula: list(A1 ~ 1, lrc1 ~ 1, A2 ~ 1)
Level: Subject
StdDev Corr
A1 0.690406 A1
lrc1
lrc1 0.179030 0.932
A2 0.153669 0.471 0.118
Residual 0.078072
. . .
The large correlation between the A1 and lrc1 random eects and the small
correlation between these random eects and the A2 random eect suggest
that a block-diagonal could be used to represent the variancecovariance
structure of the random eects.
> fm4Indom.nlme <- update( fm3Indom.nlme,
+
random = pdBlocked(list(A1 + lrc1 ~ 1, A2 ~ 1)) )
Model df
AIC
BIC logLik
Test L.Ratio
fm3Indom.nlme
1 11 -94.945 -70.859 58.473
fm4Indom.nlme
2 9 -97.064 -77.357 57.532 1 vs 2 1.8809
p-value
fm3Indom.nlme
fm4Indom.nlme 0.3904
The large p-value for the likelihood ratio test and the smaller values
for the AIC and BIC corroborate the block-diagonal variancecovariance
structure. Allowing the A1 and lrc1 random eects to be correlated causes
a signicant improvement in the log-likelihood.
Model df
AIC
BIC logLik
Test L.Ratio
fm2Indom.nlme
1 8 -93.185 -75.668 54.592
fm4Indom.nlme
2 9 -97.064 -77.357 57.532 1 vs 2 5.8795
p-value
fm2Indom.nlme
fm4Indom.nlme 0.0153
The plot of the standardized residuals versus the tted values corresponding to fm4Indom.nlme, presented in Figure 6.7, does not indicate any
departures from the NLME model assumptions, except for two possible
outlying observations for Individual 2.
> plot( fm4Indom.nlme, id = 0.05, adj = -1 )
# Figure 6.7
No signicant departures from the assumption of normality for the withingroup errors is observed in the normal probability plot of the standardized
residuals of fm4Indom.nlme, shown in Figure 6.8.

2
3
285
1
0
-1
-2
-3
2
0.0
0.5
1.0
1.5
2.0
2.5
Fitted values (mcg/ml)
FIGURE 6.7. Scatter plot of standardized residuals versus tted values for
fm4Indom.nlme.
> qqnorm( fm4Indom.nlme )
# Figure 6.8
A nal assessment of the adequacy of the fm4Indom.nlme model is given

by the plot of the augmented predictions in Figure 6.9. For comparison, and
to show how individual eects are accounted for in the NLME model, both
the population predictions (corresponding to random eects equal to zero)
and the within-group predictions (obtained using the estimated random
eects) are displayed.
> plot( augPred(fm4Indom.nlme, level = 0:1) )
# Figure 6.9
Note that the within-group predictions are in close agreement with the observed concentrations, illustrating that the NLME model can accommodate
individual eects.
We conclude that fm4Indom.nlme provides a good representation of the
concentration proles in the indomethicin data. Its summary
> summary( fm4Indom.nlme )
Model: conc ~ SSbiexp(time, A1, lrc1, A2, lrc2)
Data: Indometh
AIC
BIC logLik
-97.064 -77.357 57.532
Random effects:
Block 1: A1, lrc1
Formula: list(A1 ~ 1, lrc1 ~ 1)
286
-1
-2
-3
-2
-1
FIGURE 6.8. Normal plot of standardized residuals for the fm4Indom.nlme nlme
t.
fixed
0
5
Subject
2
Indomethicin concentration (mcg/ml)
2.5
2.0
1.5
1.0
0.5
0.0
1
2.5
2.0
1.5
1.0
0.5
0.0
0
FIGURE 6.9. Population predictions (fixed), within-group predictions

(Subject), and observed concentrations of indomethicin (circles) versus time since
injection for fm4Indom.nlme.
6.3 Growth of Soybean Plants
287
Level: Subject
StdDev Corr
A1 0.69496 A1
lrc1 0.17067 0.905
Block 2: A2
Formula: A2 ~ 1 | Subject
A2 Residual
StdDev: 0.18344 0.078226
Fixed effects: list(A1
Value Std.Error
A1 2.8045
0.31493
lrc1 0.8502
0.11478
A2 0.5887
0.13321
lrc2 -1.1029
0.16954
. . .
~ 1, lrc1 ~ 1, A2 ~ 1, lrc2 ~ 1)
DF t-value p-value
57 8.9049 <.0001
57 7.4067 <.0001
57 4.4195 <.0001
57 -6.5054 <.0001
shows that the xed-eects estimates are similar to the parameter estimates in the nls t fm1Indom.nls. The approximate standard errors for the
xed eects are substantially dierent from and, except for A1, considerably
smaller than those from the nls t. The estimated within-group standard
error is slightly larger than the residual standard error in the nlsList t
fm1Indom.lis.

The example discussed in this section illustrates an important area of application of NLME models: growth curve data. It also introduces the concept
of using covariates to explain between-group variability in NLME models.
The soybean data, displayed in Figure 6.10, are described in Davidian
and Giltinan (1995, 1.1.3, p. 7) as Data from an experiment to compare
growth patterns of two genotypes of soybeans: Plant Introduction #416937
(P), an experimental strain, and Forrest (F), a commercial variety. The
average leaf weight (in grams) of six plants chosen at random from each plot
was measured at approximately weekly intervals, between two and eleven
weeks after planting. The experiment was carried out over three dierent
planting years: 1988, 1989, and 1990. Eight plots were planted with each
genotype in each planting year, giving a total of forty-eight plots in the
study. These data are available in the nlme library as the groupedData
object Soybean and are also described in Appendix A.27.
> Soybean[1:3, ]
Grouped Data: weight ~ Time | Plot
288

20
P
1988
40
60
80
P
1989
P
1990
30
25
20
Leaf weight/plant (g)
15
10
5
0
F
1988
F
1989
F
1990
30
25
20
15
10
5
0
20
40
60
80
20
40
60
80
Time since planting (days)
FIGURE 6.10. Average leaf weight per plant of two genotypes of soybean versus
time since planting, over three dierent planting years. Within each year data
were obtained on eight plots of each variety of soybean.
1
2
3
>
Plot Variety
1988F1
F
1988F1
F
1988F1
F
plot( Soybean,
Year Time weight

1988
14 0.106
1988
21 0.261
1988
28 0.666
outer = ~ Year * Variety )
# Figure 6.10
The average leaf weight per plant in each plot is measured the same number
of times, but at dierent times, making the data unbalanced.
There is considerable variation in the growth curves among plots, but
the same overall S-shaped pattern is observed for all plots. This nonlinear
growth pattern is well described by the three parameter logistic model (6.1),
introduced in 6.1. The self-starting function SSlogis, described in Appendix C.7, can be used to automatically generate starting estimates for
the parameters in an nlsList t.
> fm1Soy.lis <- nlsList( weight ~ SSlogis(Time, Asym, xmid, scal),
+
data = Soybean )
Error in nls(y ~ 1/(1 + exp((xmid - x)/scal)..: singular gradient
matrix
Dumped
289
> fm1Soy.lis
Call:
Model: weight ~ SSlogis(Time, Asym, xmid, scal) | Plot
Data: Soybean
Coefficients:
Asym
1988F4 15.1513
1988F2 19.7455
. . .
1989P8
NA
. . .
1990P5 19.5438
1990P2 25.7873
xmid
52.834
56.575
scal
5.1766
8.4067
NA
NA
51.148 7.2920
62.360 11.6570

The error message from nls indicates that convergence was not attained for
one of the plots, 1989P8. The nlsList function is able to recover from such
nonconvergence problems and carry on with subsequent nls ts. Missing
values (NA) are assigned to the coecients of the nonconverging ts. The
coecients in fm1Soy.lis are related to the logistic model parameters as
follows: 1 = Asym, 2 = xmid, and 3 = scal
Analysis of the individual condence intervals for fm1Soy.lis suggests
that random eects are needed for all of the parameters in the logistic
model. The corresponding nonlinear mixed-eects model for the average
leaf weight per plant yij in plot i at tij days after planting is
1i
+ ij ,
yij =
1 + exp [ (tij 2i ) /3i ]

1
b1i
1i
i =
2i =
2 + b2i = + bi ,
3i
3
b3i

bi N (0, ) , ij N 0, 2 .
(6.7)
The xed eects represent the mean values of the parameters i in the
population and the random eects bi represent the deviations of the i
from their mean values. The random eects are assumed to be independent for dierent plots and the within-group errors ij are assumed to be
independent for dierent i, j and to be independent of the random eects.
Because the number of plots in the soybean data, 48, is large compared
to the number of random eects in (6.7), we use a general positive-denite
for the initial NLME model. Because we can extract information about
the model, the parameters to estimate, and the starting values for the xed
eects from the fm1Soy.lis object we can t model (6.7) with the simple
call
290
4
2
0
-2
-4
-6
0
10
15
20
25
Fitted values (g)
fm1Soy.nlme.
> fm1Soy.nlme <- nlme( fm1Soy.lis )

> fm1Soy.nlme
Model: weight ~ SSlogis(Time, Asym, xmid, scal)
Data: Soybean
Log-likelihood: -739.84
Fixed: list(Asym ~ 1, xmid ~ 1, scal ~ 1)
Asym xmid
scal
19.253 55.02 8.4033
Random effects:
Formula: list(Asym ~ 1, xmid ~ 1, scal ~ 1)
Level: Plot
StdDev Corr
Asym 5.2011 Asym xmid
xmid 4.1974 0.721
scal 1.4047 0.711 0.958
Residual 1.1235
The plot of the standardized residuals versus the tted values in Figure 6.11 shows a pattern of increasing variability for the within-group errors. We model the within-group heteroscedasticity using the power variance function, described in 5.2.1 and represented in S by the varPower
class.
> fm2Soy.nlme <- update( fm1Soy.nlme, weights = varPower() )
291
3
2
1
0
-1
-2
10
15
20
Fitted values (g)
FIGURE 6.12. Standardized residuals versus tted values in the nlme t of the
soybean data, with heteroscedastic error.
> anova( fm1Soy.nlme, fm2Soy.nlme )

Model df
AIC
BIC logLik
fm1Soy.nlme
1 10 1499.7 1539.9 -739.84
fm2Soy.nlme
2 11 737.3 781.6 -357.66 1 vs 2 764.35 <.0001
The heteroscedastic model provides a much better representation of the

data. We can assess the adequacy of the power variance function by again
plotting the standardized residuals against the tted values as in Figure 6.12. The power variance function seems to model adequately the
within-plot heteroscedasticity.
The primary question of interest for the soybean data is the possible
relationship between the growth pattern of the soybean plants and the experimental factors Variety and Year. Plots of estimates of the random effects are useful for exploring the relationship between the individual growth
patterns and the experimental factors.
> plot(ranef(fm2Soy.nlme, augFrame = T),
+
form = ~ Year * Variety, layout = c(3,1))
# Figure 6.13
In Figure 6.13 all three parameters seem to vary with year and variety. It
appears that the asymptote (Asym) and the scale (scal) are larger for the P
variety than for the F variety and that this dierence is more pronounced
in 1989. The time at which half of the asymptotic leaf weight is attained
(xmid) appears to be smaller for the P variety than for the F variety.
The fixed argument to nlme allows linear modeling of parameters with
respect to covariates. For example, we can model the dependence of all
three parameters on Year with
> soyFix <- fixef( fm2Soy.nlme )
292

Asym
xmid
scal
1990 P
1990 F
1989 P
1989 F
1988 P
1988 F
-8
-6
-4
-2
-2
-1
-0.4
-0.2
0.0
0.2
Random effects
FIGURE 6.13. Estimates of the random eects by Year and Variety in the nlme
t of the soybean data.
>
>
+
+
>
.
options( contrasts = c("contr.treatment", "contr.poly") )

fm3Soy.nlme <- update( fm2Soy.nlme,
fixed = Asym + xmid + scal ~ Year,
start = c(soyFix[1], 0, 0, soyFix[2], 0, 0, soyFix[3], 0, 0) )
fm3Soy.nlme
. .
Fixed: Asym + xmid + scal ~ Year
Asym.(Intercept) Asym.Year1989 Asym.Year1990 xmid.(Intercept)
20.222
-6.3775
-3.4995
54.118
xmid.Year1989 xmid.Year1990 scal.(Intercept) scal.Year1989
-2.4696
-4.8764
8.0515
-0.93374
scal.Year1990
-0.66884
Random effects:
Level: Plot
StdDev
Corr
Asym.(Intercept) 2.3686896 Asy(I) xmd(I)
xmid.(Intercept) 0.5863454 -0.997
scal.(Intercept) 0.0043059 -0.590 0.652
Residual 0.2147634
Variance function:
power
0.95187
. . .
293
The default parameterization for contrasts is changed to contr.treatment

to allow easier interpretation of the xed eects included in the model:
they represent dierences from the year 1988. Note that, because the xedeects model has changed, new starting values for the xed eects must be
provided.
We can assess the signicance of Year for the xed eects model using
anova with a single argument.
> anova( fm3Soy.nlme )
Asym.(Intercept)
1
356
402.4 <.0001
Asym.Year
2
356
105.0 <.0001
xmid.(Intercept)
1
356 9641.2 <.0001
xmid.Year
2
356
10.2 <.0001
scal.(Intercept)
1
356 8378.5 <.0001
scal.Year
2
356
11.7 <.0001
As suggested by Figure 6.13, Year has a very signicant eect on the growth
pattern of the soybean plants.
The estimated standard deviation for the scal random eect in the
fm3Soy.nlme t is only 0.004, corresponding to an estimated coecient of
variation with respect to the scal.(Intercept) xed eect of only 0.05%.
This suggests that scal can be treated as a purely xed eect. When we do
ret the model dropping the scal random eect, we get a p-value of 0.99
in the likelihood ratio test. It often happens that creating a better-tting
model for the xed eects, by including their dependence on covariates, reduces the need for random-eects terms. In these cases, the between-group
parameter variation is mostly being explained by the covariates included
in the model.
Proceeding sequentially in the model-building process by examining plots
of the estimated random eects against the experimental factors, testing
for the inclusion of covariates and for the elimination of random eects, we
end up with the following model, in which the only random eect is that
for Asym.
> summary( fm4Soy.nlme )
. . .
AIC
BIC logLik
616.32 680.66 -292.16
Random effects:
Formula: Asym ~ 1 | Plot
Asym.(Intercept) Residual
StdDev:
1.0349 0.21804
Variance function:
294
power
0.9426
Fixed effects: list(Asym ~ Year * Variety, xmid ~ Year + Variety,
scal ~ Year)
Asym.(Intercept) 19.434
0.9537 352 20.379 <.0001
Asym.Year1989 -8.842
1.0719 352 -8.249 <.0001
Asym.Year1990 -3.707
1.1768 352 -3.150 0.0018
Asym.Variety
1.623
1.0380 352
1.564 0.1188
Asym.Year1989Variety
5.571
1.1704 352
4.760 <.0001
Asym.Year1990Variety
0.147
1.1753 352
0.125 0.9004
xmid.(Intercept) 54.815
0.7548 352 72.622 <.0001
xmid.Year1989 -2.238
0.9718 352 -2.303 0.0218
xmid.Year1990 -4.970
0.9743 352 -5.101 <.0001
xmid.Variety -1.297
0.4144 352 -3.131 0.0019
scal.(Intercept)
8.064
0.1472 352 54.762 <.0001
scal.Year1989 -0.895
0.2013 352 -4.447 <.0001
scal.Year1990 -0.673
0.2122 352 -3.172 0.0016
. . .
The residual plots for fm4Soy.nlme do not indicate any violations in the
NLME model assumptions. An overall assessment of the adequacy of the
model is provided by the plot of the augmented predictions in Figure 6.14,
which indicates that the fm4Soy.nlme model describes the individual growth
patterns of the soybean plots well.
6.4 Clinical Study of Phenobarbital Kinetics

The data for the last example in this chapter come from a clinical pharmacokinetic study of the drug phenobarbital, used for preventing seizures
(Grasela and Donn, 1985). The study followed 59 preterm infants during
the rst 16 days after birth. Each infant received one or more intravenous
injections of phenobarbital. As part of routine clinical monitoring, blood
samples were drawn from the infants at irregular time intervals to determine the serum concentration of phenobarbital. The number of concentration measurements per individual varies between 1 and 6, with a total of
155 measurements for all 59 individuals. This is typical of clinical pharmacokinetic studies: there are relatively few observations on each of a large
number of individuals. In addition to the concentration and dosing information, each infants birth weight (in kilograms) and 5-minute Apgar score
(which gives an overall indication of health of the newborn) are provided.
These data have been analyzed in Grasela and Donn (1985), Boeckmann,
Sheiner and Beal (1994), Davidian and Giltinan (1995, 6.6), and Littell
et al. (1996, 12.5).

20
1990P8
40
60
80
20
1990P7
1990P3
40
60
80
20
1990P1
1990P6
40
60
80
20
1990P5
1990P2
40
295
60
80
1990P4
Leaf weight/plant (g)
30
25
20
15
10
5
0
1990F2
1990F3
1990F4
1990F5
1990F1
1990F8
1990F7
1990F6
1989P7
1989P4
1989P6
1989P5
1989P1
1989P3
1989P2
1989P8
30
25
20
15
10
5
0
30
25
20
15
10
5
0
1989F6
1989F5
1989F4
1989F1
1989F2
1989F7
1989F8
1989F3
1988P1
1988P5
1988P4
1988P8
1988P7
1988P3
1988P2
1988P6
30
25
20
15
10
5
0
30
25
20
15
10
5
0
1988F4
1988F2
1988F1
1988F7
1988F5
1988F8
1988F6
1988F3
30
25
20
15
10
5
0
20
40
60
80
20
40
60
80
20
40
60
80
20
40
60
80

FIGURE 6.14. Average leaf weights of soybean plants versus time since planting
and their within-group predictions from the model fm4Soy.nlme.
The phenobarbital data, displayed in Figure 6.15, are available in the

nlme library as the groupedData object Phenobarb and are described in
more detail in Appendix A.23.
The mechanistic model postulated for the phenobarbital kinetics is a onecompartment open model with intravenous administration and rst-order
elimination (Grasela and Donn, 1985) . The corresponding model for the
expected phenobarbital concentration ct at time t for an infant receiving a
single dose Dd administered at time td is

Cl
Dd
exp (t td ) ,
ct =
V
V
td < t,
where V and Cl denote, respectively, the volume of distribution and the

clearance. The model for an individual who has received several doses by
296

0
25
200
400
35
52
200
400
13
16
200
400
59
48
200
400
24
50
70
60
50
40
30
20
Serum phenobarbital concentration (ug/L)
10
70
37
20
12
17
26
49
39
54
10
58
51
19
27
53
60
50
40
30
20
10
70
60
50
40
30
20
10
70
33
44
23
57
34
18
36
21
15
38
11
45
22
40
47
31
14
41
60
50
40
30
20
10
70
60
50
40
30
20
10
42
70
28
30
56
46
55
32
43
29
60
50
40
30
20
10
0
200
400
200
400
200
400
200
400
200
400
Time (hr)
FIGURE 6.15. Serum concentrations of phenobarbital in 59 newborn infants under varying dosage regimens versus time after birth.
time t is given by the sum of the individual contributions of each dose.

ct =

" Dd
Cl
exp (t td ) .
V
V
(6.8)
d:td <t
Model (6.8) can also be expressed in recursive form (Grasela and Donn,
1985). To ensure that the estimates of V and Cl are positive, we reparameterize 6.8) using the logarithm of these parameters: lV = log V and
lCl = log Cl . The function phenoModel in the nlme library implements the
reparameterized version of model (6.8) in S. Because phenoModel is not selfstarting, initial values need to be provided for the parameters when this
function is used for estimation in S.
297
Because of the small number of concentrations recorded for each individual, the usual model-building approach of beginning the analysis with
an nlsList t cannot be used with the phenobarbital data and we must go
directly to an NLME t. The nonlinear mixed-eects model corresponding
to (6.8) representing the phenobarbital concentration yij measured at time
tij on individual i, following intravenous injections of dose Did at times tid ,
is expressed as
"
Did
exp [ exp (lCl i lV i ) (tij tid )] + ij ,
exp (lV i )
d:tid <tij

b
lCl i
= 1 + 1i = + bi , bi N (0, ) , ij N 0, 2 .
lV i
2
b2i
(6.9)
yij =
The xed eects, , represent the average log-clearance and log-volume of

distribution in the infant population, and the random eects, bi , account for
individual dierences from the population average. As usual, the random
eects are assumed to be independent for dierent individuals and the
within-group errors ij are assumed to be independent for dierent i, j and
to be independent of the random eects.
To avoid convergence problems with the optimization algorithm used in
nlme due to the sparsity of the phenobarbital concentrations in the data, a
diagonal is assumed in model (6.9), which is then tted with
> fm1Pheno.nlme <+
nlme( conc ~ phenoModel(Subject, time, dose, lCl, lV),
+
data = Phenobarb, fixed = lCl + lV ~ 1,
+
random = pdDiag(lCl + lV ~ 1), start = c(-5, 0),
+
na.action = na.include, naPattern = ~ !is.na(conc) )
> fm1Pheno.nlme
Model: conc ~ phenoModel(Subject, time, dose, lCl, lV)
Data: Phenobarb
Fixed: lCl + lV ~ 1
lCl
lV
-5.0935 0.34259
Random effects:
Formula: list(lCl ~ 1, lV ~ 1)
Level: Subject
Structure: Diagonal
lCl
lV Residual
StdDev: 0.43989 0.45048
2.7935
298

Wt
ApgarInd
1.0
lCl
0.5
0.0
-0.5
-1.0
0.5
1.0
1.5
2.0
Wt
2.5
2.0
2.5
3.0
3.5
<5
>= 5
ApgarInd
1.0
lV
0.5
0.0
-0.5
0.5
1.0
1.5
3.0
3.5
<5
>= 5
FIGURE 6.16. Estimated log-clearance and log-volume of distribution random

eects from model fm1Pheno.nlme versus birth weight (Wt) and Apgar score indicator (ApgarInd) in the phenobarbital data. A loess smoother is included in
the scatter plots of the continuous covariates to aid in visualizing possible trends.
Starting values for the xed eects are obtained from Davidian and Giltinan
(1995, 6.6). The na.action argument in the nlme call is used to preserve
those rows with dose information. (These rows contain NA for the concentration.) The naPattern argument is used to remove these rows from the
calculation of the objective function in the optimization algorithm.
One of the questions of interest for the phenobarbital data is the possible
relationship between the pharmacokinetic parameters in (6.9) and the additional covariates available on the infants, birth weight and 5-minute Apgar
score. For the purposes of modeling the pharmacokinetic parameters, the
5-minute Apgar score is converted to a binary indicator of whether the
score is < 5 or 5, represented by the column ApgarInd in the Phenobarb
data frame.
Figure 6.16 contains plots of the estimated random eects from
fm1Pheno.nlme versus birth weight (Wt) and 5-minute Apgar score indicator
(ApgarInd). It is produced by
> fm1Pheno.ranef <- ranef( fm1Pheno.nlme, augFrame = T )
> plot( fm1Pheno.ranef, form = lCl ~ Wt + ApgarInd )
> plot( fm1Pheno.ranef, form = lV ~ Wt + ApgarInd )
299
The plots in Figure 6.16 clearly indicate that both clearance and volume
of distribution increase with birth weight. A linear model seems adequate
to represent the increase in lV with birth weight. For birth weights less than
2.5 kg, the increase in lCl seems linear. Because there are few infants with
birth weights greater than 2.5 kg in this data set, it is unclear whether the
linear relationship between lCl and Wt extends beyond this limit, but we
will assume it does. The Apgar score does not seem to have any relationship
with clearance and is not included in the model for the lCl xed eect. It is
unclear whether the Apgar score and the volume of distribution are related,
so we include ApgarInd in the model for lV to test for a possible relationship.
The updated t with covariates included in the xed-eects model is
then obtained with
> fm2Pheno.nlme <- update( fm1Pheno.nlme,
+
fixed = list(lCl ~ Wt, lV ~ Wt + ApgarInd),
+
start = c(-5.0935, 0, 0.34259, 0, 0),
+
control = list(pnlsTol = 1e-6) )
> #pnlsTol reduced to prevent convergence problems in PNLS step
> summary( fm2Pheno.nlme )
. . .
Random effects:
Level: Subject
Structure: Diagonal
lCl.(Intercept) lV.(Intercept) Residual
StdDev:
0.21599
0.17206
2.7374
Fixed effects: list(lCl ~ Wt, lV ~ Wt + ApgarInd)
lCl.(Intercept) -5.9574
0.12425 92 -47.947 <.0001
lCl.Wt 0.6197
0.07569 92
8.187 <.0001
lV.(Intercept) -0.4744
0.07258 92 -6.537 <.0001
lV.Wt 0.5325
0.04141 92 12.859 <.0001
lV.ApgarInd -0.0228
0.05131 92 -0.444 0.6577
. . .
As expected, the xed eects corresponding to birth weight, lCl.Wt and

lV.Wt, are highly signicant. The large p-value for the lV.ApgarInd xed
eect indicates that the volume of distribution is not related to the Apgar
scores. The estimated standard deviations for the random eects are about
half of the corresponding values in the fm1Pheno.nlme t, indicating that
a substantial part of the between-individual variability in the pharmacokinetic parameters is explained by birth weight.
The ApgarInd variable is dropped from the lV xed eect model to give
the nal NLME model for the phenobarbital data considered here.
300
> fm3Pheno.nlme <- update( fm2Pheno.nlme,

+
fixed = lCl + lV ~ Wt, start = fixef(fm2Pheno.nlme)[-5] )
> fm3Pheno.nlme
Model: conc ~ phenoModel(Subject, time, dose, lCl, lV)
Data: Phenobarb
Fixed: lCl + lV ~ Wt
lCl.(Intercept) lCl.Wt lV.(Intercept)
lV.Wt
-5.9577 0.61968
-0.48452 0.53205
Random effects:
Level: Subject
Structure: Diagonal
lCl.(Intercept) lV.(Intercept) Residual
StdDev:
0.21584
0.17316
2.7326
. . .
The likelihood ratio tests for dropping either of the random eects in
fm3Pheno.nlme have very signicant p-values (< 0.0001), indicating that
both random eects are needed in the model to account for individual
eects.
A plot of the augmented predictions is not meaningful for the phenobarbital data, due to the small number of observations per individual, but
we can still assess the adequacy of the t with the plot of the observed
concentrations against the within-group tted values, produced with
> plot( fm3Pheno.nlme, conc ~ fitted(.), abline = c(0,1) )
and displayed in Figure 6.17. The good agreement between the observations
and the predictions attests the adequacy of the fm3Pheno.nlme model.
6.5 Chapter Summary

This chapter gives an introductory overview of the nonlinear mixed-eects
model, describing its basic concepts and assumptions and relating it to the
linear mixed-eects model described in the rst part of the book. Real-life
examples from pharmacokinetics studies and an agricultural experiment
are used to illustrate the use of the nlme function in S, and its associated
methods, for tting and analyzing NLME models.
The many similarities between NLME and LME models allow most of
the lme methods dened in the rst part of the book to also be used with
the nlme objects introduced in this section. There are, however, important
dierences between the two models, and the methods used to t them,
which translate into more complex estimation algorithms and less accurate
inference for NLME models.
Exercises
301
Serum phenobarbital concentration (ug/L)
70
60
50
40
30
20
10
10
20
30
40
50
60
Fitted values (ug/L)
FIGURE 6.17. Observed phenobarbital concentrations versus within-group tted

values corresponding to the fm3Pheno.nlme model.
The purpose of this chapter is to present the motivation for using NLME
models with grouped data and to set the stage for the following two chapters
in the book, dealing with the theory and computational methods for NLME
models (Chapter 7) and the nonlinear modeling facilities in the nlme library
(Chapter 8).
Exercises
1. The Loblolly data described in Appendix A.13 consist of the heights
of 14 Loblolly pine trees planted in the southern United States, measured at 3, 5, 10, 15, 20, and 25 years of age. An asymptotic regression
model, available in nlme as the self-starting function SSasymp (Appendix C.1), seems adequate to explain the observed growth pattern.
(a) Plot the data (using plot(Loblolly)) and verify that the same
growth pattern is observed for all trees. The similarity can be
emphasized by putting all the curves into a single panel using
plot(Loblolly, outer = ~1). What is the most noticeable difference among the curves?
(b) Fit a separate asymptotic regression model to each tree using nlsList and SSasymp. Notice that no starting values are
needed for the parameters, as they are automatically produced
by SSasymp.
302
(c) Plot the condence intervals on the individual parameters from

the nlsList t. Can you identify the ones that vary signicantly
with tree?
(d) Use the nlsList object to t an NLME model with random effects for all parameters (you can use nlme(object), with object
replaced with the name of the nlsList object). Can you evaluate the condence intervals on the NLME parameters (using
intervals(object))? The error message indicates that the maximum likelihood estimates do not correspond to a numerically
stable solution (this generally occurs when the random-eects
model is overparameterized).
(e) Update the previous nlme t using a diagonal structure for
(use random = pdDiag(Asym+R0+lrc ~ 1) in the call to update).
Print the results and examine the estimated standard errors of
the random eects. Which random eect can be dropped from
the model?
(f) Update the nlme t eliminating the random eect you identied
in the previous item (continue to use a diagonal ). Compare
this t to the one obtained in the previous item using anova.
What do you conclude? Can you drop further random eects
from the model?
(g) Plot the augmented predictions (using plot(augPred(object)))
for the nal model obtained in the previous item. Does the model
seem adequate?
2. Davidian and Giltinan (1995, 1.1, p. 2) describe a pharmacokinetic
study of the drug cefamandole in which plasma concentrations of
cefamandole at 14 time points following intravenous infusion were
collected on 6 healthy volunteers. These data are available in nlme
as the object Cefamandole and are described in greater detail in Appendix A.4.
(a) Plot the cefamandole plasma concentrations versus time by subject (use plot(Cefamandole). The concentration-time proles are
well described by the biexponential model (6.2) used with the
Indometh data in 6.2.
(b) Use the self-starting function SSbiexp (Appendix C.4) in conjunction with nlsList to produce separate ts per subject. Plot
the individual condence intervals and identify the coecients
that seem to vary signicantly with subject.
(c) Fit an NLME model to the data with random eects for all
coecients, using the nlsList object produced in the previous
item. Because of the small number of subjects, convergence is
Exercises
303
not attained for a model with a general (try it). Use a diagonal structure for (random = pdDiag(A1+lrc1+A2+lrc2~1)) and
examine the estimated standard errors for the random eects.
Can any of the random eects be dropped from the model? Does
this agree with your conclusions from the plot of the condence
intervals for the nlsList t?
(d) Update the nlme t in the previous item according to your previous conclusions for the random-eects model. Use anova to
compare the two models. Produce the condence intervals for
the parameters in the updated model. Do you think any further
random eects can be dropped from the model? If so, update
the t and compare it to the previous model using anova.
(e) Plot the standardized residuals versus the tted values for the
nal model obtained in the previous item. Do you observe any
patterns that contradict the models assumptions? Plot the observed concentrations versus the tted values by subject. Does
the tted model produce sensible predictions?
3. Data on the intensity of 23 large earthquakes in western North America between 1940 and 1980 were reported by Joyner and Boore (1981).
The data, included in the object Earthquake , are described in more
detail in Appendix A.8. The objective of the study was to predict the
maximum horizontal acceleration (accel) at a given location during
a large earthquake based on the magnitude of the quake (Richter)
and its distance from the epicenter (distance). These data are analyzed in Davidian and Giltinan (1995, 11.4, pp. 319326). The model
proposed by Joyner and Boore (1981) can be written as
log10 (accel) = 1 + 2 Richter

log10 distance2 + exp(3 ) 4 distance2 + exp(3 ).
(a) Plot the data and verify that acceleration measurements are
sparse or noisy for most of the quakes. No common attenuation
pattern is evident from the plot.
(b) No self-starting function is available in nlme for the Earthquake
data model. The estimates reported in Davidian and Giltinan
(1995) (1 = 0.764, 2 = 0.218, 3 = 1.845, and 4 = 5.657
103 ) can be used as initial estimates for an nlsList t (use
start = c(phi1 = -0.764, phi2 = 0.218, phi3 = 1.845, phi4 =
0.005657). However, due to sparse and noisy nature of the data,
convergence is not attained for any of the quakes in the nlsList
t (verify it).
(c) Fit an NLME model to the data with random eects for all
coecients and a diagonal , using as starting values for the
304
xed eects the estimates reported in Davidian and Giltinan

(1995) (listed in the previous item). Note that, even though the
model did not converge for any of the individual quakes in the
nlsList t, it converged for the combined data in the nlme t.
The individual quakes borrow strength from each other in the
nlme t.
(d) Examine the estimated standard errors for the random eects
in the nlme t relative to the absolute value of the estimated
xed eects. Can any of the random eects be dropped from
the model? If so, ret the model with fewer random eects an
compare it to the previous model using anova. Repeat the procedure until no further random eects can be removed from the
model. (You may have to use nls to t a model with no random
eects, which can also be compared to an nlme t using anova,
provided the nlme object comes rst in the argument list.)
(e) Examine the plot of the standard residuals versus the tted values and the normal plot of the standardized residuals for the
nal model obtained in the previous item. Are there any apparent departures from the models assumptions? Plot the observed
log10(accel) measurements versus the tted values. How well
does the model predict the accelerations?
(f) One of the questions of interest in this study was the possible effect of soil type (represented in Earthquake by the indicator variable soil, taking values 0 for rock and 1 for soil) on acceleration. Update the nal nlme model obtained before to include a
soil eect for the xed eects of coecients with an associated
random eect. For example, if phi4 is the only coecient with
a random eect, you can use fixed = list(phi1+phi2+phi3 ~ 1,
phi4 ~ soil). You will also need to provide initial estimates for
the xed eects, using for example start = c(fixef(object),
0), where object should be replaced with the name of the nlme
object being updated. Use summary to test the signicance of soil
type.
7
Theory and Computational Methods
for Nonlinear Mixed-Eects Models
This chapter presents the theory for the nonlinear mixed-eects model introduced in Chapter 6. A general formulation of NLME models is presented
and illustrated with examples. Estimation methods for tting NLME models, based on approximations to the likelihood function, are described and
discussed. The computational methods used in the nlme function to t
NLME models are also described. An extended class of nonlinear regression models with heteroscedastic, correlated errors, but with no random
eects, is presented.
The objective of this chapter is to give an overall description of the
theoretical and computational aspects of NLME models so as to allow one
to evaluate the strengths and limitations of such models in practice. It is not
the purpose of this chapter to present a thorough theoretical description of
NLME models. Such a comprehensive treatment of the theory of nonlinear
mixed-eects models can be found, for example, in Davidian and Giltinan
(1995) and in Vonesh and Chinchilli (1997).
Readers who are more interested in the applications of NLME models
and the use of the functions and methods in the nlme library to t such
models can, without loss of continuity, skip this chapter and go straight
to Chapter 8. If you decide to skip this chapter at a rst reading, it is
recommended that you return to it (especially 7.1) at a later time to get a
good understanding of the NLME model formulation and its assumptions
and limitations.
306
7. Theory and Computational Methods for NLME Models
7.1 The NLME Model Formulation

Nonlinear mixed-eects models are mixed-eects models in which some, or
all, of the xed and random eects occur nonlinearly in the model function.
They can be regarded either as an extension of linear mixed-eects models
in which the conditional expectation of the response given the random
eects is allowed to be a nonlinear function of the coecients, or as an
extension of nonlinear regression models for independent data (Bates and
Watts, 1988) in which random eects are incorporated in the coecients to
allow then to vary by group, thus inducing correlation within the groups.
This section presents a general formulation for NLME models proposed
by Lindstrom and Bates (1990). The NLME model for single-level grouped
data, which includes repeated measures and longitudinal data, is presented
in 7.1.1. The multilevel NLME model is described in 7.1.2.
7.1.1 Single-Level of Grouping

By far the most common application of NLME models is for repeated measures datain particular, longitudinal data. The nonlinear mixed-eects
model for repeated measures proposed by Lindstrom and Bates (1990) can
be thought of as a hierarchical model. At one level the jth observation on
the ith group is modeled as
yij = f (ij , v ij ) + ij ,
i = 1, . . . , M, j = 1, . . . , ni ,
(7.1)
where M is the number of groups, ni is the number of observations on the

ith group, f is a general, real-valued, dierentiable function of a groupspecic parameter vector ij and a covariate vector v ij , and ij is a normally distributed within-group error term. The function f is nonlinear in
at least one component of the group-specic parameter vector ij , which
is modeled as
ij = Aij + B ij bi ,
bi N (0, ),
(7.2)
where is a p-dimensional vector of xed eects and bi is a q-dimensional

random eects vector associated with the ith group (not varying with j)
with variancecovariance matrix . The matrices Aij and B ij are of appropriate dimensions and depend on the group and possibly on the values
of some covariates at the jth observation. This model is a slight generalization of that described in Lindstrom and Bates (1990) in that Aij and
B ij can depend on j. This generalization allows the incorporation of timevarying covariates in the xed eects or the random eects for the model.
It is assumed that observations corresponding to dierent groups are independent and that the within-group errors ij are independently distributed
as N (0, 2 ) and independent of the bi . The assumption of independence
307
and homoscedasticity for the within-group errors can be relaxed, as shown

in 7.4.
Because f can be any nonlinear function of ij , the representation of
the group-specic coecients ij could be chosen so that Aij and B ij are
always simple incidence matrices. However, it is desirable to encapsulate as
much modeling of the ij as possible in this second stage, as this simplies
the calculation of the derivatives of the model function with respect to
and bi , used in the optimization algorithm. In a call to nlme the arguments
fixed and random are used to specify the Aij and B ij matrices, respectively.
We can write (7.1) and (7.2) in matrix form as
y i = f i (i , v i ) + i ,
i = Ai + B i bi ,
(7.3)
for i = 1, . . . , M , where
yi1
i1
i1
f (i1 , v i1 )
..
y i = ... , i = ... , i = ... , f i (i , v i ) =
,
.
f (ini , v ini )
yini
ini
ini
v i1
Ai1
B i1
(7.4)
v i = ... , Ai = ... , B i = ... .
v ini
Aini
B ini
We use the examples of Chapter 6 to illustrate the general NLME model

formulation.
Indomethicin Kinetics
The nal model obtained in 6.2 for the indomethicin data, represented by
the object fm4Indom.nlme, expresses the concentration measurement yij for
the ith subject at time tj as
yij = 1i exp [ exp (2i ) tj ] + 3i exp [ exp (4i ) tj ] + ij ,

1 0 0 0 1
1 0 0
1i
2i 0 1 0 0 2 0 1 0 b1i
=

3i 0 0 1 0 3 + 0 0 1 b2i ,
b3i
4i
4
0 0 0 1
0 0 0 . /0
1
. /0 1 .
/0
1 . /0 1 .
/0
1 bi
ij
Aij
11
bi N 0, 12
0
12
22
0
0
0 ,
33
B ij

ij N 0, 2 .
In this case, the individual coecients ij and the design matrices Aij = I
and B ij do not vary with time. The matrix for the random eects is
block-diagonal.
308
Growth of Soybean Plants

The tted object fm4Soy.nlme represents the nal model for the soybean
data obtained in 6.3. We use plot 1990P8, for which Year = 1990 and
Variety = P, to illustrate the general NLME model representation, expressing the average leaf weight yij for plot i at tij days after planting as
yij =
1i
+ ij ,
1 + exp [ (tij 2i ) /3i ]
1
2

3

4

5
1i
1 0 1 1 0 1 0 0 0 0 0 0 0
1 $ %
6
2i = 0 0 0 0 0 0 1 0 1 1 0 0 0 7 + 0 b1i ,

3i
8
0 0 0 0 0 0 0 0 0 0 1 0 1
0 . /0 1
. /0 1 .
/0
1
9 ./01 bi
B ij
Aij
ij
10

11

12
13
. /0 1
bi N (0, ) ,

ij |i N 0, 2 [E (yij |i )] .
The correspondence between the xed eects, , and the coecient

names used in fm4Soy.nlme is: 1 = Asym.(Intercept), 2 = Asym.Year1989,
3 = Asym.Year1990, 4 = Asym.Variety, 5 = Asym.Year1989Variety, 6 =
Asym.Year1990Variety, 7 = xmid.(Intercept), 8 = xmid.Year1989, 9 =
xmid.Year1990, 10 = xmid.Variety, 11 = scal.(Intercept), 12 = scal.Year1989, and 13 = scal.Year1990. The design matrices Aij and B ij do
not vary with j in this example and, as a result, neither do the coecients
ij . The use of variance functions for the within-group errors is discussed
in 7.4, when we present extensions to the basic NLME model.
Clinical Study of Phenobarbital Kinetics

The nal model for the phenobarbital data, represented in 6.4 by the object fm3Pheno.nlme, includes only the infants birth weight wi as a covariate
for the xed eects. The phenobarbital concentration yij for individual i
measured at time tij , following intravenous injections of dose Did at times
309
tid , is expressed as
yij =
"
Did
exp [ exp (lCl i lV i ) (tij tid )] + ij ,
exp (lV i )
d:tid <tij

1

1 wi 0 0
1 0 b1i
2
lCl i
=
+
,
0 1 b2i
lV i
0 0 1 w i 3
. /0 1 . /0 1
/0
1
. /0 1 .
4
Aij
B ij
ij
bi
. /0 1

bi N 0, 11
0
0
22

,

ij N 0, 2 .
The correspondence between the xed eects, , and the coecient names
in the fm3Pheno.nlme object is: 1 = lCl.(Intercept), 2 = lCl.Wt, 3 =
lV.(Intercept), and 4 = lV.Wt. A diagonal matrix is used to represent
the independence between the random eects.
7.1.2 Multilevel NLME Models

The single-level NLME model (7.1) can be extended to data grouped according to multiple, nested factors by modifying the model for the random
eects in (7.2). For example, the multilevel version of the Lindstrom and
Bates (1990) model for two levels of nesting is written as a two-stage model
in which the rst stage expresses the response yijk for the kth observation
on the jth second-level group of the ith rst-level group as
yijk = f (ijk , v ijk ) + ijk ,
i = 1, . . . , M,
j = 1, . . . , Mi ,
k = 1, . . . , nij ,
(7.5)
where M is the number of rst-level groups, Mi is the number of secondlevel groups within the ith rst-level group, nij is the number of observations on the jth second-level group of the ith rst-level group, and ijk is a
normally distributed within-group error term. As in the single-level model,
f is a general, real-valued, dierentiable function of a group-specic parameter vector ijk and a covariate vector v ijk . It is nonlinear in at least
one component of ij . The second stage of the model expresses ij as
ijk = Aijk + B i,jk bi + B ijk bij ,
bi N (0, 1 ),
bij N (0, 2 ).
(7.6)
As in the single-level model (7.2), is a p-dimensional vector of xed

eects, with design matrix Aijk , which may incorporate time-varying covariates. The rst-level random eects bi are independently distributed
310
q1 -dimensional vectors with variancecovariance matrix 1 . The secondlevel random eects bij are q2 -dimensional independently distributed vectors with variancecovariance matrix 2 , assumed to be independent of
the rst-level random eects. The random eects design matrices B i,jk
and B ijk depend on rst- and second-level groups and possibly on the
values of some covariates at the kth observation. The within-group errors
ijk are independently distributed as N (0, 2 ) and are independent of the
random eects. The assumption of independence and homoscedasticity for
the within-group errors can be relaxed, as shown in 7.4.
We can express (7.5) and (7.6) in matrix form as

y ij = f ij ij , v ij + ij ,
(7.7)
ij = Aij + B i,j bi + B ij bij ,
for i = 1, . . . , M, j = 1, . . . , Mi , where
ij1
yij1
ij1
y ij = ... , ij = .. , ij = ... ,

ijnij
yijnij
ijnij
f (ij1 , v ij1 )
v ij1

.
..
f ij ij , v ij =
, v ij = .. ,
.
f (ijnij , v ijnij )
v ijnij
Aij1
B i,j1
B ij1
Aij = ... , B i,j = ... , B ij = ... .

Aijnij
B i,jnij
B ijnij
Extensions of the NLME model to more than two levels of nesting are
straightforward. For example, with three levels of nesting the second-stage
model for the group-specic coecients is
ijkl = Aijkl + B i,jkl bi + B ij,kl bij + B ijkl bijk ,
bi N (0, 1 ),
bij N (0, 2 ),
bijk N (0, 3 ).
7.1.3 Other NLME Models

The rst developments of nonlinear mixed-eects models appear in Sheiner
and Beal (1980). Their models and estimation methods are incorporated in
the NONMEM program (Beal and Sheiner, 1980), which is widely used in
pharmacokinetics. They introduced a model similar to (7.1) and developed
a maximum likelihood estimation method based on a rst-order Taylor
expansion of the model function around 0, the expected value of the random
eects vector b.
311
A nonparametric maximum likelihood method for nonlinear mixed-eects

models was proposed by Mallet, Mentre, Steimer and Lokiek (1988). They
use a model similar to (7.1), but make no assumptions about the distribution of the random eects. The conditional distribution of the response
given the random eects is assumed to be known. The objective of the estimation procedure is to get the probability distribution of the group-specic
coecients (ij ) that maximizes the likelihood of the data. Mallet (1986)
showed that the maximum likelihood solution is a discrete distribution
with the number of discontinuity points less than or equal to the number
of groups in the sample. Inference for this model is based on the maximum likelihood distribution from which summary statistics (e.g., means
and variancecovariance matrices) and plots are obtained.
Davidian and Gallant (1992) introduced a smooth, nonparametric maximum likelihood estimation method for nonlinear mixed eects. Their model
is similar to (7.1), but with a more general denition for the group-specic
coecients, ij = g(, bi , v ij ), where g is a general, possibly nonlinear
function. As in Mallet et al. (1988), Davidian and Gallant assume that the
conditional distribution of the response vector given the random eects is
known (up to the parameters that dene it), but the distribution of the
random eects is free to vary within a class of smooth densities dened in
Gallant and Nychka (1987).
A Bayesian approach using hierarchical models for nonlinear mixed effects is described in Bennett and Wakeeld (1993) and Wakeeld (1996).
The rst stage model is similar to (7.1) and the distributions of both the
random eects and the errors ij are assumed known up to population parameters. Prior distributions for the population parameters must be provided then Markov-chain Monte Carlo methods, such as the Gibbs sampler
(Geman and Geman, 1984) or the Metropolis algorithm (Hastings, 1970),
are used to approximate the posterior density of the random eects.
Vonesh and Carter (1992) developed a mixed-eects model that is nonlinear in the xed eects, but linear in the random eects. Their model
is
y i = f (, v i ) + Z i ()bi + i ,
where, as before, , bi , and i denote, respectively, the xed eects, the
random eects, and the within-group error term, v i is a matrix of covariates, and Z i is a full-rank matrix of known functions of the xed eects .
It is further assumed that bi N (0, ), i N (0, 2 I), and the two vectors are independent. In some sense, Vonesh and Carter incorporate in the
model the approximations suggested by Sheiner and Beal (1980) and Lindstrom and Bates (1990). Their approach concentrates more on inferences
about the xed eects, and less on the variancecovariance components of
the random eects.
312
7.2 Estimation and Inference in NLME Models

Dierent methods have been proposed to estimate the parameters in the
NLME model described in 7.1.1 and 7.1.2. In this book we restrict ourselves to methods based on the likelihood function. Descriptions and comparisons of other estimation methods proposed for NLME models can be
found, for example, in Ramos and Pantula (1995), Davidian and Giltinan
(1995), and Vonesh and Chinchilli (1997).
7.2.1 Likelihood Estimation

Because the random eects are unobserved quantities, maximum likelihood
estimation in mixed-eects models is based on the marginal density of the
responses y, which, for a model with Q levels of nesting, is calculated as

p y|, 2 , 1 , . . . , Q = p y|b, , 2 p (b|1 , . . . , Q ) db, (7.8)

where p y|, 2 , 1 , . . . , Q is the marginal density of y, p(y|b, , 2 ) is
the conditional density of y given the random eects b, and the marginal
distribution of b is p (b|1 , . . . , Q ). For the NLME model (7.1), expressing the random eects variancecovariance matrix in terms of the precision
factor , so that 1 = 2 T as described in 2.1.1, provides the
marginal density of y as

p y|, 2 , =
M
||
(2 2 )
(N +M q)/2
i=1
'
exp
y i f i (, bi ) + bi
2 2
2
dbi ,
(7.9)
where f i (, bi ) = f i [i (, bi ) , v i ].
Because the model function f can be nonlinear in the random eects, the
integral in (7.8) generally does not have a closed-form expression. To make
the numerical optimization of the likelihood function a tractable problem,
dierent approximations to (7.8) have been proposed. Some of these methods consist of taking a rst-order Taylor expansion of the model function f
around the expected value of the random eects (Sheiner and Beal, 1980;
Vonesh and Carter, 1992), or around the conditional (on ) modes of the
random eects (Lindstrom and Bates, 1990). Gaussian quadrature rules
have also been used (Davidian and Gallant, 1992).
We describe three dierent methods for approximating the likelihood
function in the NLME model. The rst, proposed by Lindstrom and Bates
(1990), approximates (7.8) by the likelihood of a linear mixed-eects
model. We call this the LME approximation. It is the basis of the estimation algorithm currently implemented in the nlme function. The second
method uses a Laplacian approximation to the likelihood function, and
313
the last method uses an adaptive Gaussian quadrature rule to improve the
Laplacian approximation. The LME, Laplacian, and adaptive Gaussian approximations have increasing degrees of accuracy, at the cost of increasing
computational complexity. The three approximations to the NLME likelihood are discussed and compared in Pinheiro and Bates (1995).
Lindstrom and Bates Algorithm
The estimation algorithm described by Lindstrom and Bates (1990) alternates between two steps, a penalized nonlinear least squares (PNLS) step,
and a linear mixed eects (LME) step, as described below. We initially
consider the alternating algorithm for the single-level NLME model (7.1).
In the PNLS step, the current estimate of (the precision factor) is
held xed, and the conditional modes of the random eects bi and the
conditional estimates of the xed eects are obtained by minimizing a
penalized nonlinear least squares objective function
M
"
y i f i (, bi ) + bi

.
(7.10)
i=1
The LME step updates the estimate of based on a rst-order Taylor

expansion of the model function f around the current estimates of and
the conditional modes of the random eects bi , which we will denote by
(w)
(w) and
b , respectively. Letting
i

(w)
f i
&
,
Xi =
T (w) ,b(w)
i
(w)
i
w

(w)
f i

Zi =
,
bTi (w) ,b(w)
i
(7.11)
(w) (w)
(w)
(w) ,

(w)
&i
(w)
= y i f i (
bi ) + X
+Z
i bi ,
the approximate log-likelihood function used to estimate is

M

1"

N
{log |i ()|
LME , 2 , | y = log 2 2
2
2 i=1
T

2

(w)
(w)
(w)
(w)
&i 1 () w
&i
i X
i X
+ 2 w
,
i
(7.12)
1 T (w)
(w)
where i () = I + Z
Zi
. This log-likelihood is identical to
i
that of a linear mixed-eects model in which the response vector is given
(w) and the xed- and random-eects design matrices are given by
by w
(w)
&
(w) , respectively. Using the results in 2.2, one can express the
X
and Z
optimal values of and 2 as functions of and work with the proled
log-likelihood of , greatly simplifying the optimization problem.
T
314
Lindstrom and Bates (1990) also proposed a restricted maximum likelihood estimation method for , which consists of replacing the log-likelihood
in the LME step of the alternating algorithm by the log-restricted-likelihood
2

R
LME , | y =

M

1"
2 (w)T 1
(w)
2

&
&

log X i
i ()X i . (7.13)
LME () , , | y
2 i=1
(w)
(w)
(w) and
&i depends on both
Note that, because X
bi , changes in either the xed eects model or the random eects model imply changes in
the penalty factor for the log-restricted-likelihood (7.13). Therefore, logrestricted-likelihoods from NLME models with dierent xed or random
eects models are not comparable.
The algorithm alternates between the PNLS and LME steps until a convergence criterion is met. Such alternating algorithms tend to be more
ecient when the estimates of the variancecovariance components (
and 2 ) are not highly correlated with the estimates of the xed eects
(). Pinheiro (1994) has shown that, in the linear mixed-eects model, the
maximum likelihood estimates of and 2 are asymptotically independent
of the maximum likelihood estimates of . These results have not yet been
extended to the nonlinear mixed-eects model (7.1).
Lindstrom and Bates (1990) only use the LME step to update the estimate of . However, the LME step also produces updated estimates of
and the conditional modes of bi . Thus, one can iterate LME steps by
re-evaluating (7.11) and (7.12) (or (7.13) for the log-restricted-likelihood)
at the updated estimates of and bi , as described in Wolnger (1993).
Because the updated estimates correspond to the values obtained in the
rst iteration of a GaussNewton algorithm for the PNLS step, iterated
LME steps will converge to the same values as the alternating algorithm,
though possibly not as quickly.
Wolnger (1993) also shows that, when a at prior is assumed for , the
LME approximation to the log-restricted-likelihood (7.13) is equivalent to a
Laplacian approximation (Tierney and Kadane, 1986) to the integral (7.9).
The alternating algorithm and the LME approximation to the NLME loglikelihood can be extended to multilevel models. For example, for an NLME
model with two levels of nesting, the PNLS step consists of minimizing the
penalized nonlinear least-squares function
Mi
M "

"
y ij f ij (, bi , bij )2 + 2 bij 2 + 1 bi 2
(7.14)
i=1
j=1
to obtain estimates for the xed eects and the conditional (on 1 and
2 ) modes of the random eects bi and bij .
315
Letting
(w)
&
X
ij =

f ij

T
(w)
(w)
,bi
(w)
,bij
(w)
Z
ij =
(w)
Z
i,j =

f ij

bTij

f ij

bT
i
,
(w)
(w)
,bi
(w)
,bij
,
(w)
(w)
,bi
(w)
,bij
(w) (w)
(w) (w)
(w)
(w) ,

(w) + Z
(w)
&ij
(w)
(w)
ij = y ij f ij (
w
bi ,
bij ) + X
+Z
i,j bi
ij bij ,
(w)
(w)
(w)

&
i1
Z
w
X
i,1
i1
(w)
(w) = .. , w
&(w) = .. , Z
i =
X
i
i
.
.
.. ,
(w)
(w)
(w)

iMi
w
&
Z
X
iMi
i,Mi
(7.15)
the approximate log-likelihood function used to estimate 1 and 2 in the
two-level NLME models is
M

1"

N
LME , 2 , 1 , 2 | y = log 2 2
{log |i (1 , 2 )|
2
2 i=1
T

2

(w)
(w)
(w)
(w)
&i 1 (1 , 2 ) w
&i
i X
i X
+ 2 w
,
i
T
6 i (w) 1 T (w)T
1 T (w)
(w)
+ M
where i (1 , 2 ) = I+Z
i 1 1 Z i
j=1 Z ij 2 2 Z ij
and denotes the direct sum operator. The corresponding log-restrictedlikelihood is

2

2

R
LME , 1 , 2 | y = LME (1 , 2 ) , , 1 , 2 | y

M
2 (w)T 1
(w)
1"
&
&

log X i
i (1 , 2 )X i .
2 i=1
This formulation can be extended to multilevel NLME models with an

arbitrary number of levels.
The alternating algorithm is the only estimation algorithm used in the
nlme function. It is implemented for maximum likelihood and restricted
maximum likelihood estimation in single and multilevel NLME models.
Laplacian Approximation
Laplacian approximations are used frequently in Bayesian inference to estimate marginal posterior densities and predictive distributions (Tierney
and Kadane, 1986; Leonard, Hsu and Tsui, 1989). These techniques can
also be used for approximating the likelihood function in NLME models.
316
We consider initially the single-level NLME models. The integral that

we want to estimate to obtain the marginal distribution of y i in (7.9) can
be written as

(ni +q)/2
%
$
2
2 2
p(y i | , , ) =
|| exp g(, , y i , bi )/2 2 dbi ,
where g(, , y i , bi ) = y i f i (, bi )2 +bi 2 , the sum of which is the
objective function for the PNLS step of the alternating algorithm dened
in (7.10). Let

bi =
bi (, , y i ) = arg min g(, , y i , bi ),
bi
g(, , y i , bi )
g (, , y i , bi ) =
,
bi
2 g(, , y i , bi )
g (, , y i , bi ) =
,
bi bTi
(7.16)
and consider a second-order Taylor expansion of g around

bi

1
T

g (, , y i , bi ) g , , y i ,
bi +
bi g , , y i ,
bi bi
bi .
bi
2
(7.17)
bi ) = 0.)
(The linear term in the expansion vanishes because g (, , y i ,
The Laplacian approximation is dened as

p y | , 2 ,
7
8
M
"
N

M
2 2
1

|| exp 22
g(, , y i , bi )
2
i=1
+
M

T
9

q

2 2

1

exp 22 bi bi g (, , y i , bi ) bi bi dbi
2
i=1
7
8M
M
1
"

N

2
M
2 2

1

= 2
|| exp 22
g(, , y i ,
bi )
(,
,
y
,
b
)
.
g
i i
i=1
i=1
The Hessian

2 f i (, bi )
y i f i (,
bi =
bi )
g , , y i ,

T
bi bi
bi

f i (, bi ) f i (, bi )
T
+

+
bi
bT
bi
bi
involves second derivatives of f but, at

bi , the contribution of

2 f i (, bi )
y i f i (,
bi )

T
bi bi
bi
317

is usually negligible compared to that of f i (, bi )/bi |bi f i (, bi )/bTi
(Bates and Watts, 1980). Therefore, we use the approximation

g , , y i ,
bi G (, , y i )

f i (, bi ) f i (, bi )
=
+ T .

T
bi
bi
bi
bi
bi
(7.18)
This approximation is similar to that used in the GaussNewton algorithm

for nonlinear least squares and has the advantage of requiring only the rstorder partial derivatives of f with respect to the random eects. These
are usually available as a by-product of the estimation of
bi , which is a
penalized least squares problem, for which standard and reliable code is
available.
The modied Laplacian approximation to the log-likelihood of the singlelevel NLME model (7.1) is then given by

N
LA , 2 , , | y = log 2 2 + M log ||
2
'M
2
M

"
"
1
log |G (, , y i )| + 2
g , , y i ,
bi
. (7.19)
2 i=1
i=1
Because
bi does not depend on 2 , for given and the maximum
likelihood estimate of 2 (based upon LA ) is
2 (, , y) =
2 =
M

"
g , , y i ,
bi /N.
i=1
We can prole LA on 2 to reduce the dimension of the optimization

problem, obtaining
LAp (, ) =
M
2 %
1"
N$
+ M log ||
1 + log (2) + log

log |G (, , y i )| .
2
2 i=1
If the model function f is linear in the random eects, then the modied
Laplacian approximation is exact because the second-order Taylor expansion in (7.17) is exact when f i (, bi ) = f i () + Z i () bi .
There does not yet seem to be a straightforward generalization of the
concept of restricted maximum likelihood to NLME models. The diculty
is that restricted maximum likelihood depends heavily upon the linearity
of the xed eects in the model function, which does not occur in nonlinear
models. Lindstrom and Bates (1990) circumvented that problem by using
318
an approximation to the model function f in which the xed eects, ,

occur linearly. This cannot be done for the Laplacian approximation, unless
we consider yet another Taylor expansion of the model function, what would
lead us back to something very similar to Lindstrom and Batess approach.
The Laplacian approximation (7.19) can be extended to multilevel NLME
models. For example, in a two-level NLME model, let
bi
bi1
..
.
=
baug
,
i
biMi
i = 1, . . . , M
denote the augmented random eects vector for the ith rst-level group,
containing the rst-level random eects bi and all the second-level random
eects bij pertaining to rst-level i. The two-level NLME likelihood can
then be expressed as
p(yi | , 2 , 1 , 2 ) =
(ni +q1 +Mi q2 )/2

2 2
|1 | |2 |Mi
% aug
$
2
exp g(, 1 , 2 , y i , baug
i )/2 dbi ,
where g(, 1 , 2 , y i , baug

i ) is the objective function for the PNLS step in
the alternating algorithm for two-level NLME models, dened in (7.14).
We proceed as in the single-level case and dene
aug
aug

bi (, 1 , 2 , y i ) = arg min
g(, 1 , 2 , y i , baug
bi =
i ),
aug
bi
g(, 1 , 2 , y i , baug
i )
=
,
g
baug
i
2 g(, 1 , 2 , y i , baug
i )
g (, 1 , 2 , y i , baug
,
i )=
aug
aug T
bi (bi )
(, 1 , 2 , y i , baug
i )
to obtain the second-order approximation

aug

)

g
,
,
y
,
b
g (, 1 , 2 , y i , baug
1
2
i i
i

T
aug
aug
aug
1 aug
+
g , 1 , 2 , y i ,
bi

bi
baug
.
bi bi
i
2
T
We note that 2 g(, 1 , 2 , y i , baug
i )/bij bik = 0 for any j = k and
use the same reasoning as in (7.18), to approximate the matrix g (, , y i ,
bi )
319
by

G1 G2
g , 1 , 2 y i ,
bi G (, 1 , 2 , y i ) =
, where
GT2 G3

f i (, baug

f i (, baug
i )
i )
+ T1 1 ,
G1 =

T
aug
bi
bi
bi
bi

f iMi (,bi ,biMi )
)
f i (, baug
f
(,b
,b
)
i i1
i
i1

G2 =
aug ,
T
T
bi1
biM

b
i
bi
bi
i

Mi

:
f ij (, bi , bij )
f ij (, bi , bij )
T

+
G3 =
,

2
2

bij
bTij
bi ,bij
j=1
bi ,bij
f i1 (, bi , bi1 )
..
f i (, baug
.
i )=
.
f iMi (, bi , biMi )
The modied, proled Laplacian approximation to the log-likelihood of
the two-level NLME model is then given by
2 %
N$
1 + log (2) + log

+ M log |1 |
2
M
M
"
1"
Mi log |2 |
log |G (, 1 , 2 , y i )| ,
+
2 i=1
i=1
LAp (, 1 , 2 ) =

M
aug
/N. This formulation can be exwhere
2 = i=1 g , 1 , 2 , y i ,
bi
tended to multilevel NLME models with an arbitrary number of levels.
The Laplacian approximation generally gives more accurate estimates
than the alternating algorithm, as it uses an expansion around the estimated random eects only, while the LME approximation in the alternating algorithm uses an expansion around the estimated xed and random
eects. Because it requires solving a dierent penalized nonlinear leastsquares problem for each group in the data and its objective function cannot be proled on the xed eects, the Laplacian approximation is more
computationally intensive than the alternating algorithm. The algorithm
for calculating the Laplacian approximation can be easily parallelized, because the individual PNLS problems are independently optimized.
Adaptive Gaussian Approximation
Gaussian quadrature rules are used to approximate integrals of functions
with respect to a given kernel by a weighted average of the integrand evaluated at predetermined abscissas. The weights and abscissas used in Gaussian quadrature rules for the most common kernels can be obtained from
the tables of Abramowitz and Stegun (1964) or by using an algorithm
proposed by Golub (1973) (see also Golub and Welsch (1969)). Gaussian
320
quadrature rules for multiple integrals are known to be numerically complex

(Davis and Rabinowitz, 1984), but using the structure of the integrand in
the nonlinear mixed-eects model we can transform the problem into successive applications of simple one-dimensional Gaussian quadrature rules.
We consider initially the single-level NLME model.
A natural candidate for the kernel function for the quadrature rule in the
single-level NLME model is the marginal distribution of the random eects,
that is, the N (0, ) density. The Gaussian quadrature rule in this case can
be viewed as a deterministic version of a Monte Carlo integration algorithm
in which random samples of the random eects, bi , are generated from
the N (0, ) distribution. The samples and the weights in the Gaussian
quadrature rule are xed beforehand, while in Monte Carlo integration
they are left to random choice. Because importance sampling tends to be
much more ecient than simple Monte Carlo integration (Geweke, 1989),
we consider an importance sample version of the Gaussian quadrature rule,
which we denote by adaptive Gaussian quadrature.
The critical step for the success of importance sampling is the choice
of an importance distribution that approximates the integrand. For the
single-level NLME model the integrand is proportional to
%
$
exp g (, , y i , bi ) /2 2 ,
which is approximated by a N (
bi , 2 G1 (, , y i )) density, with
bi dened
as in (7.16) and G(, , y i ) dened as in (7.18). This is the importance
distribution used in the adaptive Gaussian quadrature, so that the grid
bi
of abscissas in the bi scale is centered around the conditional modes
and G (, D, y i ) is used for scaling. Letting zj , wj j = 1, . . . , NGQ denote,
respectively, the abscissas and the weights for the (one-dimensional) Gaussian quadrature rule with NGQ points based on the N (0, 1) kernel, the
adaptive Gaussian quadrature rule is given by

$
%
exp g (, , y i , bi ) /2 2 dbi = q |G (, , y i )|1/2
;

<
2
exp g , , y i ,
bi + G1/2 (, , y i ) z /2 2 + z /2

2
exp z /2 dz
1/2
q |G (, , y i )|
NGQ
NGQ
<
;
"
"
exp g , , y i ,
bi + G1/2 (, , y i ) z j /2 2
j1 =1
jq =1
q

+ z j 2 /2
k=1
w jk ,
321
1/2
where [G (, , y i )]
denotes a square root of G (, , y i ) and z j =

T
zj1 , . . . , zjq .
The adaptive Gaussian approximation to the log-likelihood function in
the single-level NLME model is then

AGQ , 2 , | y = N2 log 2 2 + M log ||
M
"
"
log |G (, , y i )|

;
exp g , , y i ,
bi + G1/2 (, , y i ) z j /2 2
i=1
M
"
i=1
NGQ
log
1
2
q
<
z j /2
!
w jk
k=1
The one point (i.e., NGQ = 1) adaptive Gaussian quadrature approximation is simply the modied Laplacian approximation (7.19), because in this
case z1 = 0 and w1 = 1. The adaptive Gaussian quadrature also gives
the exact log-likelihood when the model function f is linear in the random
eects b.
The adaptive Gaussian approximation can be made arbitrarily accurate
q
grid
by increasing the number of abscissas, NGQ . However, because NGQ
points are used to calculate the adaptive Gaussian quadrature for each
group, it quickly becomes prohibitively computationally intensive as the
number of abscissas increases. In practice NGQ 7 generally suces and
NGQ = 1 often provides a reasonable approximation (Pinheiro and Bates,
1995).
The adaptive Gaussian approximation can be generalized to multilevel
NLME models, using the same steps as in the multilevel Laplacian approximation. For example, the adaptive Gaussian approximation to the
log-likelihood of a two-level NLME model is
M
"

N
Mi log |2 |
AGQ , 2 , 1 , 2 | y = log 2 2 + M log |1 | +
2
i=1
NGQ
M
M
;
"
"
"
1
log |G (, 1 , 2 , y i )| +
log
exp g , 1 , 2 , y i ,
2 i=1
i=1
j
!
i q2

< q1 +M

aug

w jk ,
bi + G1/2 (, 1 , 2 , y i ) z j /2 2 + z j 2 /2
k=1
T

where z j = zj1 , . . . , zjq1 +M q2 . In this case, the number of grid points for
q1 +Mi q2
the ith rst-level group is NGQ
, so that the computational complexity
of the calculations increases exponentially with the number of second-level
322
groups. This formulation can be extended to multilevel NLME models with

arbitrary number of levels.
7.2.2 Inference and Predictions

Because the alternating algorithm of Lindstrom and Bates (1990) is the
only estimation algorithm implemented in the nlme function, we restrict
ourselves for the rest of this section to inference and predictions using this
estimation algorithm. The results described here can be easily extended to
other likelihood estimation approaches, such as the Laplacian and adaptive
Gaussian approximations described in 7.2.1.
Inference
Inference on the parameters of an NLME model estimated via the alternating algorithm is based on the LME approximation to the log-likelihood
function, dened in 7.2.1. Using this approximation at the estimated values
of the parameters and the asymptotic results for LME models described in
2.3 we obtain standard errors, condence intervals, and hypothesis tests
for the parameters in the NLME model. We use the single-level NLME
model of 7.1.1 to illustrate the inference results derived from the LME approximation to the log-likelihood. Extensions to multilevel NLME models
are straightforward.
Under the LME approximation, the distribution of the (restricted) max of the xed eects is
imum likelihood estimators

N , 2
M
"
81
T
&i 1 X
&i
X
i
(7.20)
i=1
Ti , with X
i 1 T Z
&i and Z
i are dened as in (7.11).
where i = I + Z
The standard errors included in the summary method for nlme objects are
obtained from the approximate variancecovariance matrix in (7.20). The t
and F tests reported in the summary method and in the anova method with
a single argument are also based on (7.20). The degrees-of-freedom for t
and F tests are calculated as described in 2.4.2.
Now let denote an unconstrained set of parameters that determine the
precision factor . The LME approximation is also used to provide an
log
approximate distribution for the (RE)ML estimators (,
)T . We use
2
log in place of to give an unrestricted parameterization for which the
normal approximation tends to be more accurate.

1
N
, I (, ) ,
log
log

2
LMEp / T
2 LMEp / log T
,
I (, ) =
2
2
2
LMEp / log
LMEp / log
323
(7.21)
where LMEp = LMEp (, ) denotes the LME approximation to the loglikelihood, proled on the xed eects, and I denotes the empirical information matrix. The same approximate distribution is valid for the REML
estimators with LMEp replaced by the log-restricted-likelihood R
LME dened in (7.13).
In practice, and 2 are replaced by their respective (RE)ML estimates in the expressions for the approximate variancecovariance matrices
in (7.20) and (7.21). The approximate distributions for the (RE)ML estimators are used to produce the condence intervals reported in the intervals
method for nlme objects.
The LME approximate log-likelihood is also used to compare nested
NLME models through likelihood ratio tests, as described in 2.4.1. In the
case of REML estimation, only models with identical xed and random&i matrices depend on
eects structures can be compared, because the X
and the
both
bi . The same recommendations stated in 2.4.1, on the
use of likelihood ratio tests for comparing LME models, remain valid for
likelihood ratio tests (based on the LME approximate log-likelihood) for
comparing NLME models. Hypotheses on the xed eects should be tested
using t and F tests, because likelihood ratio tests tend to be anticonservative. Likelihood ratio tests for variancecovariance parameters tend to be
somewhat conservative, but are generally used to compare NLME models
with nested random eects structures. Information criterion statistics, for
example, AIC and BIC, based on the LME approximate log-likelihood are
also used to compare NLME models.
The inference results for NLME models based on the LME approximation
to the log-likelihood are approximately asymptotic, making them less
reliable than the asymptotic inference results for LME models described in
2.3.
Predictions
As with LME models, tted values and predictions for NLME models
may be obtained at dierent levels of nesting, or at the population level.
Population-level predictions estimate the expected response when the random eects are equal to their mean value, 0. For example, letting xh represent a vector of xed-eects covariates and v h a vector of other model
covariates, the population prediction for the corresponding response yh estimates f (xTh , v h ).
324
Predicted values at the k th level of nesting estimate the conditional expectation of the response given the random eects at levels k and with the
random eects at higher levels of nesting set to zero. For example, letting
z h (i) denote a vector of covariates corresponding to random eects associated with the i th group at the rst level of nesting, the level-1 predictions
estimate f (xTh + z h (i)T bi , v h ). Similarly, letting z h (i, j) denote a covariate vector associated with the j th level-2 group within the i th level-1 group,
the level-2 predicted values estimate f (xTh + z h (i)T bi + z h (i, j)T bij , v h ).
This extends naturally to an arbitrary level of nesting.
The (RE)ML estimates of the xed eects and the conditional modes of
the random eects, which are estimated Best Linear Unbiased Estimates
(BLUP s) of the random eects in the LME approximate log-likelihood,
are used to obtain predicted values for the response. For example, the
population, level-1, and level-2 predictions for yh are
v h ),
yh = f (xTh ,
T
yh (i) = f (xh + z h (i)T
bi , v h ),
T
T
yh (i, j) = f (xh + z h (i) bi + z h (i, j)T
bi,j , v h ).
7.3 Computational Methods

In this section we describe ecient computational methods for estimating the parameters in an NLME model using the alternating algorithm
presented in 7.2.1. The LME step of the alternating algorithm consists of
optimizing a linear mixed-eects log-likelihood, or log-restricted-likelihood,
for which ecient computational algorithms are discussed in 2.2.8. Therefore, we concentrate here on computational methods for the PNLS step
of the alternating algorithm, focusing initially on the single-level NLME
model.
The objective function optimized in the PNLS step of the alternating
algorithm for a single-level NLME model is the penalized sum of squares
M
"
y i f i (, bi )2 + bi

.
(7.22)
i=1
By adding pseudo observations to the data, we can convert (7.22) into a

simple nonlinear sum of squares. Dene the augmented response and model
function vectors

y
i = i ,
y
0

f i (, bi )
.
f i (, bi ) =
bi
325
The penalized sum of squares (7.22) can then be re-expressed as

M
"

y i f i (, bi )2 .
(7.23)
i=1
It follows from (7.23) that, conditional on , the estimation of and

bi in the PNLS step can be regarded as a standard nonlinear least-squares
problem. A common iterative estimation method for standard nonlinear
least-squares problems is the GaussNewton method (Bates and Watts,
1988, 2.2) wherein a nonlinear model f () is replaced by a rst-order
(w) as
Taylor series approximation about current estimates
(w)

f () f (

f
(w)
.
)+
T (w)
(w+1)
(w+1)
(w) for the wth iteration is
The parameter increment
=
calculated as the solution of the least-squares problem

2

f
(w)
(w)
y f (

.

T (w)
Step-halving is used at each GaussNewton iteration to ensure that the
updated parameter estimates result in a decrease of the objective function.
(w+1)
(w) +
That is, the new estimate is set to
and the corresponding value
(w) ,
of the objective function is calculated. If it is less than the value at
the value is retained and the algorithm proceed to the next step, or declares
(w+1)
(w) +
convergence. Otherwise, the new estimate is set to
/2 and the
procedure is repeated, with the increment being halved until a decrease in
the objective function is observed or some predetermined minimum step
size is reached.
The GaussNewton algorithm is used to estimate and the bi in the
PNLS step of the alternating algorithm. Because of the loosely coupled
structure of the PNLS problem (Soo and Bates, 1992), ecient nonlinear
least-squares algorithms can be employed.
The derivative matrices for the GaussNewton optimization of (7.23) are,
for i = 1, . . . , M ,

7
8
(w)
(w)
f i (, bi |)
&
X
i =
i
=X
,

(w) (w)
T
0
,bi
8
7
(w)
(w)
f i (, bi |)

i = Zi
,
=Z

(w) (w)
bTi
,b
i
326

(w)
&i and Z
(w)
with X
dened as in (7.11). The least-squares problem to be
i
solved at each GaussNewton iteration is

2
M
"

(w) (w)
(w)
(w)
(w)
(w)

y
Xi
Zi
bi bi

i f i , bi
i=1
or, equivalently,
M

"
(w) 2
(w)
(w)
i X
Z
b
w
i ,
i
i
where
i=1
(w)
i
w
(w)
i
w
,
0
(7.24)
(w) dened as in (7.11).

with w
We use the same matrix decomposition methods as in 2.2.3 to obtain
an ecient algorithm for solving (7.24). Consider rst the orthogonaltriangular decomposition
8
7

(w)
(w)
&
(w)
R11(i) R10(i) c1(i)

w
X
Z
i
i
i
,
(7.25)
= Q1(i)
0
R00(i) c0(i)
0
0
where the reduction to triangular form is halted after the rst q columns.
The numbering scheme used for the components in (7.25) is the same introduced for the LME model in 2.2.3. Because is assumed to be of full
rank, so is the upper-triangular matrix R11(i) in (7.25).
Forming another orthogonal-triangular decomposition
R00(1) c0(1)

..
.. = Q R00 c0
(7.26)
.
0
.
0
c1
R00(M ) c0(M )
and noticing that the Q1(i) and Q0 are orthogonal matrices, we can rewrite
(7.24) as
M
"

c1(i) R11(i) bi R10(i) 2 + c0 R00 |2 + c1 2 .
(7.27)
i=1
We assume that the R00 is of full-rank, in which case (7.27) is uniquely

minimized by the least-squares estimates
= R1 c0 ,
00

bi = R1
11(i) c1(i) R10(i) ,
i = 1, . . . , M.
(7.28)
The GaussNewton increments are then obtained as the dierence between

(w)
(w) and
bi .
the least-squares estimates (7.28) and the current estimates,
327
Step-halving is used to ensure that the new estimates result in a decrease

of the objective function (7.23).
The ecient GaussNewton algorithm described above can be extended
to multilevel PNLS optimization problems. For example, in the two-level
NLME model, the PNLS step consists in optimizing
Mi
M "

"
2
2
(7.29)
y ij f ij (, bi , bij )2 + 2 bij + 1 bi
i=1
j=1
over , bi , and bij .

The GaussNewton iteration is implemented by solving the least-squares
problem
Mi
M "
"
(w)
&(w) Z
(w) bi Z
(w) bij 2 + 2 bij 2 ] + 1 bi 2 ,
ij X
[w
ij
i,j
ij
i=1
j=1
(7.30)
(w)
(w) &
(w)
(w)
ij , X
with w
ij , Z i,j , and Z ij dened as in (7.15). To solve it eciently,
we rst consider the orthogonal-triangular decomposition
8
7

(w)
(w) X
&(w) w
(w) Z
R
R
c
R

22(ij)
21(ij)
20(ij)
2(ij)
Z
ij
ij
i,j
ij
,
= Q2(ij)
0
R11(i) R10(i) c1(i)
2
0
0
0
where the reduction to triangular form is halted after the rst q2 columns.
Because 2 is assumed of full rank, so is R22(ij) . We then form a second
orthogonal-triangular decomposition for each rst-level group
R10(i1)
c1(i1)
R11(i1)

..
..
..
R11(i) R10(i) c1(i)
.
.
.
,
= Q1(i)
0
R00(i) c0(i)
R
R
c
11(iMi )
10(iMi )
1(iMi )
where the reduction to triangular form is stopped after the rst q1 columns.
The 1 matrix is assumed of full rank and, as a result, so is R11(i) . A nal
orthogonal-decomposition, identical to (7.26), is then formed.
Because the matrices Q2(ij) , Q1(i) , and Q0 are orthogonal, (7.30) can be
re-expressed as
Mi
M "
"

c2(ij) R20(ij) R21(ij) bi R22(ij) bij 2
i=1
j=1

2 <
+ c1(i) R10(i) R11(i) bi + c0 R00 2 ,
328
which is uniquely minimized by the least-squares estimates

= R1 c0 ,
00

,

c
bi = R1
1(i)
10(i)
11(i)
i = 1, . . . , M,

bij = R1
22(ij) (c2(ij) R21(ij) bi R20(i) ), i = 1, . . . , M, j = 1, . . . , Mi .
(7.31)
The GaussNewton increments are then obtained as the dierence between
(w)
(w) ,
the least-squares estimates (7.31) and the current estimates
bi , and
(w)

bij , with step-halving used to ensure that the objective function (7.29) decreases. This extends naturally to multilevel NLME models with arbitrary
number of levels.
The eciency of the GaussNewton algorithm described in this section
derives from the fact that, at each iteration, the orthogonal-triangular decompositions are performed separately for each group and then once for the
xed eects. This allows ecient memory allocation for storing intermediate results and reduces the numerical complexity of the decompositions.
Also, the matrix inversions required to calculate the GaussNewton increments involve only upper-triangular matrices of small dimension, which
are easy to invert. (In fact, although (7.28) and (7.31) are written in terms
of matrix inverses, such as R1
00 , the actual calculation performed is the
= c0 , which is even
solution of the triangular system of equations R00
simpler.)
7.4 Extending the Basic NLME Model

The nonlinear mixed-eects model formulation used so far in this chapter allows considerable exibility in the specication of the random-eects
structure, but restricts the within-group errors to be independent and to
have constant variance. This basic NLME model provides an adequate
model for a broad range of applications, but there are many cases in which
the within-group errors are heteroscedastic (i.e., have unequal variances) or
are correlated or are both heteroscedastic and correlated.
This section extends the basic NLME model to allow heteroscedastic,
correlated within-group errors. We show how the estimation methods of
7.2 and the computational methods of 7.3 can be adapted to the extended
model formulation.
7.4.1 General Formulation of the Extended NLME Model

As described in 7.1.1, the basic single-level NLME
model
(7.3) assumes
that the within-group errors i are independent N 0, 2 I random vectors.
329
The extended single-level NLME model relaxes this assumption by allowing

heteroscedastic and correlated within-group errors, being expressed for i =
1, . . . , M as
y i = f i (i , v i ) + i ,
i = Ai + B i bi ,

i N 0, 2 i .
bi N (0, ),
(7.32)
The i are positive-denite matrices parametrized by a xed, generally

small, set of parameters . As in the basic NLME model, the within-group
errors i are assumed to be independent for dierent i and to be independent of the random eects bi . The 2 is factored out of the i for computational reasons (it can then be eliminated from the proled likelihood
function).
Similarly, the extended two-level NLME model generalizes the basic twolevel NLME model (7.7) described in 7.1.2 by letting

ij N 0, 2 ij , i = 1, . . . , M j = 1, . . . , Mi ,
where the ij are positive-denite matrices parametrized by a xed vector. This readily generalizes to a multilevel model with Q levels of random
eects. For simplicity, we concentrate for the remainder of this section on
the extended single-level NLME model (7.32), but the results we obtain
are easily generalizable to multilevel models with an arbitrary number of
levels of random eects.
As described in 5.1.3, the variancecovariance structure of the withingroup errors can be decomposed into two independent components: a variance structure and a correlation structure. Variance function models to
represent the variance structure component of the within-group errors are
described and illustrated in 5.2. Correlation models to represent the correlation structure of the within-group errors are presented and have their use
illustrated in 5.3. The use of the nlme function to t the extended NLME
model is described in 8.3.

1/2
Because i is positive-denite, it admits an invertible square-root i

1/2
(Thisted, 1988, 3), with inverse i
, such that
T /2
i = i
1/2
1/2
1
= i
i
and
T /2
Letting
T /2
y i = i
f i (i , v i ) =
i =
yi ,
T /2
i
f i (i , v i ) ,
T /2
i
i ,
(7.33)
330

T /2
T /2
1/2
= N 0, 2 I , we can
and noting that i N i
0, 2 i
i i
rewrite (7.32) as
y i = f i (i , v i ) + i ,
i = A i + B i bi ,

bi N (0, ), i N 0, 2 I .
That is, y i is described by a basic NLME model.
T /2
y i is
Because the dierential of the linear transformation y i = i
1/2
simply dy i = |i |
dy i , the log-likelihood function corresponding to
the extended NLME model (7.32) is expressed as
M
"

2
log p y i |, 2 , ,
, , , |y =
i=1
M
"
M

1"
2
=
log p y i |, , ,
log |i |
2 i=1
i=1
M
1"

2
= , , , |y
log |i | .
2 i=1

The log-likelihood function , 2 , , |y corresponds to a basic NLME
model with model function f i and, therefore, the approximations presented
in 7.2.1 can be applied to it. The inference results described in 7.2.2 also
remain valid.
Alternating Algorithm
The PNLS step of the alternating algorithm for the extended NLME model
consists of minimizing, over and bi , i = 1, . . . , M , the penalized nonlinear least-squares function
M

"
2
y i f i (, bi )2 + bi =
i=1
9
M +
2
"

T /2
[y i f i (, bi )] + bi 2 .
i
i=1
The derivative matrices and working vector used in the GaussNewton

algorithm for the PNLS step and also in the LME step are dened as

(w)
f i
T /2 &(w)
&
Xi ,
Xi
=
= i
T (w) (w)
,bi

T /2 (w)
(w) = f i
Z
= i
Zi ,
i
T (w) (w)
bi ,bi
(w)
i
w

= y i f i (
(w)
(w)
(w)
&
,
bi ) + X
i
(w)
T /2 (w)
(w) + Z
(w)
i ,
w
bi = i
i

(w)
(w)
331
&i , Z
i , and w
i dened as in (7.11).
with X
The GaussNewton algorithm for the PNLS step is identical to the algo(w)
&(w) , Z
(w) , and w

rithm described in 7.3, with X
replaced, respectively,
(w)
(w)
(w)
(w) , and w
&
i . The LME approximation to the log-likelihood
,Z
by X
i
i
function of the extended single-level NLME model is
M

1"

LME , 2 , , | y = LME , 2 , , | y
log |i | ,
2 i=1
which has the same form as the log-likelihood of the extended single-level
LME model described in 5.1. The log-restricted-likelihood for the extended
NLME model is similarly dened.
Laplacian and Adaptive Gaussian Approximations

For the extended single-level NLME model, the objective function which is
minimized to produce the conditional modes
bi used in the Laplacian and
adaptive Gaussian approximations is

2
T /2

g (, , , y i , bi ) = i
[y i f i (, bi )] + bi 2 .
The corresponding approximation to the second-derivative matrix of g
with respect bi evaluated at
bi is

2 g (, , , y i , bi )
G (, , , y i ) =
bi bTi
bi

f i (, bi )
1 f i (, bi )
i
+ T .

T
bi
bi
bi
bi
The modied Laplacian approximation to the log-likelihood of the extended single-level NLME model is then given by

N
LA , 2 , , , | y = log 2 2 + M log ||
2
2
'M
M
M
"
"
"
1

2
log |G (, , , y i )| +
g (, , , yi , bi ) 12
log |i |
i=1
i=1
i=1
332
and the adaptive Gaussian approximation is given by

AGQ , 2 , , , | y =
N
2

log 2 2 + M log ||
1
2
M
"
log |G (, , , y i )|
i=1
M
"
"
NGQ
log(
1
2
exp{g [, , , y i ,
bi + (G )
(, , , y i ) z j ]/2 2
i=1
2
+ z j /2}
q

k=1
1"
w jk )
log |i | .
2 i=1
M
The same comments and conclusions presented in 5.2 for the case when
the within-group variance function depends on the xed eects and/or the
random eects also apply to the extended NLME model. As in the LME
case, to keep the optimization problem feasible, an iteratively reweighted
scheme is used to approximate the variance function. The xed and random
eects used in the variance function are replaced by their current estimates
and held xed during the log-likelihood optimization. New estimates for the
xed and random eects are then produced and the procedure is repeated
until convergence. In the case of the alternating algorithm, the estimates
for the xed and random eects obtained in the PNLS step are used to
calculate the variance function weights in the LME step. If the variance
function does not depend on either the xed eects or the random eects,
then no approximation is necessary.
7.5 An Extended Nonlinear Regression Model

In many applications of nonlinear regression models to grouped data, one
wishes to represent the within-group variancecovariance structure through
the i matrices only, avoiding the use of random eects to account for
within-group dependence. This results in a simplied version of the extended single-level NLME model (7.32), which we call the extended nonlinear regression model. In this section, we present the general formulation
of the extended nonlinear regression model, describe methods for estimating its parameters, and present computational algorithms for implementing
such estimation methods.
The modeling function gnls in the nlme library ts the extended nonlinear regression model using maximum likelihood. The use of this function
is described and illustrated in 8.3.3.
333
7.5.1 General Model Formulation

The extended nonlinear regression model for the jth observation on the ith
group, yij , is

yij = f ij , v ij + ij ,
i = 1, . . . , M, j = 1, . . . , ni ,
ij = Aij .
(7.34)
The real-valued function f depends on a group-specic parameter vector

ij and a known covariate vector v ij . It is nonlinear in at least one component of ij and dierentiable with respect to the group-specic parameters.
M is the number of groups, ni is the number of observations on the ith
group, and ij is a normally distributed error term.
The extended nonlinear regression model (7.34) is a two-stage model in
which the second stage expresses the group-specic parameters ij as a
linear function of a xed set of parameters . The design matrices Aij are
known. We note that the coecients could be incorporated directly into
the model function f , thus eliminating the need of a second stage in the
model. However, there are advantages in having the second stage in (7.34),
some of which are (i) group-specic parameters generally have a more natural interpretation in the model, (ii) inclusion and elimination of covariates
in the model can be done at the second-stage model only, facilitating the
understanding of the model building process, and (iii) because the ij are
linear functions of the , derivatives with respect to ij are easily obtained
from derivatives with respect to .
Using the same denitions of vectors and matrices given in (7.4), we can
express the extended nonlinear regression in matrix form as
y i = f i (i , v i ) + i ,

i = Ai , i N 0, 2 i .
As in the extended NLME model of 7.4, the i matrices are determined
by a xed, generally small, set of parameters .
Estimation and inference under this model has been studied extensively
in the nonlinear regression literature (Carroll and Ruppert, 1988; Seber
and Wild, 1989), usually assuming that the i matrices are known, being
referred to as the generalized least-squares model (Seber and Wild, 1989,
2.1.4). We refer to it as the generalized nonlinear least-squares (GNLS)
model to dierentiate from the extended linear model described in 5.1.2.
Using the same transformations described in (7.33), the GNLS model
(7.34) can be re-expressed as a classic nonlinear regression model:
y i = f i (i , v i ) + i ,

i = Ai , i N 0, 2 I .
334

Dierent estimation methods have been proposed for the parameters in the
GNLS model (Davidian and Giltinan, 1995, 2.5). We concentrate here on
maximum likelihood estimation.
The log-likelihood function for the GNLS model is
82
7
'
M
2

"

1
y i f i ()
2
2

+ log |i |
,
N log 2 +
, , y) =
2
2
i=1
(7.35)
where N represents the total number of observations and, for simplicity,
we use f i () = f i (i , v i ).
For xed and , the maximum likelihood estimator of 2 is
2 (, ) =
M
"
y i f i () /N,
(7.36)
i=1
2 (, )
so that the proled log-likelihood, obtained by replacing 2 with
in (7.35), is
'
!
M
"
1
2
N [log (2/N ) + 1] + log

(, | y) =
y i f i ()
2
i=1
2
M
"
+
log |i | . (7.37)
i=1
A GaussSeidel algorithm (Thisted, 1988, 3.11.2) is used with the proled log-likelihood (7.37) to obtain the maximum likelihood estimates of
(w) of , a new estimate
(w) for
and . Given the current estimate

is produced by maximizing (,
(w)
). The roles are then reversed and a

(w) , ). The procedure
new estimate
is produced by maximizing (
iterates between the two optimizations until a convergence criterion is met.
It follows from (7.37) that, conditional on , the maximum likelihood
estimator of is obtained by solving an ordinary nonlinear least-squares
problem
(k+1)

()
= arg min
M
"
y i f i () ,
i=1
for which we can use a standard GaussNewton algorithm. If k is the it (k) =
(k) ((w) ) is the current
eration counter for this algorithm and
estimate of , then the derivative matrices

&(k) = T /2 X
&(k) = f i ()
&(k) ,
,
X
X
i
i
i
i
T (k)
335
(k+1) for
as the (ordinary) leastprovide the GaussNewton increment
squares solution of
2
M
"
(k)
(k)
&
w

i X i ,
i=1
(k)
(k) ). Orthogonal-triangular decomposition methi
where w
= y i f i (
ods similar to the ones described in 7.3 can be used to obtain a compact
and numerically ecient implementation of the GaussNewton algorithm
for estimating . The derivation is left to the reader as an exercise.
Inference on the parameters of the GNLS model generally relies on
classical asymptotic theory for maximum likelihood estimation (Cox and
Hinkley, 1974, 9.2), which states that, for large N , the MLEs are approximately normally distributed with mean equal to the true parameter values
and variancecovariance matrix given by the inverse of the information matrix. Because E[ 2 (, 2 , )/T ] = 0 and E[ 2 (, 2 , )/ 2 ] =
0, the expected information matrix for the GNLS likelihood is block-diagonal
and the MLE of is asymptotically uncorrelated with the MLEs of and
2 .
The approximate distributions for the MLEs in the GNLS model which
are used for constructing condence intervals and hypothesis tests are
7M
81
" T
.
N , 2
& 1 X
&i
,
X
i
i
i=1

N
, I 1 (, ) ,
log
log
2
2 / log T
/T
,
I (, ) =
2 / log
2 / 2 log

(7.38)
&i is the derivative matrix evaluated at the true parameter values.

where X
As in 7.2.2, log is used in place of 2 to give an unrestricted parameterization for which the normal approximation tends to be more accurate. In
practice, the parameters in the approximate variancecovariance matrices
in (7.38) are replaced by their respective MLEs.
To reduce the bias associated with the maximum likelihood estimation
of 2 , the following modied version of (7.36) is used,
2 =
M

"
2
T /2
/(N p),

f
y
i
i

i
i=1
with p denoting the length of . (N p)

2 is approximately distributed as
2 2
This
a N p random variable and is asymptotically independent of .
336
is used to produce approximate t and F tests for the coecients . These

tests tend to have better small sample properties than the tests obtained
from the normal approximation (7.38) alone.
7.6 Chapter Summary

This chapter presents the theoretical foundations of the nonlinear mixedeects model for single- and multilevel grouped data, including the general
model formulation and its underlying distributional assumptions. Ecient
computational methods for maximum likelihood estimation in the NLME
model are described and discussed. Dierent approximations to the NLME
model log-likelihood with varying degrees of accuracy and computational
complexity are derived.
The basic NLME model with independent, homoscedastic within-group
errors is extended to allow correlated, heteroscedastic within-group errors
and ecient computational methods are described for maximum likelihood
estimation of its parameters.
An extended class of nonlinear regression models, with correlated and
heteroscedastic errors, but with no random eects, is presented. An ecient
maximum likelihood estimation algorithm is described and approximate
inference results for the parameters in this extended nonlinear regression
are presented.
8
Fitting Nonlinear Mixed-Eects
Models
As shown in the examples in Chapter 6, nonlinear mixed-eects models

oer a exible tool for analyzing grouped data with models that depend
nonlinearly upon their parameters. As nonlinear models are usually based
on a mechanistic model of the relationship between the response and the
covariates, their parameters can have a theoretical interpretation and are
often of interest in their own right. In this chapter, we describe in detail
the facilities in the nlme library for tting nonlinear mixed-eects models.
The rst section presents a brief review of the standard nonlinear regression function in S, nls, to illustrate the use of nonlinear formulas and
the derivation of starting estimates for the model parameters. We describe
the selfStart class of nonlinear model functions that can calculate initial
values for their parameters from the data. The nlsList function, which
produces separate nls ts for each level of a grouping variable, is described
and illustrated through examples.
Section 8.2 describes the nlme function for tting nonlinear mixed-eects
models with single or multiple levels of nesting. Method functions for displaying, plotting, updating, making predictions, and obtaining condence
intervals from a tted nlme object are described and illustrated with examples.
The use of correlation structures and variance functions to extend the
basic nonlinear mixed-eects model is discussed in Section 8.3. The gnls
function for tting the extended nonlinear regression model without random eects, presented in 7.5.1, is described, together with its associated
methods.
338
8. Fitting Nonlinear Mixed-Eects Models
8.1 Fitting Nonlinear Models in S with nls and

nlsList
In this section we describe the use of the nonlinear least squares (nls)
function for tting a single nonlinear regression model and the nlsList
function for tting a set of nonlinear regression models to grouped data.
Especially with nlsList, it is very helpful to have the model function itself
dened as a self-starting model, as described in 8.1.2.
8.1.1 Using the nls Function

The S function nls uses a GaussNewton algorithm, described in 7.5.2,
to determine the nonlinear least squares estimates of the parameters in a
nonlinear regression model. A typical call to nls is of the form
nls( formula, data, start )
where formula is a two-sided nonlinear formula specifying the model, data

is a data frame with the variables used in formula, and start is a named
vector or list containing the starting estimates for the model parameters.
Several other arguments to nls are available, as described in Bates and
Chambers (1992) and Venables and Ripley (1999, Chapter 8).
We illustrate the use of nls to t nonlinear regression models with the
Orange data from a study of the growth of orange trees, reported in Draper
and Smith (1998, Exercise 24.N, p. 559). The data, shown in Figure 8.1 and
described in Appendix A.16, are the trunk circumferences (in millimeters)
of each of ve trees at each of seven occasions.
> ## outer = ~1 is used to display all five curves in one panel
> plot( Orange, outer = ~1 )
# Figure 8.1
Because all trees are measured on the same occasions these are balanced,
longitudinal data. It is clear from Figure 8.1 that a tree eect is present,
and we will take this into account when tting nlme or nlsList models.
To illustrate some of the details of tting nls models, we will temporarily
ignore the tree eect and t a single logistic model to all the data. Recall
from 6.1 that this model expresses the trunk circumference yij of tree i at
age xij for i = 1, . . . , 5 j = 1, . . . , 7 as
yij =
1
+ ij ,
1 + exp [ (tij 2 ) /3 ]
(8.1)
where the error terms ij are assumed to be distributed independently as

N (0, 2 ). As explained in 6.1, the model parameters are the asymptotic
trunk circumference 1 , the age at which the tree attains half of its asymptotic trunk circumference 2 , and the growth scale 3 . This function is
nonlinear in 2 and 3 .
Model (8.1) can be represented in S by the nonlinear formula
8.1 Fitting Nonlinear Models in S with nls and nlsList
339
Trunk circumference (mm)
200
150
100
50
200
600
1000
1400
Time since December 31, 1968 (days)
FIGURE 8.1. Circumference of ve orange trees from a grove in southern California over time. The measurement is probably the circumference at breast height
commonly used by foresters. Points corresponding to the same tree are connected
by lines.
circumference ~ Asym/(1 + exp(-(age - xmid)/scal)),
where Asym = 1 , xmid = 2 , and scal = 3 . Unlike in the linear case, the
parameters must be declared explicitly in a nonlinear model formula and
an intercept is not assumed by default.
An alternative approach is to write an S function representing the logistic
model as, say,
> logist <+
function(x, Asym, xmid, scal) Asym/(1 + exp(-(x - xmid)/scal))
and then use it in the nonlinear formula

circumference ~ logist(age, Asym, xmid, scal)
An advantage of this latter approach is that we can modify our logist

function to include a gradient attribute with its returned value. This would
then be used as the gradient matrix in the GaussNewton nonlinear leastsquares optimization, increasing the numerical stability and the rate of
convergence of the algorithm, compared to the default use of numerical
derivatives. The deriv function can be used to produce a function that
returns a gradient attribute with its value.
> logist <- deriv( ~Asym/(1+exp(-(x-xmid)/scal)),
340
+
c("Asym", "xmid", "scal"), function(x, Asym, xmid, scal){} )
> Asym <- 180; xmid <- 700; scal <- 300
> logist( Orange$age[1:7], Asym, xmid, scal )
[1] 22.617 58.931 84.606 132.061 153.802 162.681 170.962
attr(, "gradient"):
Asym
xmid
scal
[1,] 0.12565 -0.065916 0.127878
[2,] 0.32739 -0.132124 0.095129
[3,] 0.47004 -0.149461 0.017935
[4,] 0.73367 -0.117238 -0.118802
[5,] 0.85446 -0.074616 -0.132070
[6,] 0.90378 -0.052175 -0.116872
[7,] 0.94979 -0.028614 -0.084125
As mentioned in 6.1, one important dierence between linear and nonlinear regression is that the nonlinear models require starting estimates for
the parameters. Determining reasonable starting estimates for a nonlinear
regression problem is something of an art, but some general recommendations are available (Bates and Watts, 1988, 3.2). We return to this issue
in 8.1.2, where we describe the selfStart class of model functions.
Because the parameters in the logistic model (8.1) have a graphical interpretation, we can determine initial estimates from a plot of the data.
In Figure 8.1 it appears that the mean asymptotic trunk circumference is
around 170 mm and that the trees attain half of their asymptotic trunk
circumference at about 700 days of age. Therefore, we use the inital estimates of 1 = 170 for the asymptotic trunk circumference and 2 = 700
for the location of the inection point. To obtain an initial estimate for 3 ,
we note that the logistic curve reaches approximately 3/4 of its asymptotic
value when x = 2 + 3 . Inspection of Figure 8.1 suggests that the trees
attain 3/4 of their nal trunk circumference at about 1200 days, giving an
intial estimate of 3 = 500.
We combine all this information in the following call to nls
> fm1Oran.nls <- nls(circumference ~ logist(age, Asym, xmid, scal),
+
data = Orange, start = c(Asym = 170, xmid = 700, scal = 500) )
Our initial estimates are reasonable and the call converges. Following the
usual framework for modeling functions in S, the object fm1Oran.nls produced by the call to nls is of class nls, for which several methods are
available to display, plot, update, and extract components from a tted
object. For example, the summary method provides information about the
estimated parameters.
> summary( fm1Oran.nls )
Formula: circumference ~ logist(age, Asym, xmid, scal)
Parameters:
Asym 192.68
20.239 9.5203
341
-1
50
100
150
Fitted values
FIGURE 8.2. Scatter plots of standardized residuals versus tted values for
fm1Oran.nls, a nonlinear least squares t of the logistic growth model to the
entire orange tree data set.
xmid 728.71
scal 353.49
107.272
81.460
6.7931
4.3395

Correlation of Parameter Estimates:
Asym xmid
xmid 0.922
scal 0.869 0.770
The nal estimates are close to the initial values derived from Figure 8.1.
The standard errors for the parameter estimates are relatively large, suggesting that there is considerable variability in the data.
The plot method for nls objects (which is included with the nlme library),
has a syntax similar to the lme and gls plot methods described in 4.3.1
and 5.4. By default, the plot of the standardized residuals versus tted
values, shown in Figure 8.2, is produced.
> plot( fm1Oran.nls )
# Figure 8.2
The variability in the residuals increases with the tted values, but, in this
case, the wedge-shaped pattern is due to the correlation among observations
in the same tree and not to heteroscedastic errors. We can get a better
understanding of the problem by looking at the plot of the residuals by
tree presented in Figure 8.3.
> plot(fm1Oran.nls, Tree ~ resid(.), abline = 0)
# Figure 8.3
342
Tree
2
5
1
3
-40
-20
20
40
Residuals
FIGURE 8.3. Boxplots of residuals by tree for fm1Oran.nls, a nonlinear least

squares t of the logistic growth model to the entire orange tree data set.
The residuals are mostly negative for trees 1 and 3 and mostly positive for
trees 2 and 4, giving strong evidence that a tree eect should be included
in the model.
A simple approach to account for a tree eect is to allow dierent parameters for each tree, resulting in separate nls ts. This is the approach
used in the nlsList function, described in 8.1.3, which provides a valuable
tool for model building, but usually produces overparametrized models. As
illustrated in Chapter 6, nonlinear mixed-eects models strike a balance
between the simple nls model and the overparametrized nlsList model, by
allowing random eects to account for among-group variation in some of
the parameters, while preserving a moderate number of parameters in the
model.
8.1.2 Self-Starting Nonlinear Model Functions

Bates and Watts (1988, 3.2) describe several techniques for determining
starting estimates in a nonlinear regression. Some of the more eective
techniques are:
Take advantage of partially linear models, as described in Bates and
Chambers (1992, 10.2.5), so that initial estimates are needed only
for those parameters that enter the model nonlinearly.
Choose parameters that have meaningful graphical interpretations,
as we did for the logistic model.
Rene the estimates of some of the parameters by iterating on them
while holding all the other parameters xed at their current values.
The application of these techniques to a particular nonlinear regression
model applied to a given set of data can be straightforward but tedious.
343
Especially when the same model will be applied to several similar sets of
data, as is the case in many of the examples we consider here, we do not
want to have to manually repeat the same series of steps in determining
starting estimates. A more sensible approach is to encapsulate the steps
used to derive initial estimates for a given nonlinear model into a function
that can be used to generate intial estimates from any dataset. Self-starting
nonlinear regression models are S functions that contain an auxillary function to calculate the initial parameter estimates. They are represented in
S as selfStart objects.
The S objects of the selfStart class are composed of two functions; one
that evaluates the nonlinear regression model itself and an auxillary function, called the initial attribute, that determines starting estimates for the
models parameters from a set of data. When a selfStart object for a model
is available, there is no need to determine starting values for the parameters. The user can simply specify the formula for the model and the data to
which it should be applied. From a users point of view, tting self-starting
nonlinear regression models is nearly as easy as tting linear regression
models.
We illustrate the construction and use of self-starting models by building
function one for the logistic model. The basic steps in the calculation of
initial estimates for the logistic model (8.1) from the Orange dataset are:
1. Sort/average: sort the data according to the x variable and obtain
the average response y for each unique x.
2. Asymptote: use the maximum y as an initial value 1 for the asymptote.
3. Inection point: use the x corresponding to 0.51 as an initial value
2 for the inection point.
4. Scale: use the dierence between the x corresponding to 0.751 and
2 as an initial value 3 for the growth scale
Step 1, sort/average, was carried out implicitly in our graphical derivation of initial values for the orange trees example, but is now explicitly
incorporated in the algorithm.
Two auxillary functions, sortedXyData and NLSstClosestX, included in the
nlme library, are particularly useful for constructing self-starting models.
The sortedXyData function performs the sort/average step. It takes the
arguments x, y, and data and returns a data.frame with two columns: y,
the average y for each unique value of x, and x, the unique values of x,
sorted in ascending order. The arguments x and y can be numeric vectors
or they can be expressions or strings to be evaluated in data. The returned
object is of class sortedXyData. For example, the pointwise averages of the
growth curves of the orange trees are obtained with
344
> Orange.sortAvg <- sortedXyData( "age", "circumference", Orange )

> Orange.sortAvg
x
y
1 118 31.0
2 484 57.8
3 664 93.2
4 1004 134.2
5 1231 145.6
6 1372 173.4
7 1582 175.8
The NLSstClosestX function estimates the value of x corresponding to a

given y, using linear interpolation. It takes two arguments: xy, a sortedXyData object, and yval, the desired value of y, and returns a numeric value.
For example, the estimated age at which the average growth curve reaches
130 mm is
> NLSstClosestX(Orange.sortAvg, 130)
[1] 969.17
A function to serve as an initial attribute of a selfStart function must be

dened with the arguments mCall, LHS, and data, in that order, and must
return the initial values as a vector, or list, with elements named according
to the actual parameters in the model functions call. The rst argument,
mCall, is a matched call to the selfStart model. It contains the arguments
of the function call (as expressions), matched with the formal parameters,
which are the argument names used in the original denition of the model.
In the case of the logist function, dened before as
function(x, Asym, xmid, scal)
the formal parameters are x, Asym, xmid, and scal. If the function is called
as
logist(age, A, xmid, scal)
then mCall will have components x = age, Asym = A, xmid = xmid and scal
= scal. As described above, the names of these components are the formal
parameters in the model function denition. The values of these components are the names (or, more generally, the expressions) that are the actual
arguments in the call to the model function.
The LHS argument is the expression on the left-hand side of the model
formula in the call to nls. It determines the response vector. The data
argument gives a data.frame in which to nd the variables named in the
other two arguments. The function logistInit below implements a slightly
more general version of the algorithm described above for calculating initial
estimates in the logistic model, using the required syntax.
345
> logistInit
function(mCall, LHS, data)
{
xy <- sortedXyData(mCall[["x"]], LHS, data)
if(nrow(xy) < 3) {
stop("Too few distinct input values to fit a logistic")
}
Asym <- max(abs(xy[,"y"]))
if (Asym != max(xy[,"y"])) Asym <- -Asym # negative asymptote
xmid <- NLSstClosestX(xy, 0.5 * Asym)
scal <- NLSstClosestX(xy, 0.75 * Asym) - xmid
value <- c(Asym, xmid, scal)
names(value) <- mCall[c("Asym", "xmid", "scal")]
value
}
The algorithm in logistInit includes a check for an adequate number of

distinct observations to t a logistic model and allows negative as well as
positive asymptotes.
Before the starting values are returned, we ensure that the components
are named according to the actual parameters in the model function call.
As described above, the object mCall provides the correspondence between
the formal parameter names and the actual parameter names, so we assign
the names indirectly through mCall. This step is important. All functions
to be used as the initial attribute of a selfStart model should indirectly
assign the names to the result in this way.
The selfStart constructor is used to create a self-starting model. It can
be called with two functions, the model function itself and the initial
attribute, as in
> logist <- selfStart( logist, initial = logistInit )
> class( logist )
[1] "selfStart"
Alternatively, it can be called with a one-sided formula dening the nonlinear model, the function for the initial attribute, and a character vector
giving the parameter names.
> logist <- selfStart( ~ Asym/(1 + exp(-(x - xmid)/scal)),
+
initial = logistInit, parameters = c("Asym", "xmid", "scal"))
When selfStart is called like this, the model function is produced by applying deriv to the right-hand side of the model formula.
The getInitial function is used to extract the initial parameter estimates from a given dataset when using a selfStart model function. It takes
two arguments: a two-sided model formula, which must include a selfStart
function on its right-hand side, and a data.frame in which to evaluate the
variables in the model formula.
346
TABLE 8.1. Standard selfStart functions in the NLME 3.0 distribution.

SSasymp
SSasympOff
SSasympOrig
SSbiexp
SSfol
SSfpl
SSlogis
SSmicmen
asymptotic regression
asymptotic regression with an oset
asymptotic regression through the origin
biexponential
rst-order compartment
four-parameter logistic
logistic
MichaelisMenten
> getInitial(circumference ~ logist(age, Asym, xmid, scal), Orange)

Asym
xmid
scal
175.8 637.05 347.46
As expected, the initial values produced by logist are similar to those

obtained previously using the graphical interpretation of the parameters
and Figure 8.1.
When nls is called without initial values for the parameters and a selfStart model function is provided, nls calls getInitial to provide the initial
values. In this case, only the model formula and the data are required to
t the model, making the call nearly as simple as an lm call. For example,
the logistic model can be t to the orange tree data with
> nls( circumference ~ logist(age, Asym, xmid, scal), Orange )
Residual sum of squares : 17480
parameters:
Asym
xmid
scal
192.69 728.77 353.54
formula: circumference ~ logist(age, Asym, xmid, scal)
35 observations
The nlme library includes several self-starting model functions that can
be used to t nonlinear regression models in S without specifying starting
estimates for the parameters. They are listed in Table 8.1 and are described
in detail in Appendix B. The SSlogis function is a more sophisticated version of our simple logist self-starting model, but with the same argument
sequence. It uses several techniques, such as the algorithm for partially
linear models, to rene the starting estimates so the returned values are
actually the converged estimates.
> getInitial(circumference ~ SSlogis(age,Asym,xmid,scal), Orange)
Asym
xmid scal
192.68 728.72 353.5
> nls( circumference ~ SSlogis(age, Asym, xmid, scal), Orange )
Residual sum of squares : 17480
347
parameters:
Asym
xmid scal
192.68 728.72 353.5
formula: circumference ~ SSlogis(age, Asym, xmid, scal)
35 observations
We can see that selfStart model objects relieve the user of much of the
eort required for a nonlinear regression analysis. If a versatile, eective
strategy for determining starting estimates is represented carefully in the
object, it can make the use of nonlinear models nearly as simple as the
use of linear models. If you frequently use a nonlinear model not included
in Table 8.1, you should consider writing your own self-starting model to
represent it. The selfStart functions in Table 8.1 and the logist function
described in this section can be used as templates.
8.1.3 Separate Nonlinear Fits by Group: The nlsList

Function
In the indomethicin and soybean examples of Chapter 6 we saw how the
nlsList function can be used to produce separate ts of a nonlinear model
for each group in a groupedData object. These separate ts by group are
a powerful tool for model building with nonlinear mixed-eects models
because the individual estimates can suggest the type of random-eects
structure to use. Also, these estimates provide starting values for the parameters in the mixed-eects model. In this section we provide more detail
on the use of nlsList function itself and the methods for the nlsList objects
that it creates.
The nls ts performed within nlsList will require starting estimates
for the parameters. Although it is sometimes possible to use a single set
of starting estimates for all the groups, we recommend using a selfStart
function, as described in 8.1.2, to automatically generate individual initial
estimates for each group.
A typical call to nlsList is
nlsList( model, data )
where model is a two-sided formula whose right-hand side consists of two

parts separated by the | operator. The rst part denes the nonlinear
model, generally involving a selfStart function to be tted to each subset
of data, and the second part species the grouping factor. Any nonlinear
model formula allowed in nls can also be used with nlsList. The data argument gives a data frame in which the variables in model can be evaluated.
The grouping factor can be omitted from the model formula when data is
a groupedData object.
To illustrate the use of nlsList, let us continue with the analysis of the
orange trees data, tting a separate logistic curve to each tree and using
the selfStart function SSlogis to produce initial estimates.
348
> fm1Oran.lis <+

nlsList(circumference ~ SSlogis(age, Asym, xmid, scal) | Tree,
+
data = Orange)
Because Orange is a groupedData object, the grouping factor Tree could have
been omitted from the model formula. When a selfStart function depends
on only one covariate, as does SSlogis, and data is a groupedData object, the
entire form of the model formula can be inferred from the display formula
stored with the groupedData object. In this case, only the selfStart function
and the groupedData object need to be passed to the nlsList function. For
example, we could use
> fm1Oran.lis <- nlsList( SSlogis, Orange )
to obtain the same nlsList t as before.

The nlsList function can also be used with regular, non-self-starting
nonlinear functions, but in this case the same set of starting values, specied
in the start argument, will be used for every group. The use of common
starting estimates may not be a sensible choice in many applications, so
we strongly encourage the use of selfStart functions with nlsList. In the
orange trees example, the individual growth patterns are similar enough
that a common set of starting estimates can be used. For example, using
the same initial estimates as used to t fm1Oran.nls in 8.1.1, we have
> fm1Oran.lis.noSS <+
nlsList( circumference ~ Asym/(1+exp(-(age-xmid)/scal)),
+
data = Orange,
+
start = c(Asym = 170, xmid = 700, scal = 500) )
Because the model ts fm1Oran.lis and fm1Oran.lis.noSS are derived from

dierent starting values, the parameter estimates will dier slightly.
Objects returned by nlsList are of class nlsList which inherits from class
lmList. Therefore, all summary and display methods, as well as methods
for extracting components from the tted object, for class lmList can also
be applied to an nlsList object. Table 8.2 lists the most commonly used
methods for nlsList objects. We illustrate the use of some of these methods
below.
The print method gives minimal information about the individual nls
ts.
> fm1Oran.lis
Call:
Model: circumference ~ SSlogis(age, Asym, xmid, scal) | Tree
Data: Orange
Coefficients:
Asym
xmid
scal
3 158.83 734.85 400.95
TABLE 8.2. Main nlsList methods.

augPred
coef
fitted
fixef
intervals
nlme
logLik
pairs
plot
predict
print
qqnorm
ranef
resid
summary
update
1
5
2
4
154.15
207.27
218.99
225.30

coecients from individual nls ts
tted values from individual nls ts
average of individual nls coecients
condence intervals on coecients
nonlinear mixed-eects model from nlsList t
sum of individual nls log-likelihoods
predictions for individual nls ts
brief information about the lm ts
deviations of coecients from average
residuals from individual nls ts
more detailed information about nls ts
update the individual nls ts
627.12
861.35
700.32
710.69
362.50
379.99
332.47
303.13

More detailed information on the coecients is obtained with summary

> summary( fm1Oran.lis )
. . .
Coefficients:
Asym
3 158.83
19.235 8.2574
1 154.15
13.588 11.3446
5 207.27
22.028 9.4092
2 218.99
13.357 16.3959
4 225.30
11.838 19.0318
xmid
3 734.85
130.807 5.6178
1 627.12
92.854 6.7538
5 861.35
108.017 7.9742
2 700.32
61.342 11.4167
4 710.69
51.166 13.8899
349
350

Asym
Tree
|
|
|
xmid
120
160
200
240
500 600 700 800 900 1000
|
|
|
|
|
|
|
|
|
|
|
|
scal
|
|
|
|
|
200
300
400
|
500
600
FIGURE 8.4. Ninety-ve percent condence intervals on the logistic model parameters for each tree in the orange trees data.
3
1
5
2
4
.
scal
400.95
94.776 4.2306
362.50
81.185 4.4652
379.99
66.761 5.6917
332.47
49.381 6.7327
303.13
41.608 7.2853
. .
Although the estimates for all the parameters vary with tree, there appears to be relatively more variability in the Asym estimates. We can investigate this better with the intervals method.
> plot( intervals( fm1Oran.lis ), layout = c(3,1) )
# Figure 8.4
As mentioned in 6.2, condence intervals for the same parameter in dierent groups within an nlsList t do not necessarily have the same length,
even with balanced data. This is evident in Figure 8.4.
The only parameter for which all the condence intervals do not overlap
in Figure 8.4 is Asym, suggesting that it is the only parameter for which
random eects are needed to account for variation among trees.
The same plot method used for lmList objects is used to obtain diagnostic
plots for an nlsList object. The boxplots of the residuals by tree, obtained
with
> plot( fm1Oran.lis, Tree ~ resid(.), abline = 0 )
# Figure 8.5
and displayed in Figure 8.5, no longer indicate the tree eect observed
in Figure 8.3. The basic drawback of the nlsList model is that uses 15
parameters to account for the individual tree eects. A more parsimonious
representation is provided by the nonlinear mixed-eects model discussed
in 8.2.
As a second example to illustrate the use of the nlsList function, we
consider the theophylline data, which we used in 3.4. Recall that these
data, displayed in Figure 8.6, are the serum concentrations of theophylline
351
Tree
2
5
1
3
-10
-5
10
Residuals (mm)
FIGURE 8.5. Boxplots of residuals by tree for fm1Oran.lis.
measured on twelve subjects at eleven times up to 25 hours after receiving

an oral dose of the drug.
> Theoph[1:4,]
Grouped Data: conc ~ Time | Subject
Subject
Wt Dose Time conc
1
1 79.6 4.02 0.00 0.74
2
1 79.6 4.02 0.25 2.84
3
1 79.6 4.02 0.57 6.57
4
1 79.6 4.02 1.12 10.50
The column Wt in Theoph gives the Subjects weight (in kilograms).

As when modeling the indomethicin data in 6.2 and the phenobarbital
data in 6.4, we use a compartment model for these data. A rst-order
open-compartment model expresses the theophylline concentration ct at
time t after an initial dose D as
ct =
Dke ka
[exp (ke t) exp (ka t)] .
Cl (ka ke )
(8.2)
The parameters in the model are the elimination rate constant ke , the
absorption rate constant ka , and the clearance Cl. For the model to be
meaningful, all three parameters must be positive. To ensure ourselves of
positive estimates while keeping the optimization problem unconstrained,
we reparameterize model (8.2) in terms of the logarithm of the clearance
and the rate constants.
ct =
D exp (lKe + lKa lCl )

{exp [ exp (lKe) t] exp [ exp (lKa) t]} ,
exp (lKa) exp (lKe)
(8.3)
where lKe = log(ke ), lKa = log(ka ), and lCl = log(Cl).
352

0
10
10
15
20
25
5
10
Theophylline concentration in serum (mg/l)
8
6
4
2
4
12
11
10
8
6
4
2
0
10
8
6
4
2
6
10
8
6
4
2
0
0
10
15
20
25
10
15
20
25
FIGURE 8.6. Serum concentrations of theophylline versus time since oral administration of the drug in twelve subjects.
The selfStart function SSfol, described in Appendix C.5, provides a selfstarting implementation of (8.3). Because two covariates, dose and time,
are present in (8.3), and hence also in the argument sequence of SSfol, we
must specify the full model formula when calling nlsList.
> fm1Theo.lis <- nlsList( conc ~ SSfol(Dose, Time, lKe, lKa, lCl),
+
data = Theoph )
> fm1Theo.lis
Call:
Model: conc ~ SSfol(Dose, Time, lKe, lKa, lCl) | Subject
Data: Theoph
Coefficients:
lKe
lKa
6 -2.3074 0.15171
7 -2.2803 -0.38617
8 -2.3863 0.31862
11 -2.3215 1.34779
3 -2.5081 0.89755
2 -2.2862 0.66417
4 -2.4365 0.15834
9 -2.4461 2.18201
12 -2.2483 -0.18292
10 -2.6042 -0.36309
1 -2.9196 0.57509
lCl
-2.9733
-2.9643
-3.0691
-2.8604
-3.2300
-3.1063
-3.2861
-3.4208
-3.1701
-3.4283
-3.9158

lKe
1
10
12
Subject
-3.0
-2.5
| |
|
-2.0
-1
|
0
11
| |
|
|
|
2
-4.0
-3.5
| | |
|
|
|
| | |
| | |
|
| | |
lCl
| | |
lKa
353
-3.0
FIGURE 8.7. Ninety-ve percent condence intervals on the parameters in the

rst-order open-compartment model (8.3) for each subject in the theophylline
data.
5 -2.4254
0.38616 -3.1326

The individual estimates suggest that the absorption rate constant is

more variable among subjects than either the elimination rate constant or
the clearance. As usual, we recommend using the plot of the condence
intervals on the individual parameters to analyze their between-group variation.
> plot( intervals( fm1Theo.lis ), layout = c(3,1) )
# Figure 8.7
The individual condence intervals in Figure 8.7 indicate that there is substantial subject-to-subject variation in the absorption rate constant and
moderate variation in the clearance. The elimination rate constant does
not seem to vary signicantly with subject.
The main purpose of the preliminary analysis provided by nlsList is
to suggest a structure for the random eects to be used in a nonlinear
mixed-eects model. We must decide which random eects to include in
the model (intervals and its associated plot method are often useful for
that) and what covariance structure these random eects should have. The
pairs method, which is the same for lmList and nlsList objects, provides
one view of the random eects covariance structure.
> pairs( fm1Theo.lis, id = 0.1 )
# Figure 8.8
354

-3.4
-3.2
-3.0
-3.0
-3.2
lCl
-3.4
-3.4
-3.6
-3.8
-3.8
1.0
2.0
1.5
-3.6
-3.4
2.0
1.5
1.0
lKa
1.0
0.5
0.0
0.0
-2.6
-2.3
-2.5
-2.4
0.5
1.0
-2.3
-2.4
-2.5
lKe
-2.6
-2.6
-2.7
-2.8
-2.9
-2.8
-2.7
-2.6
-2.9
FIGURE 8.8. Pairs plot for the random eects estimates corresponding to
fm1Theo.lis.
The scatter plots in Figure 8.8 suggest that Subject 1 has an unusually
low elimination rate constant and clearance and that Subject 9 has an
unusually high absorption rate constant. Overall, there do not appear to
be signicant correlations between the individual parameter estimates.
8.2 Fitting Nonlinear Mixed-Eects Models with

nlme
The general formulation of a nonlinear mixed-eects model, and the estimation methods used to t it, are described in 7.1. This section concentrates on the capabilities available in the nlme library for tting nonlinear
mixed-eects models with independent, homoscedastic within-group errors.
Fitting nlme models to single-level grouped data is described in 8.2.1. The
use of covariates to explain between-group variability and to reduce the
number of random eects in an nlme model is presented in 8.2.2. In 8.2.3,
we describe how to t multilevel nonlinear mixed-eects models with nlme.
8.2.1 Fitting Single-Level nlme Models

Several optional arguments can be used with the nlme function, but a typical
call looks like
nlme( model, data, fixed, random, groups, start )
8.2 Fitting Nonlinear Mixed-Eects Models with nlme
355
The model argument is required and consists of either a two-sided formula

specifying the nonlinear model to be tted, or an nlsList object. Any S
nonlinear formula can be used, giving the function considerable exibility.
For example, in the orange trees example one could either write down the
logistic model explicitly
circumference ~ Asym/(1 + exp(-(age - xmid)/scal))
or use the selfStart function SSlogis

circumference ~ SSlogis(age, Asym, xmid, scal)
As with nls ts, there is an advantage of encapsulating the model expression in an S function when tting an nlme model, in that it allows
analytic derivatives of the model function to be passed to nlme and used
in the optimization algorithm. The S function deriv can be used to create
model functions that return the value of the model and its derivatives as
a gradient attribute. If the value returned by the model function does not
have a gradient attribute, numerical derivatives are used in the optimization.
The arguments fixed and random are formulas, or lists of formulas, dening the structures of the xed and random eects in the model. In these
formulas a 1 on the right-hand side of a formula indicates that a single
parameter is associated with the eect, but any linear formula in S can
be used. This gives considerable exibility to the model, as time-dependent
parameters can be incorporated easily (e.g., when a formula in the fixed list
involves a covariate that changes with time). Each parameter in the model
will usually have an associated xed eect, but it may, or may not, have an
associated random eect. Because the nlme model assumes that all random
eects have expected value zero, the inclusion of a random eect without a
corresponding xed eect would be unusual. Any covariates dened in the
fixed and random formulas can, alternatively, be directly incorporated in
the model formula. However, declaring the covariates in fixed and random
allows for more ecient calculation of derivatives and is useful for update
methods. Fixed is required when model is declared as a formula. By default,
when random is omitted, all xed eects in the model are assumed to have
an associated random eect.
In the theophylline example, to have random eects only for the log of
the absorption rate constant, lKa, and the log-clearance, lCl, we use
fixed = list(lKe ~ 1, lKa ~ 1, lCl ~ 1),
random = list(lKa ~ 1, lCl ~ 1)
Model parameters with common right-hand side expressions in the fixed

or random formulas can be collapsed into a single formula, in which the
left-hand side lists the parameters separated by the + operator. For example, fixed and random in the theophylline example can be expressed more
compactly as
356

fixed = lKe + lKa + lCl ~ 1
random = lKa + lCl ~ 1
If we wanted to allow the log-clearance xed eect to depend linearly on

the subjects weight while retaining a single component for the other xed
eects, we would use
fixed = list(lKe + lKa ~ 1, lCl ~ Wt)
Note that there would be two xed eects associated with lCl in this case:
an intercept and a slope with respect to Wt.
Data names a data frame in which any variables used in model, fixed,
random, and groups are to be evaluated. By default, data is set to the environment from which nlme was called.
The groups argument is a one-sided formula, or an S expression, which,
when evaluated in data, returns a factor with the group label of each observation. This argument does not need to be specied when object is an
nlsList object, or when data is a groupedData object, or when random is a
named list (in which case the name is used as the grouping factor).
The start argument provides a list, or a vector, of starting values for the
iterative algorithm. When given as a vector, it is used as starting estimates
for the xed eects only. It is only required when model is given as a formula
and the model function is not a selfStart object. In this case, starting values
for the xed eects must be specied. Starting estimates for the remaining
parameters are generated automatically. By default, the random eects are
initialized to zero.
Objects returned by nlme are of class nlme, which inherits from class lme.
As a result, all the methods available for lme objects can also be applied to
an nlme object. In fact, most methods are common to both classes. Table 8.3
lists the most important methods for class nlme. We illustrate the use of
these methods through the examples in the next sections.
Growth of Orange Trees
The nonlinear mixed-eects model corresponding to the logistic model 8.1,
with random eects for all parameters, is
1i
yij =
+ ij ,
1 + exp [ (tij 2i ) /3i ]

1
b1i
1i
i = 2i = 2 + b2i = + bi ,
3i
3
b3i

bi N (0, ) ,
ij N 0, 2 .
(8.4)
The model parameters i have the same interpretation as in (8.1), but

are now allowed to vary with tree. The xed eects, , represent the average value of the individual parameters, i , in the population of orange
357
TABLE 8.3. Main nlme methods.

ACF
anova
augPred
coef
fitted
fixef
intervals
logLik
pairs
plot
predict
print
qqnorm
ranef
resid
summary
update
Variogram
empirical autocorrelation function of within-group residuals

likelihood ratio or Wald-type tests
estimated coecients for dierent levels of grouping
tted values for dierent levels of grouping
xed-eects estimates
predictions for dierent levels of grouping
random-eects estimates
residuals for dierent levels of grouping
update the nlme t
semivariogram of within-group residuals
trees and the random eects, bi , represent the deviations of the i from
their population average. The random eects are assumed to be independent for dierent trees and the within-group errors ij are assumed to be
independent for dierent i, j and to be independent of the random eects.
The nonlinear mixed-eects model (8.4) uses ten parameters to represent
the xed eects (three parameters), the random-eects variancecovariance
structure (six parameters), and the within-group variance (one parameter).
These numbers remain unchanged if we increase the number of trees being
analyzed. In comparison, the number of parameters in the corresponding
nlsList model, described in 8.1.3, is equal to three times the number of
trees (15, in the orange trees example).
A nonlinear mixed-eects t of model (8.4) can be obtained with
> ## no need to specify groups, as Orange is a groupedData object
> ## random is omitted - by default it is equal to fixed
> fm1Oran.nlme <+ nlme( circumference ~ SSlogis(age, Asym, xmid, scal),
+
data = Orange,
+
fixed = Asym + xmid + scal ~ 1,
+
start = fixef(fm1Oran.lis) )
but a much simpler, equivalent form is

> fm1Oran.nlme <- nlme( fm1Oran.lis )
358
The fm1Oran.lis object stores information about the model function, the
parameters in the model, the groups formula, and the data used to t the
model. These are retrieved by nlme, allowing a more compact call. Another
important advantage of using an nlsList object as the rst argument to
nlme is that it automatically provides initial estimates for the xed eects,
the random eects, and the random-eects covariance matrix.
We can now use the nlme methods to display the results and to assess
the quality of the t. As with lme objects, the print method gives some
brief information about the tted object.
> fm1Oran.nlme
Model: circumference ~ SSlogis(age, Asym, xmid, scal)
Data: Orange
Fixed: list(Asym ~ 1, xmid ~ 1, scal ~ 1)
Asym
xmid
scal
192.12 727.74 356.73
Random effects:
Level: Tree
StdDev
Corr
Asym 27.0302 Asym
xmid
xmid 24.3761 -0.331
scal 36.7363 -0.992 0.447
Residual 7.3208
Number of Groups: 5
Note that the default estimation method in nlme is maximum likelihood

(ML), while the default in lme is restricted maximum likelihood (REML).
The reason for this, as described in 7.2.2, is because nested nlme models
which dier in either their xed eects, or their random eects, cannot
be compared using their REML likelihoods, thus invalidating most REML
likelihood ratio tests of practical interest. In the linear case, REML likelihoods of nested mixed-eects models with common xed-eects structure
are comparable.
More detailed information about the t is provided by the summary method.
> summary(fm1Oran.nlme)
Model: circumference ~ SSlogis(age, Asym, xmid, scal)
Data: Orange
AIC
BIC logLik
279.98 295.53 -129.99
359
Random effects:
Level: Tree
StdDev
Corr
Asym 27.0302 Asym
xmid
xmid 24.3761 -0.331
scal 36.7363 -0.992 0.447
Residual 7.3208
Fixed effects: list(Asym ~ 1, xmid ~ 1, scal ~ 1)
Asym 192.12
14.045 28 13.679 <.0001
xmid 727.74
34.618 28 21.022 <.0001
scal 356.73
30.537 28 11.682 <.0001
Correlation:
Asym
xmid
xmid 0.275
scal -0.194 0.666
. . .
It is interesting, at this point, to compare the nlme t with the simple

nls t that ignores the grouped structure of the data, obtained in 8.1.1.
> summary(fm1Oran.nls)
. . .
Parameters:
Asym 192.68
20.239 9.5203
xmid 728.71
107.272 6.7931
scal 353.49
81.460 4.3395
. . .
The xed-eects estimates are similar, but the standard errors are much
smaller in the nlme t. The estimated within-group standard error is also
considerably smaller in the nlme t. This is because the between-group
variability is not incorporated in the nls model, being absorbed in the
standard error. This pattern is generally observed when comparing mixedeects versus xed-eects ts.
In the fm1Oran.nlme t the estimated correlation of 0.992 between Asym
and xmid suggests that the estimated variancecovariance matrix is illconditioned and that the random-eects structure may be over-parameterized. The scatter-plot matrix of the estimated random eects, produced
by the pairs method, provides a useful diagnostic plot for assessing overparameterization problems.
> pairs( fm1Oran.nlme )
# Figure 8.9
360

400
360
380
400
380
360
scal
360
340
320
320
730
340
360
740
740
730
xmid
730
720
720
190
220
200
210
730
220
210
200
Asym
190
190
180
170
160
170
180
190
160
FIGURE 8.9. Pairs plot for the random-eects estimates corresponding to

fm1Oran.nlme.
The nearly perfect alignment between the Asym random eects and the scal
random eects further indicates that the model is over-parameterized.
The individual condence intervals for the parameters in the nlsList
model described by fm1Oran.lis, displayed in Figure 8.4 and discussed
in 8.1.3, suggested that Asym was the only parameter requiring a random eect to account for its variation among trees. We can t the corresponding model using the update method and compare it to the full model
fm1Oran.nlme using the anova method.
> fm2Oran.nlme <- update( fm1Oran.nlme, random = Asym ~ 1 )
> anova( fm1Oran.nlme, fm2Oran.nlme )
Model df
AIC
BIC logLik
fm1Oran.nlme
1 10 279.98 295.53 -129.99
fm2Oran.nlme
2 5 273.17 280.95 -131.58 1 vs 2 3.1896 0.6708
The large p-value for the likelihood ratio test conrms that the xmid and
scal random eects are not needed in the nlme model and that the simpler
model fm2Oran.nlme is to be preferred.
As in the linear case, we must check if the assumptions underlying the
nonlinear mixed-eects model appear valid for the model tted to the data.
The two basic distributional assumptions are the same as for the lme model:
Assumption 1 : the within-group errors are independent and identically normally distributed, with mean zero and variance 2 , and they
are independent of the random eects.
361
-1
-2
50
100
150
200
Fitted values (mm)
fm1Oran.nlme.
Assumption 2 : the random eects are normally distributed, with

mean zero and covariance matrix (not depending on the group)
and are independent for dierent groups;
The plot and qqnorm methods provide the most useful tools for assessing
these assumptions. For example, a plot of the standardized residuals versus
the tted values in Figure 8.10
> plot( fm1Oran.nlme )
# Figure 8.10
shows that the residuals are distributed symmetrically around zero, with
an approximately constant variance. It does not indicate any violations of
the assumptions for the within-group error:
The adequacy of the tted model is better visualized by displaying the
tted and observed values in the same plot. The augPred method, which
produces the augmented predictions, and its associated plot method are
used for that. For comparison, both the population predictions (obtained
by setting the random eects to zero) and the within-group predictions
(using the estimated random eects) are displayed in Figure 8.11, produced
with
> ## level = 0:1 requests fixed (0) and within-group (1) predictions
> plot( augPred(fm2Oran.nlme, level = 0:1),
# Figure 8.11
+
layout = c(5,1) )
The tree-specic predictions follow the observed values closely, indicating

that the logistic mixed-eects model (8.4) explains the trunk circumference
growth in the orange trees data well.
The normal plot of the standardized residuals, shown in Figure 8.12,
does not indicate any violations of the normality assumption for the withingroup errors.
362
Trunk circumference (mm)
200
fixed
600
Tree
1000
1400
200
150
100
50
200
600
1000
1400
200
600
1000
1400
200
600
1000
1400
Time since December 31, 1968 (days)
FIGURE 8.11. Population predictions (fixed), within-group predictions (Tree),

and observed trunk circumferences (circles) versus age of tree, for the
fm2Oran.nlme model .
> qqnorm( fm2Oran.nlme, abline = c(0,1) )
#Figure 8.12
Because there are only ve trees in the data, with just seven observations each, we cannot reliably test assumptions about the random-eects
distribution and the independence of the within-group errors.
-1
-2
-2
-1
FIGURE 8.12. Normal probability plot of the standardized residuals from a nonlinear mixed-eects model t fm1Oran.nlme to the orange data.
363
Theophylline Kinetics
The nonlinear mixed-eects version of the rst-order open-compartment
model (8.3), with all parameters as mixed-eects, is
cij =
Di exp (lKei + lKai lCli )

exp (lKai ) exp (lKei )
{exp [ exp (lKei ) tij ] exp [ exp (lKai ) tij ]} + ij ,

1
b1i
lKe i
i = lKa i = 2 + b2i = + bi ,
lCl i
3
b3i

bi N (0, ) , ij N 0, 2 ,
(8.5)
where cij is the theophylline concentration for patient i measured at time

tij , with an initial dose Di . The pharmacokinetic parameters i are interpreted as in (8.3), but are now allowed to vary with subject. The xed
eects, , represent the population average of the i and the random effects, bi , their deviations from the population average.
Fitting the nonlinear mixed-eects model (8.5) to the theophylline data
is done with the simple call
> fm1Theo.nlme <- nlme( fm1Theo.lis )
> fm1Theo.nlme
Model: conc ~ SSfol(Dose, Time, lKe, lKa, lCl)
Data: Theoph
Fixed: list(lKe ~ 1, lKa ~ 1, lCl ~ 1)
lKe
lKa
lCl
-2.4327 0.45146 -3.2145
Random effects:
Formula: list(lKe ~ 1, lKa ~ 1, lCl ~ 1)
Level: Subject
StdDev
Corr
lKe 0.13104 lKe
lKa
lKa 0.63783 0.012
lCl 0.25118 0.995 -0.089
Residual 0.68183
The estimated correlation between the lKe and lCl random eects is
near one, indicating that the estimated random-eects variancecovariance
matrix is ill-conditioned. We can investigate the precision of the correlation
estimates with the intervals method.
364
> intervals( fm1Theo.nlme, which = "var-cov" )

Random Effects:
Level: Subject
lower
est.
sd(lKe) 0.058102 0.131142
sd(lKa) 0.397740 0.637838
sd(lCl) 0.156811 0.251216
cor(lKe,lKa) -0.399511 0.011723
cor(lKe,lCl) -0.995248 0.994868
cor(lKa,lCl) -0.520140 -0.089480
upper
0.29600
1.02287
0.40246
0.41903
1.00000
0.37746

lower
est.
upper
0.59598 0.68183 0.78005
All three condence intervals for the correlations include zero. The interval
on the correlation between lKe and lCl shows this quantity is not estimated
with any precision whatsoever. It must lie in the interval [1, 1] and the
condence interval is essentially that complete range.
As a rst attempt at simplifying model (8.5), we investigate the assumption that the random eects are independent, that is, the matrix
is diagonal. Structured random-eects variancecovariance matrices are
specied in nlme in the same way as in lme: by using a pdMat constructor to
specify the desired class of positive-denite matrix. The pdMat classes and
methods are described in 4.2.2 and the standard pdMat classes available in
the nlme library are listed in Table 4.3. By default, when no pdMat class is
specied, a general positive-denite matrix (pdSymm class) is used to represent the random-eects variancecovariance structure. Alternative pdMat
classes are specied by calling the corresponding constructor with the random eects formula, or list of formulas, as its rst argument. For example,
we specify a model with independent random eects for the theophylline
data with either
> fm2Theo.nlme <- update( fm1Theo.nlme,
+
random = pdDiag(list(lKe ~ 1, lKa ~ 1, lCl ~ 1)) )
or
+
random = pdDiag(lKe + lKa + lCl ~ 1) )
> fm2Theo.nlme
. . .
lKe
lKa
lCl
-2.4547 0.46554 -3.2272
365
Random effects:
Formula: list(lKe ~ 1, lKa ~ 1, lCl ~ 1)
Level: Subject
Structure: Diagonal
lKe
lKa
lCl Residual
StdDev: 1.9858e-05 0.64382 0.16692 0.70923
. . .
The very small estimated standard deviation for lKe suggests that the corresponding random eect could be omitted from the model.
> fm3Theo.nlme <+
update( fm2Theo.nlme, random = pdDiag(lKa + lCl ~ 1) )
We use the anova method to test the equivalence of the dierent nlme
models used so far for the theophylline data. By including fm3Theo.nlme as
the second model, we obtain the p-values comparing this model with each
of the other two.
> anova( fm1Theo.nlme, fm3Theo.nlme, fm2Theo.nlme )
Model df
AIC
BIC logLik
fm1Theo.nlme
1 10 366.64 395.47 -173.32
fm3Theo.nlme
2 6 366.04 383.34 -177.02 1 vs 2 7.4046 0.1160
fm2Theo.nlme
3 7 368.05 388.23 -177.02 2 vs 3 0.0024 0.9611
The simpler fm3Theo.nlme model, with two independent random eects for
lKa and lCl, has the smallest AIC and BIC. Also, the large p-values for the
likelihood ratio tests comparing it to the other two models indicate that it
should be preferred.
The plot of the standardized residuals versus the tted values in Figure 8.13 gives some indication that the within-group variability increases
with the drug concentration.
> plot( fm3Theo.nlme )
# Figure 8.13
The use of variance functions in nlme to model within-group heteroscedasticity is described in 8.3.1. We postpone further investigation of the withingroup error assumptions in the theophylline until that section.
The qqnorm method is used to investigate the normality of the random
eects.
> qqnorm( fm3Theo.nlme, ~ ranef(.) )
# Figure 8.14
Figure 8.14 does not indicate any violations of the assumption of normality
for the random eects.
8.2.2 Using Covariates with nlme

The random eects in a mixed-eects model represent deviations of the
individual parameters from the xed eects. In some applications, these
366
-2
10
Fitted values (mg/l)
fm3Theo.nlme.
lKa
lCl
-1
-0.5
0.0
0.5
1.0
-0.3
-0.1
0.0
0.1
0.2
Random effects
FIGURE 8.14. Normal plot of the estimated random eects corresponding to

fm3Theo.nlme t.
367
deviations arise from unexplained intergroup variation but, frequently, they

can be at least partially explained by dierences in covariate values among
groups. For example, dierences in the absorption rate constant among
the subjects in the theophylline example of 8.2.1 may be attributed to
dierences in age, weight, blood pressure, etc.
Including covariates in the model to explain intergroup variation generally reduces the number of random eects in the model and leads to a
better understanding of the mechanism producing the response. We have
seen this in the soybean and the phenobarbital examples in Chapter 6.
Some of the questions that need to be addressed in the covariate modeling
process are the following:
1. Among the candidate covariates, which are potentially useful in explaining the random-eects variation?
2. Which random eects have their variation best explained by covariates?
3. How should the potentially useful covariates be tested for inclusion
in the model?
4. Should random eects be included in, or eliminated from, the modied model?
This section describes a model-building strategy for addressing these questions in the context of nonlinear mixed-eects models and using the capabilities of the nlme library.
Our general approach to address questions 1 and 2 above is to start with
an nlme model with no covariates to explain random-eects variation, and
use plots of the estimated random eects versus the candidate covariates
to identify interesting patterns. Because the random eects accommodate
individual departures from the population mean, plotting the estimated
random eects against the candidate covariates provides useful information
for the model-building process. A systematic pattern in a given random
eect with respect to a covariate would indicate that the covariate should
be included in the model.
If no interesting patterns are observed, we keep the current model. Otherwise, we choose the covariate-coecient pair with the most promising
pattern and test for the inclusion of the covariate in the model. After a
covariate has been included in the model, we repeat the procedure for the
estimated random eects in the updated model and the remaining candidate covariates, searching for further useful covariates.
The number of additional parameters to be estimated tends to grow
considerably with the inclusion of covariates and their associated random
eects in the model. If the number of covariate-coecient combinations is
large, we suggest using a forward stepwise approach in which covariatecoecient pairs are included in the model one at a time and the potential
368
importance of the remaining covariates is graphically assessed at each step.

The signicance of the xed-eects associated with a covariate included in
the model is assessed using the Wald-type tests that are described in 7.2.2
and are included in the output of the summary and anova methods for nlme
objects.
The inclusion of new random eects in the model when a covariate is
added is rare, but should be investigated. The more common situation is
that random eects can be eliminated from the model after covariates are
included to account for intergroup variation. In both cases we proceed by
comparing nested models using either likelihood ratio tests, or information
criterion statistics (AIC and BIC).
We illustrate the use of the proposed model-building strategy with two
examples, one from an experiment to evaluate the cold tolerance of grass
species and the other from a clinical study of the antiarrhytimic drug quinidine.
Carbon Dioxide Uptake
Potvin, Lechowicz and Tardif (1990) report data from a study of the cold
tolerance of a C4 grass species, Echinochloa crus-galli. A total of 12 fourweek-old plants, 6 from Quebec and 6 from Mississippi, were divided into
two groups: control plants that were kept at 26 C and chilled plants that
were subject to 14 h of chilling at 7 C. After 10 h of recovery at 20 C,
carbon dioxide (CO2 ) uptake rates (in mol/m2 s) were measured for each
plant at seven concentrations of ambient CO2 (L/L). The objective of the
experiment was to evaluate the eect of plant type and chilling treatment on
the CO2 uptake. The CO2 data, displayed in Figure 8.15 (and in Figure 3.6,
p. 113), are available in the nlme library as the groupedData object CO2.
More details about these data are given in Appendix A.5.
> CO2
Grouped Data: uptake ~ conc | Plant
Plant
Type Treatment conc uptake
1
Qn1
Quebec nonchilled
95
16.0
2
Qn1
Quebec nonchilled 175
30.4
. . .
83
Mc3 Mississippi
chilled 675
18.9
84
Mc3 Mississippi
chilled 1000
19.9
> plot(CO2, outer = ~Treatment*Type, layout = c(4,1)) # Figure 8.15
It is clear from Figure 8.15 that the CO2 uptake rates of Quebec plants
are greater than those of Mississippi plants and that chilling the plants
reduces their CO2 uptake rates.
An asymptotic regression model with an oset is used in Potvin et al.
(1990) to represent the expected CO2 uptake rate U (c) as a function of the
ambient CO2 concentration c:
U (c) = 1 {1 exp [ exp (2 ) (c 3 )]} ,
(8.6)
200
400
Quebec
nonchilled
600
800
1000
200
Quebec
chilled
Mississippi
nonchilled
400
600
369
800
1000
Mississippi
chilled
40
30
20
10
200
400
600
800
1000
200
400
600
800
1000
FIGURE 8.15. CO2 uptake versus ambient CO2 by treatment and type for
Echinochloa crus-galli plants, 6 from Quebec and 6 from Mississippi. Half the
plants of each type were chilled overnight before the measurements were taken.
where 1 is the asymptotic CO2 uptake rate, 2 is the logarithm of the rate
constant, and 3 is the maximum ambient concentration of CO2 at which
there is no uptake. The logarithm of the rate constant is used to enforce the
positivity of the estimated rate constant, while keeping the optimization
problem unconstrained.
The selfStart function SSasympOff gives a self-starting implementation of
model (8.6), which is used to automatically generate starting estimates for
the parameters in an nlsList t.
> fm1CO2.lis <- nlsList( SSasympOff, CO2)
> fm1CO2.lis
Call:
Model: uptake ~ SSasympOff(conc, Asym, lrc, c0) | Plant
Data: CO2
Coefficients:
Asym
lrc
c0
Qn1 38.140 -4.3807 51.221
Qn2 42.872 -4.6658 55.856
. . .
Mc3 18.535 -3.4654 67.843
Mc1 21.787 -5.1422 -20.395
with 1 = Asym, 2 = lrc, and 3 = c0.

The plot of the individual condence intervals from fm1CO2.lis (not
shown) indicates that there is substantial between-plant variation in the
370
asymptote Asym and only moderate variation in the log-rate lrc and the
oset c0. We initially consider a nonlinear mixed-eects version of the CO2
uptake model (8.6) with all parameters as mixed eects and no treatment
covariates. The corresponding model for the CO2 uptake uij of plant i at
ambient CO2 concentration cij is
uij = 1i {1 exp [ exp (2i ) (cij 3i )]} + ij ,

1
b1i
1i
i = 2i = 2 + b2i = + bi ,
3i
3
b3i

bi N (0, ) , ij N 0, 2 ,
(8.7)
where 1i , 2i , and 3i have the same interpretation as in model (8.6),

but are now allowed to vary with plant. The xed eects, , represent
the population average of the individual parameters, i , and the random
eects, bi , represent the deviations of the i from their population average.
The random eects are assumed to be independent for dierent plots and
the within-group errors ij are assumed to be independent for dierent i, j
and to be independent of the random eects.
The nlsList object fm1CO2.lis is used to produce starting estimates for
the nlme t of model (8.7).
> fm1CO2.nlme <- nlme( fm1CO2.lis )
> fm1CO2.nlme
Model: uptake ~ SSasympOff(conc, Asym, lrc, c0)
Data: CO2
Fixed: list(Asym ~ 1, lrc ~ 1, c0 ~ 1)
Asym
lrc
c0
32.474 -4.6362 43.543
Random effects:
Formula: list(Asym ~ 1, lrc ~ 1, c0 ~ 1)
Level: Plant
StdDev
Corr
Asym 9.50999 Asym
lrc
lrc 0.12828 -0.160
c0 10.40519 0.999 -0.139
Residual 1.76641
The very high correlation between Asym and c0 suggests that the randomeects model is over-parameterized. The scatter plot matrix of the estimated random eects (not shown) conrms that Asym and c0 are in almost
371
Qc3
2
1
0
-1
-2
Qc2
-3
Qc3
10
20
30
40
Fitted values (umol/m^2 s)
fm2CO2.nlme.
perfect linear alignment. A simpler model with just Asym and lrc as random
eects gives an equivalent t of the data.
> fm2CO2.nlme <- update( fm1CO2.nlme, random = Asym + lrc ~ 1 )
> fm2CO2.nlme
. . .
Random effects:
Formula: list(Asym ~ 1, lrc ~ 1)
Level: Plant
StdDev
Corr
Asym 9.65939 Asym
lrc 0.19951 -0.777
Residual 1.80792
. . .
> anova( fm1CO2.nlme, fm2CO2.nlme )
Model df
AIC
BIC logLik
fm1CO2.nlme
1 10 422.62 446.93 -201.31
fm2CO2.nlme
2 7 419.52 436.53 -202.76 1 vs 2 2.8961 0.4079
The plot of the standardized residuals versus the tted values in Figure 8.16 does not indicate any violations from the assumptions on the
within-group error. The residuals are distributed symmetrically around
zero, with uniform variance. Two large standardized residuals are observed
for plant Qc3 and one for plant Qc2.
> plot( fm2CO2.nlme,id = 0.05,cex = 0.8,adj = -0.5 )
# Figure 8.16
The normal plot of the within-group residuals, not shown, does not indicate
violations in the normality of the within-group errors.
372
The primary question of interest for the CO2 data is the eect of plant
type and chilling treatment on the individual model parameters i . The
random eects accommodate individual deviations from the xed eects.
Plotting the estimated random eects against the candidate covariates provides useful information for choosing covariates to include in the model.
First, we need to extract the estimated random eects from the tted
model and combine them with the covariates. The ranef method accomplishes that.
> fm2CO2.nlmeRE <- ranef( fm2CO2.nlme, augFrame = T )
> fm2CO2.nlmeRE
Asym
lrc
Type Treatment conc uptake
Qn1
6.17160 0.0483563
Quebec nonchilled 435 33.229
Qn2 10.53264 -0.1728531
Qn3 12.21810 -0.0579930
Qc1
3.35213 -0.0755880
Quebec
chilled 435 29.971
Qc3
7.47431 -0.1924203
Quebec
chilled 435 32.586
Qc2
7.92855 -0.1803391
Quebec
chilled 435 32.700
Mn3 -4.07333 0.0334485 Mississippi nonchilled 435 24.114
Mn2 -0.14198 0.0056463 Mississippi nonchilled 435 27.343
Mn1
0.24056 -0.1938500 Mississippi nonchilled 435 26.400
Mc2 -18.79914 0.3193732 Mississippi
chilled 435 12.143
Mc3 -13.11688 0.2994393 Mississippi
chilled 435 17.300
Mc1 -11.78655 0.1667798 Mississippi
chilled 435 18.000
> class( fm2CO2.nlmeRE )
[1] "ranef.lme" "data.frame"
The augFrame argument, when TRUE, indicates that summary values for all
the variables in the data frame should be returned along with the random
eects. The summary values are calculated as in the gsummary function
(3.4). When a covariate is constant within a group, such as Treatment and
Type in the CO2 data, its unique values per group are returned. Otherwise,
if the covariate varies within the group and is numeric, such as conc and
uptake in CO2, the group means are returned; if it is a categorical variable
(factor or ordered), the most frequent values (modes) within each group are
returned.
The plot method for the ranef.lme class is the most useful tool for identifying relationships between individual parameters and covariates. The form
argument is used to specify the desired covariates and plot type. A onesided formula on the right-hand side, with covariates separated by the *
operator, results in a dotplot of the estimated random eects versus all
combinations of the unique values of the variables named in the formula.
This plot is particularly useful for a moderate number of categorical covariates (factor or ordered variables) with a relatively small number of levels,
as in the CO2 example.
> plot( fm2CO2.nlmeRE, form = ~ Type * Treatment )
# Figure 8.17

Asym
373
lrc
Mississippi chilled
Mississippi nonchilled
Quebec chilled
Quebec nonchilled
20
-15
-10
-5
10
0.2
0.0
0.1
0.2
0.3
Random effects
FIGURE 8.17. Dotplots of estimated random eects corresponding to

fm2CO2.nlme versus all combinations of plant type and chilling treatment.
Figure 8.17 shows a strong relationship between the asymptotic update

rate and the covariates: Asym decreases when the plants are chilled and
is higher among Quebec plants than Mississippi plants. The increase in
Asym from chilled to nonchilled plants is larger among Mississippi plants
than Quebec plants, suggesting an interaction between Type and Treatment.
There is also some evidence of a Type:Treatment interaction on the lograte lrc, but it is less striking than in the case of Asym. We include both
covariates in the model to explain the Asym plant-to-plant variation. The
only change required in model (8.7) is in the formulation of 1i .
1i = 1 + 1 x1i + 2 x2i + 3 x1i x2i + b1i ,
+
1, Type of Plant i = Quebec,
x1i =
1, Type of Plant i = Mississippi,
+
1, Treatment of Plant i = nonchilled,
x2i =
1, Treatment of Plant i = chilled,
(8.8)
where 1 represents the average asymptotic uptake rate, 1 and 2 represent, respectively, the plant type and chilling treatment main eects,
and 3 represents the plant typechilling treatment interaction. The parameterization used for x1i and x2i in (8.8) is consistent with the default
parameterization for factors in S.
> contrasts(CO2$Type)
[,1]
Quebec
-1
Mississippi
1
> contrasts(CO2$Treatment)
[,1]
nonchilled
-1
chilled
1
The update method is used to t the model with the covariate terms,
which are specied through the fixed argument.
374
> fm3CO2.nlme <- update( fm2CO2.nlme,

+
fixed = list(Asym ~ Type * Treatment, lrc + c0 ~ 1),
+
start = c(32.412, 0, 0, 0, -4.5603, 49.344) )
Because the xed-eects model has been reformulated, new starting values
must be provided. We use the previous estimates for 1 , 2 and 3 and set
the initial values for 1 , 2 and 3 to zero. The xed eects are represented
internally in nlme in the same order they appear in fixed.
The summary method gives information about the signicance of the individual xed eects.
> summary( fm3CO2.nlme )
. . .
AIC
BIC logLik
393.68 417.98 -186.84
Random effects:
Level: Plant
StdDev
Corr
Asym.(Intercept) 2.92980 Asym.(
lrc 0.16373 -0.906
Residual 1.84957
Fixed effects: list(Asym ~ Type * Treatment, lrc + c0 ~ 1)
Asym.(Intercept) 32.447
0.9359 67 34.670 <.0001
Asym.Type -7.108
0.5981 67 -11.885 <.0001
Asym.Treatment -3.815
0.5884 67 -6.483 <.0001
Asym.Type:Treatment -1.197
0.5884 67 -2.033
0.046
lrc -4.589
0.0848 67 -54.108 <.0001
c0 49.479
4.4569 67 11.102 <.0001
. . .
The names of the xed-eects terms include the parameter name. All xed
eects introduced in the model to explain the variability in Asym are significantly dierent from zero at the 5% level, conrming the previous conclusions from Figure 8.17. The joint signicance of the xed eects introduced
in the model can be tested with the anova method.
> anova( fm3CO2.nlme, Terms = 2:4 )
F-test for: Asym.Type, Asym.Treatment, Asym.Type:Treatment
1
3
67 54.835 <.0001
As expected, the approximate F-test indicates that the added terms are
highly signicant.
The inclusion of the experimental factors in the model resulted in a
reduction in the estimated standard deviation for the Asym random eects

Asym.(Intercept)
375
lrc
Mississippi chilled
Mississippi nonchilled
Quebec chilled
Quebec nonchilled
-6
-4
-2
-0.1
0.0
0.1
0.2
0.3
Random effects
FIGURE 8.18. Dotplots of estimated random eects corresponding to

fm3CO2.nlme versus all combinations of plant type and chilling treatment.
from 9.66 to 2.93, indicating that a substantial part of the plant-to-plant

variation in the asymptotic uptake rate is explained by dierences in plant
type and chilling treatment. The standard deviations for the lrc random
eects and the within-group error remained about the same.
We now investigate if any covariates should be included to account for
the variability in the lrc random eects.
> fm3CO2.nlmeRE <- ranef( fm3CO2.nlme, aug = T )
> plot( fm3CO2.nlmeRE, form = ~ Type * Treatment )
# Figure 8.18
No systematic pattern can be observed for the estimated Asym.(Intercept)

random eects in Figure 8.18, but it appears that the chilling treatment
has opposite eects on Quebec and Mississippi plants, suggesting an interaction. As before, we t the augmented model with update, setting the
initial values for the new xed eects in the model to zero, and test the
signicance of the new terms using summary.
> fm3CO2.fix <- fixef( fm3CO2.nlme )
# for starting values
> fm4CO2.nlme <- update( fm3CO2.nlme,
+
fixed = list(Asym + lrc ~ Type * Treatment, c0 ~ 1),
+
start = c(fm3CO2.fix[1:5], 0, 0, 0, fm3CO2.fix[6]) )
> summary( fm4CO2.nlme )
. . .
Random effects:
Level: Plant
StdDev
Corr
Asym.(Intercept) 2.349663 Asym.(
lrc.(Intercept) 0.079608 -0.92
Residual 1.791950
Fixed effects: list(Asym + lrc ~ Type * Treatment, c0 ~ 1)
376
Asym.(Intercept)
Asym.Type
Asym.Treatment
Asym.Type:Treatment
lrc.(Intercept)
lrc.Type
lrc.Treatment
lrc.Type:Treatment
c0
. . .

32.342
0.7849 64 41.208 <.0001
-7.990
0.7785 64 -10.264 <.0001
-4.210
0.7781 64 -5.410 <.0001
-2.725
0.7781 64 -3.502 0.0008
-4.509
0.0809 64 -55.743 <.0001
0.133
0.0552 64
2.417 0.0185
0.100
0.0551 64
1.812 0.0747
0.185
0.0554 64
3.345 0.0014
50.512
4.3646 64 11.573 <.0001
The lrc.Type:Treatment coecient is highly signicant and the lrc.Type

coecient is moderately signicant. Even though lrc.Treatment is not signicant (at a 5% level) we keep it in the model because the Treatment
eect on lrc is involved in a highly signicant interaction. The estimated
standard deviation for the lrc.(Intercept) random eect is about 50% of
the corresponding estimate in the fm3CO2.nlme t. The remaining standard
deviations are about the same as in the previous t.
After covariates have been introduced in the model to account for intergroup variation, a natural question is which random eects, if any, are still
needed. The ratio between a random-eects standard deviation and the
absolute value of the corresponding xed eect gives an idea of the relative
intergroup variability for the coecient, which is often useful in deciding
which random eects should be tested for deletion from the model. For the
fm4CO2.nlme t these ratios are 7.3% for Asym.(Intercept) and 1.8% for
lrc.(Intercept), suggesting that the latter should be tested for exclusion
rst.
> fm5CO2.nlme <- update( fm4CO2.nlme, random = Asym ~ 1)
> anova( fm4CO2.nlme, fm5CO2.nlme )
Model df
AIC
BIC logLik
fm4CO2.nlme
1 13 388.42 420.02 -181.21
fm5CO2.nlme
2 11 387.06 413.79 -182.53 1 vs 2 2.6369 0.2675
The large p-value for the likelihood ratio test and the smaller AIC and BIC
values for fm5CO2.nlme indicate that no random eects are needed for lrc.
To test if a random eect is needed for the asymptotic uptake rate,
we need to t a nonlinear xed-eects model to the CO2 data. The nls
function can be used for that, though it is not designed to eciently handle
parameters that are expressed as linear combinations of covariates. (The
gnls function, described in 8.3.3, is better suited for this type of model.)
To use nls, we must rst create variables representing the contrasts of
interest
> CO2$type <- 2 * (as.integer(CO2$Type) - 1.5)
> CO2$treatment <- 2 * (as.integer(CO2$Treatment) - 1.5)
and then dene all coecients explicitly in the model formula.

fixed
200
Mn3
400
600
800
Plant
1000
200
Mn2
377
Mn1
400
600
800
1000
200
Mc2
Mc3
400
600
800
1000
Mc1
40
30
20
10
Qn1
Qn2
Qn3
Qc1
Qc3
Qc2
40
30
20
10
200
400
600
800
1000
200
400
600
800
1000
200
400
600
800
1000
FIGURE 8.19. Plant-specic (Plant) and population average (fixed) predicted

CO2 uptake rates obtained from fm5CO2.nlme.
> fm1CO2.nls <- nls(uptake ~ SSasympOff(conc, Asym.Intercept +

+
Asym.Type * type + Asym.Treatment * treatment +
+
Asym.TypeTreatment * type * treatment, lrc.Intercept +
+
lrc.Type * type + lrc.Treatment * treatment +
+
lrc.TypeTreatment * type * treatment, c0), data = CO2,
+
start = c(Asym.Intercept = 32.371, Asym.Type = -8.0086,
+
Asym.Treatment = -4.2001, Asym.TypeTreatment = -2.7253,
+
lrc.Intercept = -4.5267, lrc.Type = 0.13112,
+
lrc.Treatment = 0.093928, lrc.TypeTreatment = 0.17941,
+
c0 = 50.126) )
The anova method can then be used to compare the models. (Note that the
nlme object must appear rst in the calling sequence to anova, so that the
correct method is invoked.)
> anova( fm5CO2.nlme, fm1CO2.nls )
Model df
AIC
BIC logLik
fm5CO2.nlme
1 11 387.06 413.79 -182.53
fm1CO2.nls
2 10 418.34 442.65 -199.17 1 vs 2 33.289 <.0001
The very signicant p-value for the likelihood ratio test indicates that the
Asym.(Intercept) random eect is still needed in the model.
A nal assessment of the quality of the tted model is provided by the
plot of the augmented predictions included in Figure 8.19.
> plot( augPred(fm5CO2.nlme, level = 0:1),
+
layout = c(6,2) )
# Figure 8.19
The plant-specic predictions are in good agreement with the observed

CO2 uptake rates, attesting to the adequacy of the asymptotic regression
378
model. Note that the population average predictions vary with plant type
and chilling treatment.
Clinical Study of Quinidine

The Quinidine data were described in 3.4 where we explored the structure
of the data through summaries. Like the phenobarbital data analyzed in
6.4, the quinidine data are routine clinical pharmacokinetic data characterized by extensive dosage histories for each patient, but relatively sparse
information on concentration. Recall that a total of 361 quinidine concentration measurements were made on 136 hospitalized patients under varying dosage regimens. The times since hospitalization at which the quinidine
concentrations were measured varied between 0.13 and 8095.5 hours. Most
patients have only a few concentration measurements: 34% have only one
and 80% have three or fewer. Only 5% of the patients have seven or more
observations.
Additional demographic and physiological data were collected for each
subject. The additional available covariates are described in Table 8.4.
Some of these covariates, such as age, body weight, and creatinine clearance, were time-varying. That is, their value for a particular patient
could change during the course of the study. Others, such as race, remained constant. One of the main objectives of the study was to investigate
relationships between the individual pharmacokinetic parameters and the
covariates. Statistical analyses of these data using alternative modeling approaches are given in Davidian and Gallant (1992) and in Wakeeld (1996).
The model that has been suggested for the quinidine data is the onecompartment open model with rst-order absorption. This model can be
dened recursively as follows. Suppose that, at time t, a patient receives
a dose dt and prior to that time the last dose was given at time t . The expected concentration in the serum compartment, Ct , and in the absorption
TABLE 8.4. Demographic and physiological covariates in the quinidine data.
Age (yr)
Glycoprotein concentration (mg/100 dL)
Body weight (kg)
Congestive heart failure
Creatinine clearance (ml/min)
Ethanol abuse
Height (in.)
Race
Smoking status
4292
0.393.16
41119
no/mild, moderate, severe
< 50, 50
none, current, former
6079
Caucasian, Latin, Black
no, yes
379
compartment, Ca t , are given by

Ca t ka
ka ke
{exp [ke (t t )] exp [ka (t t )]} ,
dt
Ca t =Ca t exp [ka (t t )] + ,
V
Ct =Ct exp [ke (t t )] +
(8.9)
where V is the apparent volume of distribution, ka is the absorption rate

constant, and ke is the elimination rate constant.
When a patient receives the same dose d at regular time intervals ,
model (8.9) converges to the steady state model

1
d ka
1
Ct =
,
V (ka ke ) 1 exp (ke ) 1 exp (ka )
(8.10)
d
.
Cat =
V [1 exp (ka )]
Finally, for a between-dosages time t, the model for the expected concentration Ct , given that the last dose was received at time t , is identical
to (8.9).
Using the fact that the elimination rate constant ke is equal to the ratio
between the clearance (Cl) and the volume of distribution (V ), we can
reparameterize models (8.9) and (8.10) in terms of V , ka , and Cl .
To ensure that the estimates of V , ka , and Cl are positive, we can rewrite
models (8.9) and (8.10) in terms of lV = log(V ), lKa = log(ka ) and
lCl = log(Cl).
The initial conditions for the recursive models (8.9) and (8.10) are C0 = 0
and Ca0 = d0 /V , with d0 denoting the initial dose received by the patient. It is assumed in the models denition that the bioavailability of the
drugthe percentage of the administered dose that reaches the measurement compartmentis equal to one.
The function quinModel in the nlme library implements the recursive
models (8.9) and (8.10) in S, parameterized in terms of lV , lKa and lCl .
This is not a self-starting model, so initial values for the xed eects need
to be provided when calling nlme. We used values reported in the literature
as starting estimates for the xed eects.
Preliminary analyses of the data, without using any covariates to explain intersubject variation, indicates that only lCl and lV need random
eects to account for their variability in the patient population, and that
the corresponding random eects can be assumed to be independent. The
corresponding model for the xed and random eects is
lCl i = 1 + b1i , lV i = 2 + b2i , lKa i = 3 ,

b1i
1 0
N 0,
,
bi =
b2i
0 2
(8.11)
380
which is tted in S with

> fm1Quin.nlme <+
nlme(conc ~ quinModel(Subject, time, conc, dose, interval,
+
lV, lKa, lCl),
+
data = Quinidine, fixed = lV + lKa + lCl ~ 1,
+
random = pdDiag(lV + lCl ~ 1), groups = ~ Subject,
+
start = list(fixed = c(5, -0.3, 2)),
+
na.action = na.include, naPattern = ~ !is.na(conc) )
> fm1Quin.nlme
Model: conc ~ quinModel(Subject, time, conc, dose, interval, ...
Data: Quinidine
Fixed: lV + lKa + lCl ~ 1
lV
lKa
lCl
5.3796 -0.20535 2.4687
Random effects:
Formula: list(lV ~ 1, lCl ~ 1)
Level: Subject
Structure: Diagonal
lV
lCl Residual
StdDev: 0.31173 0.32276 0.73871
The na.action and naPattern arguments in this call to nlme are described
in 6.4.
To investigate which covariates may account for patient-to-patient variation in the pharmacokinetic parameters, we rst extract the estimated
random eects, augmented with summary values for the available covariates (the modal value is used for time-varying factors and the mean for
time-varying numeric variables).
> fm1Quin.nlmeRE <- ranef( fm1Quin.nlme, aug = T )
> fm1Quin.nlmeRE[1:3,]
lV
lCl time
conc dose interval
109 0.0005212 -0.0028369 61.58 0.50000
NA
NA
70 0.0362214 0.3227614 1.50 0.60000
NA
NA
23 -0.0254211 0.4402551 91.14 0.56667
NA
NA
Weight
Race Smoke Ethanol
Heart Creatinine
109
58 Caucasian
no
none No/Mild
>= 50
70
75 Caucasian
no former No/Mild
>= 50
23
108 Caucasian
yes
none No/Mild
>= 50
Age Height
70
67
68
69
75
72
glyco
0.46000
1.15000
0.83667
The dotplot displays used to visualize the relationships between the estimated random eects and the covariates in the CO2 example do not scale
381
up well when there are a large number of groups, or a large number of

covariates in the data, as in the quinidine study. Also, they cannot be used
with numeric covariates, like Weight and Age. The plot method for class
ranef allows a more exible type of trellis display for these situations. Relationships between estimated random eects and factors are displayed using
boxplots, while scatter plots are used for displaying the relationships between the estimated random eects and numeric covariates. Specifying a
two-sided formula in the form argument, with the random eect on lefthand side and the desired covariates, separated by the + operator, on the
right-hand side, indicates to the plot method that the more general trellis
display should be used. For example, to plot the estimated lCl random
eects against the available covariates we use
> plot( fm1Quin.nlmeRE, form = lCl ~ Age + Smoke + Ethanol +
+
Weight + Race + Height + glyco + Creatinine + Heart,
+
control = list(cex.axis = 0.7) )
# Figure 8.20
The resulting plot, shown in Figure 8.20, indicates that clearance decreases with glycoprotein concentration and age, and increases with creatinine clearance and weight. There is also evidence that clearance decreases
with severity of congestive heart failure and is smaller in Blacks than in
both Caucasians and Latins. The glycoprotein concentration is clearly the
most important covariate for explaining the lCl interindividual variation.
A straight line seems adequate to model the observed relationship.
Figure 8.21 presents the plots of the estimated lV random eects versus
the available covariates. None of the covariates seems helpful in explaining
the variability of this random eect and we do not pursue the modeling of
its variability any further.
Initially, only the glycoprotein concentration is included in the model to
explain the lCl random-eect variation according to a linear model. This
modication of model (8.11) is accomplished by writing
lCl ij = (1 + b1i ) + 2 glycoij .
(8.12)
Because the glycoprotein concentration may change with time on the same
patient, the random eects for lCl need to be indexed by both patient i
and time j. We t the mixed-eects model corresponding to (8.12) with
>
>
+
+
>
.
fm1Quin.fix <- fixef( fm1Quin.nlme)

# for initial values
fm2Quin.nlme <- update( fm1Quin.nlme,
fixed = list(lCl ~ glyco, lKa + lV ~ 1),
start = c(fm1Quin.fix[3], 0, fm1Quin.fix[2:1]) )
summary( fm2Quin.nlme )
. .
AIC
BIC logLik
891.3 918.52 -438.65
382

glyco
Creatinine
Heart
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
0.5
1.0
1.5
2.0
2.5
3.0
< 50
Weight
>= 50
No/Mild
Moderate
Race
Severe
Height
0.6
0.4
0.2
lCl
0.0
-0.2
-0.4
-0.6
-0.8
40
60
80
100
120 Caucasian
Age
Latin
Black
60
65
Smoke
70
75
Ethanol
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
-0.8
40
50
60
70
80
90
no
yes
none
current
former
FIGURE 8.20. Estimated log-clearance random eects from model fm1Quin.nlme

versus demographic and physiological covariates in the quinidine data. A loess
smoother is included in the scatter plots of the continuous covariates to aid in
visualizing possible trends.
Random effects:
Level: Subject
Structure: Diagonal
lV lCl.(Intercept) Residual
StdDev: 0.26764
0.27037 0.63746
Fixed effects: list(lCl ~ glyco, lKa + lV ~ 1)
lCl.(Intercept) 3.1067
0.06473 222 47.997 <.0001
lCl.glyco -0.4914
0.04263 222 -11.527 <.0001
lKa -0.6662
0.30251 222 -2.202 0.0287
lV 5.3085
0.10244 222 51.818 <.0001
. . .

glyco
0.6
Creatinine
383
Heart
0.4
0.2
0.0
-0.2
0.5
1.0
1.5
2.0
2.5
3.0
< 50
Weight
>= 50
No/Mild
Race
Moderate
Severe
Height
0.6
0.4
lV
0.2
0.0
-0.2
40
60
80
100
120 Caucasian
Age
0.6
Latin
Black
60
65
Smoke
70
75
Ethanol
0.4
0.2
0.0
-0.2
40
50
60
70
80
90
no
yes
none
current
former
FIGURE 8.21. Estimated log-volume random eects from model fm1Quin.nlme

The estimated lCl.glyco xed eect is very signicant, indicating that the
glycoprotein concentration should be kept in the model.
To search for further variables to include in the model, we consider the
plots of the estimated lCl.(Intercept) random eects from the fm2Quin.nlme
t versus the covariates, presented in Figure 8.22.
These plots indicate that the estimated lCl.(Intercept) random eects
increase with creatinine clearance, weight, and height, decrease with age
and severity of congestive heart failure, and are smaller in Blacks than
in Caucasians and Latins. The most relevant variable appears to be the
creatinine clearance, which is included in the model as a binary variable
taking value 0 when creatinine is < 50 and 1 when creatinine is 50.
384

glyco
Creatinine
Heart
0.6
0.4
0.2
0.0
-0.2
-0.4
0.5
1.0
1.5
2.0
2.5
3.0
< 50
Weight
>= 50
No/Mild
Race
Moderate
Severe
Height
0.6
lCl.(Intercept)
0.4
0.2
0.0
-0.2
-0.4
40
60
80
100
120 Caucasian
Age
Latin
Black
60
65
Smoke
70
75
Ethanol
0.6
0.4
0.2
0.0
-0.2
-0.4
40
50
60
70
80
90
no
yes
none
current
former
FIGURE 8.22. Estimated log-clearance random eects from model fm2Quin.nlme

> fm2Quin.fix <- fixef( fm2Quin.nlme )

> fm3Quin.nlme <- update( fm2Quin.nlme,
+
fixed = list(lCl ~ glyco + Creatinine, lKa + lV ~ 1),
+
start = c(fm2Quin.fix[1:2], 0.2, fm2Quin.fix[3:4]) )
> summary( fm3Quin.nlme )
. . .
Fixed effects: list(lCl ~ glyco + Creatinine, lKa + lV ~ 1)
0.06387 221 47.426 <.0001
lCl.glyco -0.4631
0.04117 221 -11.249 <.0001
lCl.Creatinine 0.1503
0.03175 221
4.732 <.0001
lKa -0.7458
0.29619 221 -2.518 0.0125
lV 5.2893
0.10625 221 49.784 <.0001
. . .
385
The nal model produced by this stepwise model-building approach includes an extra term for the patients body weight to explain the clearance
variation. The corresponding model for the log-clearance is expressed as
lCl ij = (1 + b1i ) + 4 glycoij + 5 Creatinineij + 6 Weightij
(8.13)
and is t in S with
>
>
+
+
>
.
fm3Quin.fix <- fixef( fm3Quin.nlme )

fm4Quin.nlme <- update( fm3Quin.nlme,
fixed = list(lCl ~ glyco + Creatinine + Weight, lKa + lV ~ 1),
start = c(fm3Quin.fix[1:3], 0, fm3Quin.fix[4:5]) )
summary( fm4Quin.nlme )
. .
AIC
BIC logLik
870.94 905.94 -426.47
Random effects:
Level: Subject
Structure: Diagonal
StdDev: 0.28154
0.24128 0.63083
Fixed effects: list(lCl~glyco + Creatinine + Weight, lKa+lV ~ 1)
0.15167 220 18.384 <.0001
lCl.glyco -0.4645
0.04100 220 -11.328 <.0001
0.03264 220
4.207 <.0001
lCl.Weight 0.0031
0.00180 220
1.749 0.0816
lKa -0.7974
0.29959 220 -2.662 0.0083
lV 5.2833
0.10655 220 49.587 <.0001
. . .
The lCl.Weight coecient is not signicant at a 5% level, but it is signicant at a less conservative 10% level. Given the high level of noise and the
small number of observations per patient in the quinidine data, we considered a p-value of 8.2% to be small enough to justify the inclusion of Weight
in the model.
As reported in previous analyses of the quinidine data (Davidian and
Giltinan, 1995, 9.3), there is evidence that the variability in the concentration measurements increases with the quinidine concentration. We
postpone the investigation of heteroscedasticity in the quinidine data until
8.3.1, when we describe the use of variance functions in nlme.
8.2.3 Fitting Multilevel nlme Models

Nonlinear mixed-eects models with nested grouping factors, called multilevel nonlinear mixed-eects models, are described in 7.1.2. In this section
386
we describe how to t and analyze them with the nlme library. The Wafer
example of 4.2.3 is used to illustrate the various methods available in nlme
for multilevel NLME models.
As in the linear case, the only dierence between a single-level and a
multilevel t in nlme is in the specication of the random argument. In the
multilevel case, random must provide information about the nested grouping
structure and the random-eects model at each level of grouping. This is
generally accomplished by specifying random as a named list, with names
corresponding to the grouping factors. The order of nesting is assumed to
be the same as the order of the elements in the list, with the outermost
grouping factor appearing rst and the innermost grouping factor appearing last. Each element of the random list has the same structure as random
in a single-level call: it can be a formula, a list of formulas, a pdMat object,
or a list of pdMat objects. Most of the nlme extractors, such as resid and
ranef, include a level argument indicating the desired level(s) of grouping.
A multilevel linear mixed-eects analysis of the Wafer data is presented in
4.2.3 to illustrate the multilevel capabilities of lme. The nal multilevel
model obtained in that section to represent the intensity of current yijk at
the kth level of voltage vk in the jth site within the ith wafer is expressed,
for i = 1, . . . , 10, j = 1, . . . , 8, and k = 1, . . . , 5, as
yijk = (0 + b0i + b0i,j ) + (1 + b1i + b1i,j ) vk + (2 + b2i + b2i,j ) vk2
+ 3 cos (vk ) + 4 sin (vk ) + ijk
b0i
b0i,j
bi = b1i N (0, 1 ) ,
bi,j = b1i,j N (0, 2 ) ,
b2i
b2i,j

2
ijk N 0, .
(8.14)
where 0 , 1 , and 2 are the xed eects in the quadratic model, 3 and
4 are the xed eects for the cosine wave of frequency , bi is the waferlevel random eects vector, bi,j is the site within wafer-level random-eects
vector, and ijk is the within-group error. The bi are assumed to be independent for dierent i, the bi,j are assumed to be independent for dierent i, j and independent of the bi , and the ijk are assumed to be independent for dierent i, j, k and independent of the random eects. The
wafer-level variancecovariance matrix 1 is general positive-denite and
the site-within-wafer-level matrix 2 is block-diagonal, with a 1 1 block
corresponding to the variance of b0i,j and a 2 2 block corresponding to
variancecovariance matrix of [b1i,j , b2i,j ]T .
Because its model function is nonlinear in the frequency , model (8.14)
is actually an example of a multilevel nonlinear mixed-eects model. It
387
is treated as a linear model in 4.2.3 by holding xed at a previously

estimated value. Some of the disadvantages of this approach are that no
precision can be attached to the estimate of and we cannot test if random
eects are needed to account for variation in , either at the wafer level, or
at the site-within-wafer level. In this section we treat (8.14) as a multilevel
NLME model, allowing to be estimated with the other parameters and
testing if random eects are needed to account for its variation in the data.
We can rewrite (8.14) in the usual two-stage NLME model formulation.
yijk = 1ij + 2ij cos (vk ) + 3ij sin (vk ) + ijk ,
1ij = (0 + b0i + b0i,j ) + (1 + b1i + b1i,j ) vk + (2 + b2i + b2i,j ) vk2 ,
2ij = 3 ,
3ij = 4 ,
(8.15)
where the xed eects, = (1 , . . . , 4 ), the random eects, bi , bi,j , and
the within-group errors, ijk , are dened as in (8.14). To illustrate some of
the multilevel capabilities in nlme, we initially t model (8.15) with xed
= 4.5679 as in 4.2.3. To get results that are comparable to the fm5Wafer
t in 4.2.3, we need to set the estimation method in nlme to REML.
> fm1Wafer.nlmeR <- nlme( current ~ A + B * cos(4.5679 * voltage) +
+
C * sin(4.5679 * voltage), data = Wafer,
+
fixed = list(A ~ voltage + voltage^2, B + C ~ 1),
+
random = list(Wafer = A ~ voltage + voltage^2,
+
Site = pdBlocked(list(A~1, A~voltage+voltage^2-1))),
+
start = fixef(fm4Wafer), method = "REML")
> fm1Wafer.nlmeR
. . .
Fixed: list(A ~ voltage + voltage^2, B + C ~ 1)
A.(Intercept) A.voltage A.I(voltage^2)
B
C
-4.2554
5.6224
1.2585 -0.095557 0.10435
Random effects:
Formula: A ~ voltage + voltage^2 | Wafer
StdDev
Corr
A.(Intercept) 0.131805 A.(In) A.vltg
A.voltage 0.354743 -0.967
A.I(voltage^2) 0.049955 0.814 -0.935
Block 1: A.(Intercept)
Formula: A ~ 1 | Site %in% Wafer
A.(Intercept)
StdDev:
0.066563
388
Block 2: A.voltage, A.I(voltage^2)

Formula: A ~ voltage + voltage^2 - 1 | Site %in% Wafer
StdDev
Corr
A.voltage 0.2674083 A.volt
A.I(voltage^2) 0.0556444 -0.973
Residual 0.0091086
. . .
The only dierence in the multilevel model specication in random between

lme and nlme is the use of one-sided formulas in the former and two-sided
formulas in the latter. As expected, the estimation results for fm1Wafer.nlme
are almost identical to the ones for fm5Wafer.
Because only ve distinct voltages are used in the MOS circuit experiment, at most ve dierent xed eects can be used in a mixed-eects
model tted to the Wafer data. This is true in general: the number of estimable xed eects in a mixed-eects model cannot exceed the number of
distinct design points in the data used to t it. Therefore, in order to allow
the frequency to be estimated from the data, we need to drop at least
one xed eect from model (8.15).
Examining the xed eects estimates in fm1Wafer.nlme we see that 3
4 . We make the assumption that 3 = 4 in (8.15), which, using the
identity cos() sin() = cos( + /4), gives the modied model
yijk = 1ij + 2ij cos (ij vk + /4) + ijk .
(8.16)
To compensate for the restriction that 3 = 4 , we include random eects

for 2ij at the wafer and site-within-wafer levels. Preliminary analyses of
the modied model indicated that random eects for ij are needed at the
wafer level only. The corresponding model for the xed and random eects
is
1ij = (0 + b0i + b0i,j ) + (1 + b1i + b1i,j ) vk + (2 + b2i + b2i,j ) vk2 ,
2ij = 3 + b3i + b3i,j ,
ij = 4 + b4i ,
b0i
b0i,j
b1i
b1i,j
bi,j =
bi =
b2i N (0, 1 ) ,
b2i,j N (0, 2 ) ,
b3i
b3i,j
b4i

ijk N 0, 2 .
(8.17)
Because of the large number of random eects at each grouping level, to
make the estimation problem more numerically stable, we make the simplifying assumption that the random eects are independent. That is, we
assume that 1 and 2 are diagonal matrices.
389
A maximum likelihood t of model (8.16), with xed and random-eects

structures given in (8.17), is obtained with
> fm2Wafer.nlme <- nlme( current ~ A + B * cos(w * voltage + pi/4),
+
data = Wafer, fixed = list(A ~ voltage + voltage^2, B + w ~ 1),
+
random = list(Wafer = pdDiag(list(A ~ voltage + voltage^2,
+
B + w ~ 1)),
+
Site = pdDiag(list(A ~ voltage+voltage^2, B ~ 1))),
+
start = c(fixef(fm1Wafer.nlme)[-5], 4.5679) )
> fm2Wafer.nlme
Model: current ~ A + B * cos(w * voltage + pi/4)
Data: Wafer
Log-likelihood: 766.44
Fixed: list(A ~ voltage + voltage^2, B + w ~ 1)
B
w
-4.2653
5.6329
1.256 -0.14069 4.5937
Random effects:
Formula: list(A ~ voltage + voltage^2, B ~ 1, w ~ 1)
Level: Wafer
Structure: Diagonal
B
w
StdDev:
0.1332
0.34134
0.048243 0.0048037 0.014629
Formula: list(A ~ voltage + voltage^2, B ~ 1)
Level: Site %in% Wafer
Structure: Diagonal
B Residual
StdDev:
0.084082
0.30872
0.067259 0.0067264 0.0008428
. . .
Even though models (8.15) and (8.16) are not nested, they can compared
using information criterion statistics. The anova method can be used for
that, but we must rst obtain a maximum likelihood t of model (8.15).
> fm1Wafer.nlme <- update( fm1Wafer.nlmeR, method = "ML" )
> anova( fm1Wafer.nlme, fm2Wafer.nlme, test = F )
Model df
AIC
BIC logLik
fm1Wafer.nlme
1 16 -1503.9 -1440.1 767.96
fm2Wafer.nlme
2 15 -1502.9 -1443.0 766.44
The more conservative BIC favors the model with fewer parameters,
fm2Wafer.nlme, while the more liberal AIC favors the model with larger
log-likelihood, fm1Wafer.nlme.
The intervals method is used to obtain condence intervals on the xed
eects and the variance components.
> intervals( fm2Wafer.nlme )
390
Fixed effects:
lower
est.
upper
A.(Intercept) -4.35017 -4.26525 -4.18033
A.voltage 5.40994 5.63292 5.85589
A.I(voltage^2) 1.22254 1.25601 1.28948
B -0.14403 -0.14069 -0.13735
w 4.58451 4.59366 4.60281
Random Effects:
Level: Wafer
sd(A.(Intercept))
sd(A.voltage)
sd(A.I(voltage^2))
sd(B)
sd(w)
Level: Site
sd(A.(Intercept))
sd(A.voltage)
sd(A.I(voltage^2))
sd(B)
lower
0.0693442
0.1715436
0.0222685
0.0022125
0.0078146
est.
0.1331979
0.3413426
0.0482433
0.0048037
0.0146290
upper
0.255849
0.679214
0.104516
0.010430
0.027385
lower
0.0664952
0.2442003
0.0532046
0.0053128
est.
0.0840825
0.3087244
0.0672589
0.0067264
upper
0.1063214
0.3902973
0.0850257
0.0085162

lower
est.
upper
0.00066588 0.0008428 0.0010667
The xed eects and the within-group standard error are estimated with
more relative precision than the random-eects variance components. In
the random-eects variance components, the site-within-wafer standard
deviations are estimated with greater precision than the wafer standard
deviations.
The plot of the within-group residuals versus voltage by wafer, displayed
in Figure 8.23, and produced with
> plot( fm2Wafer.nlme, resid(.) ~ voltage | Wafer,
+
panel = function(x, y, ...) {
+
panel.grid()
+
panel.xyplot(x, y)
+
panel.loess(x, y, lty = 2)
+
panel.abline(0, 0)
+
} )
# Figure 8.23
does not reveal any periodic patterns as observed, for example, in Figure 4.27, indicating that the inclusion of a random eect for accounted
successfully for variations in frequency among wafers.
8.3 Extending the Basic nlme Model

1.0
6
1.5
2.0
1.0
1.5
391
2.0
10
0.001
Residuals (mA)
0.0
-0.001
-0.002
0.001
0.0
-0.001
-0.002
1.0
1.5
2.0
1.0
1.5
2.0
1.0
1.5
2.0
Voltage (V)
FIGURE 8.23. Scatter plots of within-group residuals versus voltage by wafer

for the fm2Wafer.nlme t. A loess smoother has been added to each panel to
enhance the visualization of the residual pattern.
The normal plots of the within-group residuals and of the estimated sitewithin-wafer random eects, not shown here, do not indicate any violations
of the NLME model assumptions.

The extended nonlinear mixed-eects model with heteroscedastic, correlated within-group errors was introduced in 7.4.1. In this section, we describe the use of the nlme function for tting such models. The use of
variance functions for modeling heteroscedasticity in NLME models are discussed and illustrated in 8.3.1. Correlation structures for modeling withingroup error dependence in NLME models are illustrated in 8.3.2. The gnls
function to t the extended nonlinear regression model presented in 7.5.1
is described and illustrated in 8.3.3, together with its associated class and
methods.
8.3.1 Variance Functions in nlme

Variance functions are specied for an nlme model in the same way as for
an lme model: through the weights argument. Any of the varFunc classes
described in 5.2.1, and listed in Table 5.1, can also be used with nlme.
The same diagnostic plots discussed in 5.2.1 for identifying within-group
heteroscedasticity and assessing the adequacy of a variance function for lme
objects, can also be used with nlme objects. We revisit the theophylline
392
example of 8.2.1 and the quinidine example of 8.2.2 to illustrate the use
of variance functions in nlme.
Theophylline Kinetics
The plot of the standardized residuals versus the tted values for the tted
object fm3Theo.nlme, displayed in Figure 8.13, suggests that the withingroup variance increases with the concentration of theophylline.
The denition of the rst-order open-compartment model (8.2) implies
that the tted value for the concentration at time t = 0 is
c0 = 0. Therefore, a power variance function, a natural candidate for this type of heteroscedastic pattern, cannot be used in this example, as the corresponding
weights are undened at t = 0. (Davidian and Giltinan (1995, 5.5, p. 145)
argue that the observations at t = 0 do not add any information for the
model and should be omitted from the data. We retain them here for illustration.) The constant plus power variance function, described in 5.2 and
represented in the nlme library by the varConstPower class, accommodates
the problem with
c0 = 0 by adding a constant to the power of the tted
value. In the theophylline example, the varConstPower variance function is
cij2 . We incorporate it in the nlme t using
expressed as g(
cij , ) = 1 +
+
weights = varConstPower(power = 0.1) )
> fm4Theo.nlme
Model: conc ~ SSfol(Dose, Time, lKe, lKa, lCl)
Data: Theoph
lKe
lKa
lCl
-2.4538 0.43348 -3.2275
Random effects:
Formula: list(lKa ~ 1, lCl ~ 1)
Level: Subject
Structure: Diagonal
lKa
lCl Residual
StdDev: 0.6387 0.16979
0.3155
Variance function:
Structure: Constant plus power of variance covariate
const
power
0.71966 0.31408
An initial value for the power parameter (2 ) is specied in the call to the
varConstPower constructor to avoid convergence problems associated with
the default value power=0, for this example.
393
3
2
1
0
-1
-2
-3
0
10
Fitted values (mg/l)
fm4Theo.nlme.
The anova method is used to assess the statistical signicance of the

variance function.
> anova( fm3Theo.nlme, fm4Theo.nlme )
Model df
AIC
BIC logLik
fm3Theo.nlme
1 6 366.04 383.34 -177.02
fm4Theo.nlme
2 8 351.35 374.41 -167.68 1 vs 2 18.694
1e-04
The small p-value for the likelihood ratio test indicates that the incorporation of the variance function in the model produced a signicant increase in
the log-likelihood. Both the AIC and the BIC also favor the fm4Theo.nlme
t. The plot of the standardized residuals versus the tted values, shown in
Figure 8.24, conrms the adequacy of the varConstPower variance function.
> plot( fm4Theo.nlme )
# Figure 8.24
Clinical Study of Quinidine

Figure 8.25 presents the scatter plot of the standardized residuals versus
the tted values, corresponding to the nal model for the quinidine data
in 8.2.2, represented by the tted object fm4Quin.nlme.
> ## xlim used to hide an unusually high fitted value and enhance
> ## visualization of the heteroscedastic pattern
> plot( fm4Quin.nlme, xlim = c(0, 6.2) )
# Figure 8.25
As reported in previous analyses of the quinidine data (Davidian and

Giltinan, 1995, 9.3), and indicated in Figure 8.25, the variance of the
within-group errors appears to increase with the quinidine concentration.
The tted values do not get suciently close to zero to cause problems in
394
3
2
1
0
-1
-2
-3
0
Fitted values (mg/L)
fm4Quin.nlme.
the calculation of weights for the power variance function and we choose
the varPower class to model the within-group heteroscedasticity.
> fm5Quin.nlme <- update( fm4Quin.nlme, weights = varPower() )
> summary( fm5Quin.nlme )
. . .
Random effects:
Level: Subject
Structure: Diagonal
StdDev: 0.32475
0.25689 0.25548
Variance function:
power
0.96616
Fixed effects: list(lCl ~ glyco + Creatinine + Weight, lKa + lV ~ 1)
0.15262 220 17.741 <.0001
lCl.glyco -0.4110
0.04487 220 -9.161 <.0001
0.03494 220
3.696 0.0003
lCl.Weight 0.0033
0.00179 220
1.828 0.0689
lKa -0.4269
0.25518 220 -1.673 0.0958
lV 5.3700
0.08398 220 63.941 <.0001
. . .
395
3
2
1
0
-1
-2
-3
0
Fitted values (mg/L)
fm5Quin.nlme.
> anova( fm4Quin.nlme, fm5Quin.nlme )

Model df
AIC
BIC logLik
fm4Quin.nlme
1 9 870.94 905.94 -426.47
fm5Quin.nlme
2 10 812.13 851.01 -396.06 1 vs 2 60.813 <.0001
The incorporation of the power variance function in the NLME model for
the quinidine data produced a signicant increase in the log-likelihood, as
evidenced by the small p-value for the likelihood ratio statistics. The plot
of the standardized residuals versus tted values, displayed in Figure 8.26,
gives further evidence of the adequacy of the variance function model.
> plot( fm5Quin.nlme, xlim = c(0, 6.2) )
# Figure 8.26
8.3.2 Correlation Structures in nlme

Just as in lme and gls, correlation structures are specied in nlme through
the correlation argument. All of the corStruct classes described in 5.3.3,
and listed in Table 5.3, can be used with nlme. As in the linear case, we
investigate the need for within-group correlation structures in the NLME
model by looking at plots of the empirical autocorrelation function (ACF)
and the sample semivariogram, and assess the adequacy of a particular
corStruct class by examining plots of the normalized residuals. We revisit
the ovary example of 5.3.4 to illustrate the use of correlation structures
with nlme.
Counts of Ovarian Follicles
The ovary example was analyzed in 5.3.4 using an LME model, by assuming that the number of ovarian follicles was a periodic function of time with
396
known frequency equal to 1. A more general model formulation assumes

that the frequency is an unknown parameter to be estimated from the data,
with a possible random eect associated with it. Because the frequency enters the model nonlinearly, this becomes an NLME model, represented as
yij = 0i + 1i sin (22i tij ) + 3i cos (22i tij ) + ij ,
(8.18)
where yij represents the number of follicles observed for mare i at time tij ,
0i , 1i , and 3i represent the intercept and the terms dening the amplitude and the phase of the cosine wave for mare i, 2i is the frequency of
cosine wave for mare i, and ij is the within-group error. We initially assume
that the within-group errors are independently distributed as N (0, 2 ).
The nal LME model used to t the Ovary data in 5.3.4 used independent random eects for the 0i and 1i coecients in model (8.18). We
use this as a starting point for the random-eects model, incorporating
an extra independent random eect for the frequency 2i . The xed- and
random-eects models corresponding to (8.18) are then expressed as
0i = 0 + b0i ,
1i = 1 + b1i , 2i = 2 + b2i ,
b0i
bi = b1i N (0, ) ,
b2i
3i = 3 ,
(8.19)
where is diagonal. The random eects, bi , are assumed to be independent

for dierent i and to be independent of the within-group errors.
We t model (8.18), with xed- and random-eects structures given
by (8.19), using
> fm1Ovar.nlme <- nlme(follicles ~ A + B * sin(2 * pi * w * Time) +
+
C * cos(2 * pi * w *Time), data = Ovary,
+
fixed = A + B + C + w ~ 1, random = pdDiag(A + B + w ~ 1),
+
start = c(fixef(fm5Ovar.lme), 1))
> fm1Ovar.nlme
Model: follicles ~ A + B * sin(2 * pi * w * Time) +
C * cos(2 * pi * w * Time)
Data: Ovary
Fixed: A + B + C + w ~ 1
A
B
C
w
12.184 -3.376 -1.6812 0.93605
Random effects:
Formula: list(A ~ 1, B ~ 1, w ~ 1)
Level: Mare
Structure: Diagonal
A
B
w Residual
StdDev: 2.9051 2.0061 0.073598
2.9387
. . .
397
The estimated xed eects for fm5Ovar.lme are used as initial values for
0 , 1 , and 3 .
As mentioned in 5.3.4, the observations in the Ovary data were collected
at equally spaced calendar times, and then converted to an ovulation cycle
scale. Therefore, the empirical ACF can be used to investigate the correlation at dierent lags. The ACF method can also be used with nlme objects.
> ACF(
lag
1
0
2
1
3
2
4
3
5
4
6
5
7
6
8
7
9
8
10
9
11 10
12 11
13 12
14 13
15 14
fm1Ovar.nlme )
ACF
1.0000000
0.3110027
0.0887701
-0.0668554
-0.0314934
-0.0810381
-0.0010647
0.0216463
0.0137578
0.0097497
-0.0377027
-0.0741284
-0.1504872
-0.1616297
-0.2395797
Because they are based on fewer residual pairs, empirical autocorrelations

at larger lags are less reliable. We can control the number of lags calculated
in ACF using the maxLag argument. We use it in the plot of empirical ACF,
displayed in Figure 8.27 and obtained with
> plot( ACF(fm1Ovar.nlme,
+
alpha = 0.05 )
maxLag = 10),
# Figure 8.27
Figure 8.27 shows that only the lag-1 autocorrelation is signicant at the
5% level, but the lag-2 autocorrelation, which is approximately equal to the
square of the lag-1 autocorrelation, is nearly signicant. This suggests two
dierent candidate correlation structures for modeling the within-group
error covariance structure: AR(1) and MA(2). The two correlation models
are not nested, but can be compared using using the information criteria
provided by the anova method, AIC and BIC. The empirical lag-1 autocorrelation is used as a starting value for the corAR1 coecient.
> fm2Ovar.nlme <- update( fm1Ovar.nlme, corr = corAR1(0.311) )
> fm3Ovar.nlme <- update( fm1Ovar.nlme, corr = corARMA(p=0, q=2) )
> anova( fm2Ovar.nlme, fm3Ovar.nlme, test = F )
Model df
AIC
BIC logLik
fm2Ovar.nlme
1 9 1568.3 1601.9 -775.15
fm3Ovar.nlme
2 10 1572.1 1609.4 -776.07
398
Autocorrelation
1.0
0.8
0.6
0.4
0.2
0.0
0
10
Lag
FIGURE 8.27. Empirical autocorrelation function corresponding to the standardized residuals of the fm1Ovar.nlme tted object.
The AR(1) model uses one fewer parameter than the MA(2) model to give
a larger log-likelihood and hence is the preferred model by both AIC and
BIC.
The approximate 95% condence intervals for the variance components
in fm2Ovar.nlme, obtained with
> intervals( fm2Ovar.nlme )
. . .
Random Effects:
Level: Mare
lower
est.
upper
sd(A) 1.5465e+00 3.3083316 7.0772e+00
sd(B) 3.4902e-01 1.4257894 5.8245e+00
sd(w) 2.4457e-89 0.0020967 1.7974e+83
. . .
indicate that there is no precision in the estimate of the standard deviation

for the frequency 2i and little precision in the estimate of the standard
deviation for 1i . The incorporation of the within-group autocorrelation
structure into the NLME model seems to have reduced the need for random eects in the model. This is not uncommon: the random-eects model
and the within-group correlation model compete with each other, in the
sense that fewer random eects may be needed when within-group correlation structures are present, and viceversa. We test if the two variance
components can be dropped from the model using anova.
> fm4Ovar.nlme <- update( fm2Ovar.nlme, random = A ~ 1)
> anova( fm2Ovar.nlme, fm4Ovar.nlme )
Model df
AIC
BIC logLik
fm2Ovar.nlme
1 9 1568.3 1601.9 -775.15
fm4Ovar.nlme
2 7 1565.2 1591.4 -775.62 1 vs 2 0.94001
0.625
The high p-value for the likelihood ratio test suggests that the two models
give essentially equivalent ts so the simpler model is preferred.
399
Autocorrelation
1.0
0.8
0.6
0.4
0.2
0.0
0
10
Lag
FIGURE 8.28. Empirical autocorrelation function corresponding to the normalized residuals of the fm1Ovar.nlme tted object.
An alternative, intermediate model between the AR(1) and MA(2)

correlation structures is the ARMA(1, 1) model. This structure has an exponentially decaying ACF for lags 2, but allows greater exibility in
the lag-1 autocorrelation. Because the AR(1) model is nested within the
ARMA(1, 1) model, they can be compared via a likelihood ratio test.
> fm5Ovar.nlme <- update( fm4Ovar.nlme, corr = corARMA(p=1, q=1))
> anova( fm4Ovar.nlme, fm5Ovar.nlme )
Model df
AIC
BIC logLik
fm4Ovar.nlme
1 7 1565.2 1591.4 -775.62
fm5Ovar.nlme
2 8 1562.1 1592.0 -773.07 1 vs 2 5.1134 0.0237
The ARMA(1, 1) gives a signicantly better representation of the withingroup correlation, as indicated by the small p-value for the likelihood ratio
test.
The plot of the empirical ACF of the normalized residuals, displayed
in Figure 8.28, attests the the adequacy of the ARMA(1, 1) model for the
Ovary data. No signicant autocorrelations are detected, indicating that the
normalized residuals behave like uncorrelated noise, as expected under the
appropriate correlation model.
> plot( ACF(fm5Ovar.nlme,
+
alpha = 0.05 )
maxLag = 10, resType = "n"),

# Figure 8.28
It is illustrative, at this point, to compare the nlme t represented by

fm5Ovar.nlme to the lme t corresponding to fm5Ovar.lme of 5.3.4. To have
comparable log-likelihoods, we need to rst obtain a maximum likelihood
version of fm5Ovar.lme.
> fm5Ovar.lmeML <- update( fm5Ovar.lme, method = "ML" )
> intervals( fm5Ovar.lmeML )
. . .
Random Effects:
400

Level: Mare
lower
est.
upper
sd((Intercept)) 0.988120 2.45659 6.1074
sd(sin(2 * pi * Time)) 0.071055 0.85199 10.2158
. . .
The wide condence interval for the standard deviation of the random
eect corresponding to sin(2*pi*Time) indicates that the t is not very
sensitive to the value of this coecient and perhaps it could be eliminated
from the model. We test this assumption using the likelihood ratio test.
> fm6Ovar.lmeML <- update( fm5Ovar.lmeML, random = ~1 )
> anova( fm5Ovar.lmeML, fm6Ovar.lmeML )
Model df
AIC
BIC logLik
Test L.Ratio
fm5Ovar.lmeML
1 8 1562.7 1592.6 -773.37
fm6Ovar.lmeML
2 7 1561.0 1587.1 -773.51 1 vs 2 0.28057
p-value
fm5Ovar.lmeML
fm6Ovar.lmeML 0.5963
The large p-value for the test indicates that the two models are essentially
equivalent so the simpler model with a single random intercept is preferred.
The LME model represented by fm6Ovar.lmeML is nested within the NLME
model represented by fm5Ovar.nlme, corresponding to the case of 2 = 1.
Hence, we can test the assumption that the frequency of the ovulation cycle
is equal to 1 using the likelihood ratio test.
> anova( fm6Ovar.lmeML, fm5Ovar.nlme )
Model df
AIC
BIC logLik
Test L.Ratio
fm6Ovar.lmeML
1 7 1561.0 1587.1 -773.51
fm5Ovar.nlme
2 8 1562.1 1592.0 -773.07 1 vs 2 0.87881
p-value
fm6Ovar.lmeML
fm5Ovar.nlme 0.3485
There is no signicant evidence that 2 = 1. This conclusion is also supported by the approximate condence interval for 2 , which contains 1.
> intervals( fm5Ovar.nlme, which = "fixed" )
Fixed effects:
lower
est.
A 10.37917 12.15566
B -3.91347 -2.87191
C -3.07594 -1.56879
w
0.81565
0.93111
upper
13.932151
-1.830353
-0.061645
1.046568
401
8.3.3 Fitting Extended Nonlinear Regression Models with

gnls
The general formulation of the extended nonlinear regression model, as well
as the estimation methods used to t it, have been described in 7.5.1 and
7.5.2. In this section, we present and illustrate the capabilities available
in the nlme library for tting and analyzing such models.
The gnls function ts the extended nonlinear regression model (7.34)
using maximum likelihood. It can be viewed either as a version of nlme
without the argument random, or as a version of nls with the arguments
weights and correlation. Several arguments are available in gls, but typical
calls are of the form
gnls(model, data, params, start, correlation)
# correl. errors
gnls(model, data, params, start, weights)
# heterosc. errors
gnls(model, data, params, start, correlation, weights) # both
The rst argument, model, is a two-sided nonlinear formula specifying the

model for the expected value of the response. It uses the same syntax as
the model argument to nlme. Correlation and weights are used as in lme,
gls, and nlme to dene, respectively, the correlation model and the variance
function model for the error term. Data species a data frame in which the
variables named in model, correlation, and weights can be evaluated. The
parameters in the model are specied via the params argument, which can
be either a two-sided linear formula, or a list of two-sided linear formulas.
The syntax for params is identical to that of the fixed argument to nlme.
Starting values for the model parameters are specied in start, which uses
the same syntax as the argument with the same name to nlme. Starting
values need not be given when the model function dened in model is a
self-starting model and the the right-hand side of the parameter formulas
in param do not include any covariates.
The tted object returned by gnls is of class gnls, for which several
methods are available to display, plot, update, and further explore the
estimation results. Table 8.5 lists the most important gnls methods. The
syntax of the gnls methods is identical to the syntax of the gls methods,
described in 5.4. In fact, with the exception of coef, formula, logLik,
predict, and update, all methods listed in Table 8.5 are common to both
classes.
The use of the gnls function and its associated methods is described and
illustrated through the re-analysis of the hemodialyzer example introduced
in 5.2.2.
The hemodialyzer ultraltration rates data were analyzed in 5.2.2 and 5.4
using an empirical polynomial model suggested by Littell et al. (1996). The
model originally proposed for these data by Vonesh and Carter (1992) is
402
TABLE 8.5. Main gnls methods.

ACF
anova
augPred
coef
fitted
intervals
logLik
plot
predict
print
qqnorm
resid
summary
update
Variogram
empirical autocorrelation function of residuals

likelihood ratio or Wald-type tests
estimated coecients for expected response model
tted values
predicted values
residuals
update the gnls t
semivariogram of residuals
an asymptotic regression model with an oset, identical to the one used for
the CO2 uptake data in 8.2.2. The model for the expected ultraltration
rate y at transmembrane pressure x is written as
E[y] = 1 {1 exp [ exp(2 ) (x 3 )]} .
(8.20)
The parameters in model (8.20) have a physiological interpretation: 1 is

the maximum ultraltration rate that can be attained, 2 is the logarithm
of the hydraulic permeability transport rate, and 3 is the transmembrane
pressure required to oset the oncotic pressure.
Vonesh and Carter (1992) suggest using dierent parameters in (8.20)
for each blood ow rate level. We use the self-starting function SSasympOff
and the nlsList function to investigate which parameters in the asymptotic
regression model (8.20) depend on the blood ow rate.
> fm1Dial.lis <+
nlsList( rate ~ SSasympOff(pressure, Asym, lrc, c0) | QB,
+
data = Dialyzer )
> fm1Dial.lis
. . .
Coefficients:
Asym
lrc
c0
200 44.988 0.76493 0.22425
300 62.217 0.25282 0.22484
The coecient estimates suggest that Asym (1 ) and lrc (log 2 ) depend on
the blood ow rate level, but c0 (3 ) does not. The plot of the individual

Asym
QB
200
lrc
300
| |
45
c0
| | |
|
|
50
55
60
65
0.2
0.4
403
0.6
|
0.8
|
|
19
|
0.21
|
0.23
0.25
FIGURE 8.29. Ninety-ve percent condence intervals on the asymptotic regression model parameters for each level of blood ow rate (QB) in the dialyzer
data.
condence intervals in Figure 8.29 conrms that only Asym and lrc vary
with blood ow level.
> plot( intervals(fm1Dial.lis) )
# Figure 8.29
The ultraltration rate yij at the jth transmembrane pressure xij for
the ith subject is represented by the nonlinear model
yij = (1 + 1 Qi ) {1 exp [ exp (2 + 2 Qi) (xij 3 )]} + ij ,
(8.21)
where Qi is a binary variable taking values 1 for 200 dl/min hemodialyzers and 1 for 300 dl/min hemodialyzers; 1 , 2 , and 3 are, respectively,
the asymptotic ultraltration rate, the log-transport rate, and the transmembrane pressure oset averaged over the levels of Q; i is the blood ow
eect associated with the coecient i ; and ij is the error term, initially
assumed to be independently distributed N (0, 2 ) random variables.
The nonlinear model (8.21) can be tted with nls, but it is easier to
express the dependency of the asymptote and the log-rate on the blood
ow rate using gnls. The average of the fm1Dial.lis coecients are used
as the initial estimates for 1 , 2 , and 3 , while the half dierences between
the rst two coecients are used as initial estimates for 1 and 2 .
> fm1Dial.gnls <- gnls( rate ~ SSasympOff(pressure, Asym, lrc, c0),
+
data = Dialyzer, params = list(Asym + lrc ~ QB, c0 ~ 1),
+
start = c(53.6, 8.6, 0.51, -0.26, 0.225) )
> fm1Dial.gnls
Generalized nonlinear least squares fit
Model: rate ~ SSasympOff(pressure, Asym, lrc, c0)
Data: Dialyzer
Coefficients:
Asym.(Intercept) Asym.QB lrc.(Intercept)
lrc.QB
c0
53.606
8.62
0.50874 -0.25684 0.22449
404
To t the same model in nls, we need rst to create a binary variable

representing Q in (8.21) and then include all coecients explicitly in the
model formula.
> Dialyzer$QBcontr <- 2 * (Dialyzer$QB == 300) - 1
> fm1Dial.nls <+
nls( rate ~ SSasympOff(pressure, Asym.Int + Asym.QB * QBcontr,
+
lrc.Int + lrc.QB * QBcontr, c0), data = Dialyzer,
+
start = c(Asym.Int = 53.6, Asym.QB = 8.6, lrc.Int = 0.51,
+
lrc.QB = -0.26, c0 = 0.225) )
> summary( fm1Dial.nls )
Formula: rate ~ SSasympOff(pressure, Asym.Int + Asym.QB * QBcontr,
lrc.Int + lrc.QB * QBcontr, c0)
Parameters:
Asym.Int
Asym.QB
lrc.Int
lrc.QB
c0
Value Std. Error

53.60660
0.705409
8.61999
0.679240
0.50872
0.055233
-0.25683
0.045021
0.22448
0.010623
t value
75.9937
12.6906
9.2105
-5.7047
21.1318

> logLik( fm1Dial.nls )
[1] -382.65
As expected, the results are nearly identical.

The plot method is the primary tool for assessing the quality of a gnls
t. It uses the same syntax as the other plot methods in the nlme library.
For example, the plot of the residuals versus the transmembrane pressure,
shown in Figure 8.30 and obtained with
> plot(fm1Dial.gnls, resid(.) ~ pressure, abline = 0) # Figure 8.30
indicates that the error variability increases with the transmembrane pressure. This heteroscedastic pattern is also observed in the linear model ts
of the hemodialyzer data, presented in 5.2.2 and 5.4.
As in the previous analyses of the hemodialyzer data presented in 5.2.2
and 5.4, the power variance function, represented in nlme by the varPower
class, is used to model the heteroscedasticity in the ultraltration rates.
> fm2Dial.gnls <- update( fm1Dial.gnls,
+
> anova( fm1Dial.gnls, fm2Dial.gnls)
Model df
AIC
BIC logLik
fm1Dial.gnls
1 6 777.29 794.94 -382.65
fm2Dial.gnls
2 7 748.47 769.07 -367.24 1 vs 2 30.815 <.0001
As expected, the likelihood ratio test strongly rejects the assumption of

homoscedasticity. The plot of the standard residuals for fm2Dial.gnls
405
10
Residuals (ml/hr)
-5
-10
0.5
1.0
1.5
2.0
2.5
3.0
FIGURE 8.30. Plot of residuals versus transmembrane pressure for the homoscedastic tted object fm1Dial.gls.
versus pressure, shown in Figure 8.31, indicates that the power variance
function successfully models the heteroscedasticity in the data.
The hemodialyzer ultraltration rates measurements made sequentially
on the same subject are correlated. Random eects can be used in an
NLME model to account for the within-group correlation, but we choose
here to model the within-subject dependence directly by incorporating a
correlation structure for the error term in the gnls model. Because the
measurements are equally spaced in time, the empirical autocorrelation
function can be used to investigate the within-subject correlation. The ACF
method is used to obtain the empirical ACF, with the time covariate and
the grouping factor specied via the form argument.
> ACF( fm2Dial.gnls, form = ~ 1 | Subject )
lag
ACF
1
0 1.00000
2
1 0.71567
3
2 0.50454
4
3 0.29481
5
4 0.20975
6
5 0.13857
7
6 -0.00202
The empirical ACF values conrm the within-group correlation and indicates that the correlation decreases with lag. As usual, it is more informative to look at a plot of the empirical ACF, displayed in Figure 8.32 and
obtained with
406
-1
-2
0.5
1.0
1.5
2.0
2.5
3.0

the heteroscedastic tted object fm2Dial.gnls.
> plot( ACF( fm2Dial.gnls, form = ~ 1 | Subject),

+
alpha = 0.05 )
# Figure 8.32
The autocorrelation pattern in Figure 8.32 suggests that an AR(1) model,

represented in nlme by the corAR1 class, may be appropriate to describe
the within-group correlation.
> fm3Dial.gnls <+ update(fm2Dial.gnls, corr = corAR1(0.716, form = ~ 1 | Subject))
> fm3Dial.gnls
. . .
Coefficients:
Asym.(Intercept) Asym.QB lrc.(Intercept)
lrc.QB
c0
55.111 8.1999
0.37193 -0.16974 0.21478
Phi
0.7444
Variance function:
Formula: ~ pressure
power
0.5723
Autocorrelation
407
1.0
0.8
0.6
0.4
0.2
0.0
-0.2
-0.4
0
Lag
FIGURE 8.32. Empirical autocorrelation function corresponding to the standardized residuals of the fm2Dial.gnls tted object.
The lag-1 empirical autocorrelation is used as initial value for the corAR1.
The variability in the estimates is assessed with the intervals method.
> intervals( fm3Dial.gnls )
. . .
lower
est.
upper
Phi 0.55913 0.7444 0.85886
. . .
The condence interval on the autocorrelation parameter is bounded away

from zero, suggesting that the AR(1) model provides a signicantly better
t. We conrm this with the likelihood ratio test.
> anova( fm2Dial.gnls, fm3Dial.gnls )
Model df
AIC
BIC logLik
fm2Dial.gnls
1 7 748.47 769.07 -367.24
fm3Dial.gnls
2 8 661.04 684.58 -322.52 1 vs 2 89.433 <.0001
The plot of the empirical ACF for the normalized residuals corresponding
to fm3Dial.gnls, displayed in Figure 8.33, does not show any signicant
correlations, indicating that the AR(1) adequately represents the withinsubject dependence in the gnls model for the hemodialyzer data.
The plot of the standardized residuals versus the transmembrane pressure
in Figure 8.34 suggests a certain lack-of-t for the asymptotic regression
model (8.21): the residuals for the highest two transmembrane pressures
are predominantly negative. This is consistent with the plot of the hemodialyzer data, shown in Figure 5.1, and also with Figure 3 in Vonesh and
Carter (1992), which indicate that, for many subjects, the ultraltration
rates decrease for the highest two transmembrane pressures. The asymptotic regression model is monotonically increasing in the transmembrane
pressure and cannot properly accommodate the nonmonotonic behavior of
the ultraltration rates.
Autocorrelation
408
1.0
0.8
0.6
0.4
0.2
0.0
-0.2
-0.4
0
Lag
FIGURE 8.33. Empirical autocorrelation function for the normalized residuals

corresponding to the fm3Dial.gnls tted object.
-1
-2
0.5
1.0
1.5
2.0
2.5
3.0

the fm3Dial.gnls tted object.
8.4 Chapter Summary
409
The gnls model corresponding to fm3Dial.gnls may be compared to

the best models obtained for the Dialyzer data in 5.2.2 and 5.4, corresponding, respectively, to the objects fm2Dial.lme and fm3Dial.gls. To
have comparable ts, we need to rst obtain ML versions of fm2Dial.lme
and fm3Dial.gls.
> fm2Dial.lmeML <- update( fm2Dial.lme, method = "ML")
> fm3Dial.glsML <- update( fm3Dial.gls, method = "ML")
As the models are not nested, only the information criterion statistics can
be compared.
> anova(
fm2Dial.lmeML, fm3Dial.glsML, fm3Dial.gnls, test = F )

Model df
AIC
BIC logLik
fm2Dial.lmeML
1 18 651.75 704.70 -307.87
fm3Dial.glsML
2 13 647.56 685.80 -310.78
fm3Dial.gnls
3 8 661.04 684.58 -322.52
The more conservative BIC favors the fm3Dial.gnls model because of the
fewer number of parameters it uses; the more liberal AIC favors
fm3Dial.glsML because of its larger log-likelihood value. In practice, the
choice of the best model should take into account other factors besides
the information criteria, such as the interpretability of the parameters.
8.4 Chapter Summary

This chapter describes the nonlinear modeling capabilities available in the
nlme library. A brief review of the nonlinear least-squares function nls in
S is presented and self-starting models for automatically producing starting values for the coecients in a nonlinear model are introduced and
illustrated. The nlsList function for tting separate nonlinear regression
models to data partitioned according to the levels of a grouping factor is
described and its use for model building of nonlinear mixed-eects models
illustrated.
Nonlinear mixed-eects models are tted with the nlme function. Data
from several real-life applications are used to illustrate the various capabilities available in nlme for tting and analyzing single and multilevel NLME
models. Variance functions and correlation structures to model the withingroup variancecovariance structure are used with nlme in the exact same
way as with lme, the linear mixed-eects modeling function. Several examples are used to illustrate the use of varFunc and corStruct classes with
nlme.
A new modeling function, gnls, for tting the extended nonlinear model
with heteroscedastic, correlated errors is introduced. The gnls function can
be regarded as an extended version of nls which allows the use of varFunc
and corStruct objects to model the error variancecovariance structure, or
410
as a simplied version of nlme, without random eects. The hemodialyzer

example is used to illustrate the use of gnls and its associated methods.
Exercises
1. Plots of the DNase data in 3.3.2 and in Appendix A.7 suggest a
sigmoidal relationship between the density and log(conc). Because
there are no data points at suciently high DNase concentrations to
dene the upper part of the curve, the sigmoidal relationship is only
suggested and is not denite.
A common model for this type of assay data is the four-parameter
logistic model available as SSfpl (Appendix C.6).
(a) Create separate ts of the SSfpl model to each run using nlsList.
Note that the display formula, density ~conc | Run, denes the
primary covariate as conc but the model should be t as a function of log(conc). You will either need to give explicit arguments
to the SSfpl function or dene a new groupedData object with
a formula based on log(conc).
(b) Examine the plot of the residuals versus the tted values from
this nlsList t. Does the plot indicate nonconstant variance?
(c) Fit an NLME model to these data using random eects for each
model parameter and a general positive-denite . Perform the
usual checks on the model using diagnostic plots, summaries,
and intervals. Can the condence intervals on the variance
covariance parameters be evaluated?
(d) Davidian and Giltinan (1995, 5.2.4) conclude from a similar
analysis that the variance of the ij increases with increasing
optical density. They estimate a variance function that corresponds to the varPower variance function. Update the previous
t by adding weights = varPower(). Does this updated model
converge? If not, change the denition of the random eects so
that is diagonal.
(e) Compare the model t with the varPower variance function to
one t without it. Note that if you have t the model with the
variance function by imposing a diagonal , you should ret
the model without the variance function but with a diagonal
before comparing. Does the addition of the varPower variance
function make a signicant contribution to the model?
(f) Examine the condence intervals on your best-tting model to
this point. Are there parameters that can be modeled as purely
Exercises
411
xed eects? (In our analysis the standard deviation of the random eect for the xmid parameter turned out to be negligible.)
If you determine that some random eects are no longer needed,
remove them and ret the model.
(g) If you have modied the model to reduce the number of random
eects, ret with a general positive-denite rather than a
diagonal . Does this t converge? If so, compare the ts with
diagonal and general positive-denite .
(h) Write a report on the analysis, including data plots and diagnostic plots where appropriate.
2. As shown in Figure 3.4 (p. 107), the relationship between deltaBP
and log(dose) in the phenybiguanide data PBG is roughly sigmoidal.
There is a strong indication that the eect of the Treatment is to shift
the curve to the right.
(a) Fit separate four-parameter logistic models (SSfpl, Appendix
C.6) to the data from each Treatment within each Rabbit using
nlsList. Recall from 3.2.1 that the primary covariate in the
display formula for PBG is dose but we want the model to be t as
a function of log(dose). You will either need to specify the model
explicitly or to re-dene the display formula for the data. Also
note that the grouping formula should be ~Rabbit/Treatment,
but the grouping in the display formula is ~Rabbit.
(b) Plot the condence intervals on the coecients. Which parameters appear to be constant across all Rabbit/Treatment combinations? Does there appear to be a systematic shift in the xmid
parameter according to Treatment?
(c) Fit a two-level NLME model with a xed eect for Treatment on
the xmid parameter and with random eects for Rabbit on the
B parameter and for Treatment within Rabbit on the B and xmid
parameters. Begin with a diagonal 2 matrix. If that model
t converges, update to a general positive-denite 2 matrix
and compare the two tted models with anova. Which model is
preferred?
(d) Summarize your preferred model. Is the xed eect for Treatment
on xmid signicant? Also check using the output from intervals.
(e) Plot the augmented predictions from your preferred model. Remember that if the display formula for the data has dose as the
primary covariate then you will need to use scales = list(x =
list(log = 2)) to get the symmetric shape of the logistic curve.
Adjust the layout argument for the plot so the panels are aligned
by Rabbit. Do these plots indicate deciencies in the model?
412
(f) Examine residual plots for other possible deciencies in the model.
Also check plots of the random eects versus covariates to see if
important xed eects have been omitted.
(g) Is it necessary to use a four-parameter logistic curve? Experiment with tting the three-parameter logistic model (SSlogis,
Appendix C.7) to see if a comparable t can be obtained. Check
residual plots from your tted models and comment.
3. The Glucose2 data described in Appendix A.10 consist of blood glucose levels measured 14 times over a period of 5 hours on 7 volunteers
who took alcohol at time 0. The same experiment was repeated on a
second occasion with the same subjects but with a dietary additive
used for all subjects. These data are analyzed in Hand and Crowder (1996, Example 8.4, pp. 118120), where the following empirical
model relating the expected glucose level to Time is proposed.
glucose = 1 + 2 Time3 exp (3 Time)
Note that there are two levels of grouping in the data: Subject and
Date within Subject.
(a) Plot the data at the Subject display level (use plot(Glucose2,
display = 1). Do there appear to be systematic dierences between the two dates on which the experiments were conducted
(which could be associated with the dietary supplement)?
(b) There is no self-starting function representing the model for the
glucose level included in the nlme library. Use nlsList with starting values start = c(phi1=5, phi2=-1, phi3=1) (derived from
Hand and Crowder (1996)) to t separate models for each Subject
and for each Date within Subject. Plot the individual condence
intervals for each of the two nlsList ts. Verify that phi1 and
phi2 seem to vary signicantly for both levels of grouping, but
phi3 does not. (There is an unusual estimate of phi3 for Subject
6, Date 1 but all other condence intervals overlap.)
(c) Fit a two-level NLME model with random eects for phi1 and
phi2, using as starting values for the xed eects the estimates
from either of the nlsList ts (start = fixef(object), with
object replaced with the name of the nlsList object). Examine the condence intervals on the variancecovariance components; what can you say about the precision of the estimated
correlation coecients?
(d) Ret the NLME model using diagonal q matrices for both
grouping levels and compare the new t to the previous one
using anova. Investigate if there are random eects that can be
dropped from the model using intervals. If so, ret the model
Exercises
413
with fewer random eects and compare it to the previous t

using anova. Plot the residuals versus Time and comment on the
apparent adequacy of the empirical model.
(e) Plot the semivariogram of the standardized residuals corresponding to the nal NLME t obtained in the last item. Use Time as
the covariate to dene the distances in the semivariogram and
consider only distances 22 (that is, use plot(Variogram(object,
form = ~Time, maxDist = 22))). Notice the increase in the semivariogram for smaller distances, which suggests that there the
within-group errors may be correlated.
(f) Hand and Crowder (1996) use a continuous AR1 model for
the within-group errors, with Time as the covariate. Update the
NLME t adding a continuous AR1 correlation structure on Time
(corr = corCAR1(form = Time). Compare the ts using anova.
Examine the plot of the semivariogram of the normalized residuals for the NLME with corCAR1 correlation structure and comment on the adequacy of the within-group correlation model.
(g) The NLME model used by Hand and Crowder (1996) includes
random eects for phi1 and phi2 only at the Subject level and
uses a corCAR1 correlation structure on Time for the Date within
Subject errors, with errors from dierent Dates within the same
Subject assumed to be independent. Fit such model using
random = B1 + B2 ~ 1 | Subject,
corr = corCAR1(form = ~Time | Subject/Date)
and compare it to the multilevel NLME model obtained in the

previous item. Which one do you think gives a better t?
(h) The main objective of the experiment was to determine if there
were signicant dierences between the blood glucose level proles over time associated with the use of the dietary additive
(which is totally confounded with Dose in this case). Investigate
the signicance of the dietary additive by retting the NLME
model (with corCAR1 correlation) incorporating a Date eect
for each xed eect (use fixed = phi1 + phi2 + phi3 Date).
You will need to give new starting values for the xed eects
(use the previous xed eects estimates for the Intercept terms
and 0 for the Date terms). Use summary to assess the signicance
of the dietary additive eect; does it seem to make any dierence on the blood glucose levels? Are your conclusions consistent
with the plot of the data?
4. An NLME analysis of the Theoph data is presented in 8.2.1. The
nal nlme t obtained in that section, fm3Theo.nlme, includes random
eects for the lKa and lCl coecients and a diagonal .
414
(a) Use the gnls function described in 8.3.3 to t the SSfol model
to the theophylline concentrations with no random eects. Compare this t to fm3Theo.nlme using anova. Obtain the boxplots of
the residuals by Subject (plot(object, Subject~resid(.))) and
comment on the observed pattern.
(b) Print and plot the ACF of the standardized residuals for the gnls
t (use form = ~1 | Subject to specify the grouping structure).
The decrease in the ACF with lag suggests that an AR1 model
may be adequate.
(c) Update the gnls t incorporating an AR1 correlation structure,
using the lag-1 autocorrelation from the ACF output as an initial
estimate for the correlation parameter (corr = corAR1(0.725,
form = ~1 | Subject)). Compare this t to the previous gnls
t using anova. Is there signicant evidence of autocorrelation?
Examine the plot of the ACF of the normalized residuals. Does
the AR1 model seem adequate?
(d) Compare the gnls t with AR1 correlation structure to the
fm3Theo.nlme t of 8.2.1 using anova with the argument test
set to FALSE (why?). Which model seems better?
References
Abramowitz, M. and Stegun, I. A. (1964). Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, Dover, New
York.
Bates, D. M. and Chambers, J. M. (1992). Nonlinear models, in Chambers and Hastie (1992), Chapter 10, pp. 421454.
Bates, D. M. and Pinheiro, J. C. (1998). Computational methods for multilevel models, Technical Memorandum BL0112140-980226-01TM, Bell
Labs, Lucent Technologies, Murray Hill, NJ.
Bates, D. M. and Watts, D. G. (1980). Relative curvature measures of
nonlinearity, Journal of the Royal Statistical Society, Ser. B 42: 125.
Bates, D. M. and Watts, D. G. (1988). Nonlinear Regression Analysis and
Its Applications, Wiley, New York.
Beal, S. and Sheiner, L. (1980). The NONMEM system, American Statistician 34: 118119.
Becker, R. A., Cleveland, W. S. and Shyu, M.-J. (1996). The visual design
and control of trellis graphics displays, Journal of Computational and
Graphical Statistics 5(2): 123156.
Bennett, J. E. and Wakeeld, J. C. (1993). Markov chain Monte Carlo for
nonlinear hierarchical models, Technical Report TR-93-11, Statistics
Section, Imperial College, London.
416
References
Boeckmann, A. J., Sheiner, L. B. and Beal, S. L. (1994). NONMEM Users

Guide: Part V, NONMEM Project Group, University of California,
San Francisco.
Box, G. E. P., Hunter, W. G. and Hunter, J. S. (1978). Statistics for
Experimenters, Wiley, New York.
Box, G. E. P., Jenkins, G. M. and Reinsel, G. C. (1994). Time Series
Analysis: Forecasting and Control, 3rd ed., Holden-Day, San Francisco.
Brillinger, D. (1987). Comment on a paper by C. R. Rao, Statistical Science
2: 448450.
Bryk, A. and Raudenbush, S. (1992). Hierarchical Linear Models for Social
and Behavioral Research, Sage, Newbury Park, CA.
Carroll, R. J. and Ruppert, D. (1988). Transformation and Weighting in
Regression, Chapman & Hall, New York.
Chambers, J. M. (1977). Computational Methods for Data Analysis, Wiley,
New York.
Chambers, J. M. and Hastie, T. J. (eds) (1992). Statistical Models in S,
Chapman & Hall, New York.
Cleveland, W. S. (1994). Visualizing Data, Hobart Press, Summit, NJ.
Cleveland, W. S., Grosse, E. and Shyu, W. M. (1992). Local regression
models, in Chambers and Hastie (1992), Chapter 8, pp. 309376.
Cochran, W. G. and Cox, G. M. (1957). Experimental Designs, 2nd ed.,
Wiley, New York.
Cox, D. R. and Hinkley, D. V. (1974). Theoretical Statistics, Chapman &
Hall, London.
Cressie, N. A. C. (1993). Statistics for Spatial Data, Wiley, New York.
Cressie, N. A. C. and Hawkins, D. M. (1980). Robust estimation of the
variogram, Journal of the International Association of Mathematical
Geology 12: 115125.
Crowder, M. and Hand, D. (1990). Analysis of Repeated Measures, Chapman & Hall, London.
Davidian, M. and Gallant, A. R. (1992). Smooth nonparametric maximum
likelihood estimation for population pharmacokinetics, with application to quinidine, Journal of Pharmacokinetics and Biopharmaceutics
20: 529556.
References
417
Davidian, M. and Giltinan, D. M. (1995). Nonlinear Models for Repeated

Measurement Data, Chapman & Hall, London.
Davis, P. J. and Rabinowitz, P. (1984). Methods of Numerical Integration,
2nd ed., Academic Press, New York.
Dempster, A. P., Laird, N. M. and Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm, Journal of the
Royal Statistical Society, Ser. B 39: 122.
Devore, J. L. (2000). Probability and Statistics for Engineering and the
Sciences, 5th ed., Wadsworth, Belmont, CA.
Diggle, P. J., Liang, K.-Y. and Zeger, S. L. (1994). Analysis of Longitudinal
Data, Oxford University Press, New York.
Dongarra, J. J., Bunch, J. R., Moler, C. B. and Stewart, G. W. (1979).
Linpack Users Guide, SIAM, Philadelphia.
Draper, N. R. and Smith, H. (1998). Applied Regression Analysis, 3rd ed.,
Wiley, New York.
Gallant, A. R. and Nychka, D. W. (1987). Seminonparametric maximum
likelihood estimation, Econometrica 55: 363390.
Geman, S. and Geman, D. (1984). Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images, IEEE Transactions on
Pattern Analysis and Machine Intelligence 6: 721741.
Geweke, J. (1989). Bayesian inference in econometric models using Monte
Carlo integration, Econometrica 57: 13171339.
Gibaldi, M. and Perrier, D. (1982). Pharmacokinetics, Marcel Dekker, New
York.
Goldstein, H. (1987). Multilevel Models in Education and Social Research,
Oxford University Press, Oxford.
Goldstein, H. (1995). Multilevel Statistical Models, Halstead Press, New
York.
Golub, G. H. (1973). Some modied matrix eigenvalue problems, SIAM
Review 15: 318334.
Golub, G. H. and Welsch, J. H. (1969). Calculation of Gaussian quadrature
rules, Mathematical Computing 23: 221230.
Grasela and Donn (1985). Neonatal population pharmacokinetics of phenobarbital derived from routine clinical data, Developmental Pharmacology and Therapeutics 8: 3740383.
418
References
Hand, D. and Crowder, M. (1996). Practical Longitudinal Data Analysis,

Texts in Statistical Science, Chapman & Hall, London.
Harville, D. A. (1977). Maximum likelihood approaches to variance component estimation and to related problems, Journal of the American
Statistical Association 72: 320340.
Hastings, W. K. (1970). Monte Carlo sampling methods using Markov
chains and their applications, Biometrika 57: 97109.
Jones, R. H. (1993). Longitudinal Data with Serial Correlation: A Statespace Approach, Chapman & Hall, London.
Joyner and Boore (1981). Peak horizontal acceleration and velocity from
strong-motion records including records from the 1979 Imperial Valley, California, earthquake, Bulletin of the Seismological Society of
America 71: 20112038.
Kennedy, William J., J. and Gentle, J. E. (1980). Statistical Computing,
Marcel Dekker, New York.
Kung, F. H. (1986). Fitting logistic growth curve with predetermined carrying capacity, ASA Proceedings of the Statistical Computing Section
pp. 340343.
Kwan, K. C., Breault, G. O., Umbenhauer, E. R., McMahon, F. G. and
Duggan, D. E. (1976). Kinetics of indomethicin absorption, elimination, and enterohepatic circulation in man, Journal of Pharmacokinetics and Biopharmaceutics 4: 255280.
Laird, N. M. and Ware, J. H. (1982). Random-eects models for longitudinal data, Biometrics 38: 963974.
Lehmann, E. L. (1986). Testing Statistical Hypotheses, Wiley, New York.
Leonard, T., Hsu, J. S. J. and Tsui, K. W. (1989). Bayesian marginal
inference, Journal of the American Statistical Association 84: 1051
1058.
Lindley, D. and Smith, A. (1972). Bayes estimates for the linear model,
Journal of the Royal Statistical Society, Ser. B 34: 141.
Lindstrom, M. J. and Bates, D. M. (1988). NewtonRaphson and EM
algorithms for linear mixed-eects models for repeated-measures data
(corr: 94v89 p1572), Journal of the American Statistical Association
83: 10141022.
Lindstrom, M. J. and Bates, D. M. (1990). Nonlinear mixed-eects models
for repeated measures data, Biometrics 46: 673687.
References
419
Littell, R. C., Milliken, G. A., Stroup, W. W. and Wolnger, R. D. (1996).

SAS System for Mixed Models, SAS Institute Inc., Cary, NC.
Longford, N. T. (1993). Random Coecient Models, Oxford University
Press, New York.
Ludbrook, J. (1994). Repeated measurements and multiple comparisons in
cardiovascular research, Cardiovascular Research 28: 303311.
Mallet, A. (1986). A maximum likelihood estimation method for random
coecient regression models, Biometrika 73(3): 645656.
Mallet, A., Mentre, F., Steimer, J.-L. and Lokiek, F. (1988). Nonparametric maximum likelihood estimation for population pharmacokinetics,
with applications to Cyclosporine, Journal of Pharmacokinetics and
Biopharmaceutics 16: 311327.
Matheron, G. (1962). Traite de Geostatistique Appliquee, Vol. I of Memoires du Bureau de Recherches Geologiques et Minieres, Editions Technip, Paris.
Milliken, G. A. and Johnson, D. E. (1992). Analysis of Messy Data. Volume
1: Designed Experiments, Chapman & Hall, London.
Patterson, H. D. and Thompson, R. (1971). Recovery of interblock information when block sizes are unequal, Biometrika 58: 545554.
Pierson, R. A. and Ginther, O. J. (1987). Follicular population dynamics during the estrus cycle of the mare, Animal Reproduction Science
14: 219231.
Pinheiro, J. C. (1994). Topics in Mixed-Eects Models, Ph.D. thesis, University of Wisconsin, Madison, WI.
Pinheiro, J. C. and Bates, D. M. (1995). Approximations to the loglikelihood function in the nonlinear mixed-eects model, Journal of
Computational and Graphical Statistics 4(1): 1235.
Pottho, R. F. and Roy, S. N. (1964). A generalized multivariate analysis of variance model useful especially for growth curve problems,
Biometrika 51: 313326.
Potvin, C., Lechowicz, M. J. and Tardif, S. (1990). The statistical analysis
of ecophysiological response curves obtained from experiments involving repeated measures, Ecology 71: 13891400.
Ramos, R. Q. and Pantula, S. G. (1995). Estimation of nonlinear random
coecient models, Statistics & Probability Letters 24: 4956.
420
References
Sakamoto, Y., Ishiguro, M. and Kitagawa, G. (1986). Akaike Information

Criterion Statistics, Reidel, Dordrecht, Holland.
Schwarz, G. (1978). Estimating the dimension of a model, Annals of Statistics 6: 461464.
Searle, S. R., Casella, G. and McCulloch, C. E. (1992). Variance Components, Wiley, New York.
Seber, G. A. F. and Wild, C. J. (1989). Nonlinear Regression, Wiley, New
York.
Self, S. G. and Liang, K. Y. (1987). Asymptotic properties of maximum
likelihood estimators and likelihood ratio tests under nonstandard conditions, Journal of the American Statistical Association 82: 605610.
Sheiner, L. B. and Beal, S. L. (1980). Evaluation of methods for estimating
population pharmacokinetic parameters. I. MichaelisMenten model:
Routine clinical pharmacokinetic data, Journal of Pharmacokinetics
and Biopharmaceutics 8(6): 553571.
Snedecor, G. W. and Cochran, W. G. (1980). Statistical Methods, 7th ed.,
Iowa State University Press, Ames, IA.
Soo, Y.-W. and Bates, D. M. (1992). Loosely coupled nonlinear least
squares, Computational Statistics and Data Analysis 14: 249259.
Stram, D. O. and Lee, J. W. (1994). Variance components testing in the
longitudinal mixed-eects models, Biometrics 50: 11711177.
Stroup, W. W. and Baenziger, P. S. (1994). Removing spatial variation
from wheat yield trials: a comparison of methods, Crop Science 34: 62
66.
Thisted, R. A. (1988). Elements of Statistical Computing, Chapman &
Hall, London.
Tierney, L. and Kadane, J. B. (1986). Accurate approximations for posterior moments and densities, Journal of the American Statistical Association 81(393): 8286.
Venables, W. N. and Ripley, B. D. (1999). Modern Applied Statistics with
S-PLUS, 3rd ed., Springer-Verlag, New York.
Verme, C. N., Ludden, T. M., Clementi, W. A. and Harris, S. C. (1992).
Pharmacokinetics of quinidine in male patients: A population analysis,
Clinical Pharmacokinetics 22: 468480.
Vonesh, E. F. and Carter, R. L. (1992). Mixed-eects nonlinear regression
for unbalanced repeated measures, Biometrics 48: 118.
References
421
Vonesh, E. F. and Chinchilli, V. M. (1997). Linear and Nonlinear Models

for the Analysis of Repeated Measures, Marcel Dekker, New York.
Wakeeld, J. (1996). The Bayesian analysis of population pharmacokinetic
models, Journal of the American Statistical Association 91: 6275.
Wilkinson, G. N. and Rogers, C. E. (1973). Symbolic description of factorial models for analysis of variance, Applied Statistics 22: 392399.
Wolnger, R. D. (1993). Laplaces approximation for nonlinear mixed models, Biometrika 80: 791795.
Wolnger, R. D. and Tobias, R. D. (1998). Joint estimation of location,
dispersion, and random eects in robust design, Technometrics 40: 62
71.
Yates, F. (1935). Complex experiments, Journal of the Royal Statistical
Society (Supplement) 2: 181247.
Appendix A
Data Used in Examples and Exercises
We have used several sets of data in our examples and exercises. In this
appendix we list all the data sets that are available as the NLMEDATA
library included with the nlme 3.1 distribution and we describe in greater
detail the data sets referenced in the text.
The title of each section in this appendix gives the name of the corresponding groupedData object from the nlme library, followed by a short
description of the data. The formula stored with the data and a short description of each of the columns is also given.
We have adopted certain conventions for the ordering and naming of
columns in these descriptions. The rst column provides the response, the
second column is the primary covariate, if present, and the next column
is the primary grouping factor. Other covariates and grouping factors, if
present, follow. Usually we use lowercase for the names of the response and
the primary covariate. One exception to this rule is the name Time for a
covariate. We try to avoid using the name time because it conicts with a
standard S function.
Table A.1 lists the groupedData objects in the NLMEDATA library that is
part of the nlme distribution.
424
Appendix A. Data Used in Examples and Exercises
TABLE A.1: Data sets included with the nlme library distribution. The
data sets whose names are shown in bold are described in this appendix.
Alfalfa
Assay
BodyWeight
CO2
Cephamadole
ChickWeight
Dialyzer
DNase
Earthquake
ergoStool
Fatigue
Gasoline
Glucose
Glucose2
Gun
IGF
Indometh
Loblolly
Machines
MathAchSchool
MathAchieve
Meat
Milk
Muscle
Nitrendipene
Oats
Orange
Orthodont
Ovary
Oxboys
Oxide
PBG
PBIB
Phenobarb
Pixel
Quinidine
Rail
RatPupWeight
Relaxin
Remifentanil
Soybean
Yields of three varieties of alfalfa

Laboratory data on a biochemical assay
Rat weight over time for dierent diets
Carbon dioxide uptake by grass plants
Pharmacokinetic data
Growth of chicks on dierent diets
Performance of high-ux hemodialyzers
Assay of DNase
Severity of earthquakes
Ergometrics experiment with stool types
Metal fatique data
Gasoline yields for dierent crude samples
Glucose levels over time
Glucose levels over time after alcohol ingestion
Naval gun ring data from Hicks (1993)
Assay data on Insulin-like Growth Factor
Pharmacokinetic data on indomethicin
Growth of Loblolly pines
Productivity of workers on machines
School demographic data for MathAchieve
Mathematics achievement scores
Tenderness of meat
Milk production by diet
Muscle response by conc of CaCl2
Assay of nitrendipene
Yield under dierent fertilizers
Growth of orange trees
Orthodontic measurement over time
Number of large ovarian follicles over time
Heights of boys in Oxford, England
Oxide coating on a semiconductor
Change in blood pressure vs. dose of phenylbiguanide
A partially balanced incomplete block design
Neonatal pharmacokinetics of phenobarbitol
X-ray pixel intensities over time
Pharmacokinetic study of quinidine
Travel times of ultrasonic waves in railway rails
Weights of rat pups by litter
Assays of relaxin
Pharmacokinetics of remifentanil
Soybean growth by variety
A.2 AssayBioassay on Cell Culture Plate
425
TABLE A.1: (continued)
Spruce
Tetracycline1
Tetracycline2
Theoph
Wafer
Wheat
Wheat2
Spruce tree growth

Pharmacokinetics of tetracycline
Pharmacokinetics of tetracycline
Pharmacokinetics of theophylline
Current vs. voltage on semiconductor wafers
Yields by growing conditions
Yields from a randomized complete block design
Other data sets may be included with later versions of the library, which
will be made available at http://nlme.stat.wisc.edu.
A.1 AlfalfaSplit-Plot Experiment on Varieties of

Alfalfa
These data are described in Snedecor and Cochran (1980, 16.15) as an
example of a split-plot design. The treatment structure used in the experiment was a 34 full factorial, with three varieties of alfalfa and four
dates of third cutting in 1943. The experimental units were arranged into
six blocks, each subdivided into four plots. The varieties of alfalfa (Cossac,
Ladak, and Ranger ) were assigned randomly to the blocks and the dates
of third cutting (None, S1 September 1, S20 September 20, and O7
October 7) were randomly assigned to the plots. All four dates were used
on each block. The data are presented in Figure A.1.
The display formula for these data is
Yield ~ Date | Block / Variety
based on the columns named:

Yield: the plot yield (T/acre).
Date: the third cutting dateNone, S1, S20, or O7.
Block: a factor identifying the block1 through 6.
Variety: alfalfa varietyCossac, Ladak, or Ranger.
A.2 AssayBioassay on Cell Culture Plate

These data, courtesy of Rich Wolfe and David Lansky from Searle, Inc.,
come from a bioassay run on a 96-well cell culture plate. The assay is performed using a split-block design. The 8 rows on the plate are labeled AH
426

+
Block/Variety
None
4/Ladak
4/Ranger
4/Cossack
1/Cossack
1/Ladak
1/Ranger
3/Ranger
3/Cossack
3/Ladak
2/Ladak
2/Cossack
2/Ranger
5/Cossack
5/Ladak
5/Ranger
6/Ladak
6/Cossack
6/Ranger
>
S1
+
+
+
S20
>
O7
s
s>
>
s
+
>
+
+ >s
>
+
>
+
+
+
>
s
>
>
s
s
+
>
+>
> +
s
s
>
>
s
s>
+
s
>
+s
>
1.0
1.2
1.4
1.6
1.8
2.0
2.2
Yield in 1944 following third date of cutting in 1943 (T/Acre)
FIGURE A.1. Plot yields in a split-plot experiment on alfalfa varieties and dates
of third cutting.
from top to bottom and the 12 columns on the plate are labeled 112 from
left to right. Only the central 60 wells of the plate are used for the bioassay
(the intersection of rows BG and columns 211). There are two blocks in
the design: Block 1 contains columns 26 and Block 2 contains columns
711. Within each block, six samples are assigned randomly to rows and
ve (serial) dilutions are assigned randomly to columns. The response variable is the logarithm of the optical density. The cells are treated with a
compound that they metabolize to produce the stain. Only live cells can
make the stain, so the optical density is a measure of the number of cells
that are alive and healthy. The data are displayed in Figure 4.13 (p. 164).
Columns
logDens ~ 1 | Block

logDens: log-optical density.
Block: a factor identifying the block where the wells are measured.
sample: a factor identifying the sample corresponding to the well, vary-
ing from a to f.
A.4 CefamandolePharmacokinetics of Cefamandole
427
dilut: a factor indicating the dilution applied to the well, varying from
1 to 5.
A.3 BodyWeightBody Weight Growth in Rats

Hand and Crowder (1996) describe data on the body weights of rats measured over 64 days. These data also appear in Table 2.4 of Crowder and
Hand (1990). The body weights of the rats (in grams) are measured on day
1 and every seven days thereafter until day 64, with an extra measurement
on day 44. The experiment started several weeks before day 1. There are
three groups of rats, each on a dierent diet. A plot of the data is presented
in Figure 3.2 (p. 104).
Columns
weight ~ Time | Rat

weight: body weight of the rat (grams).
Time: time at which the measurement is made (days).
Rat: a factor identifying the rat whose weight is measured.
Diet: a factor indicating the diet the rat receives.
A.4 CefamandolePharmacokinetics of
Cefamandole
Davidian and Giltinan (1995, 1.1, p. 2) describe data, shown in Figure A.2,
obtained during a pilot study to investigate the pharmacokinetics of the
drug cefamandole. Plasma concentrations of the drug were measured on six
healthy volunteers at 14 time points following an intraveneous dose of 15
mg/kg body weight of cefamandole.
Columns
conc ~ Time | Subject

conc: observed plasma concentration of cefamandole (mcg/ml).
428

0
100
200
300
Cefamandole concentration (mcg/ml)
250
200
150
100
50
0
2
250
200
150
100
50
0
0
100
200
300
100
200
300
Time post-dose (min)
FIGURE A.2. Plasma concentration of cefamandole versus time post-injection

for six healthy volunteers.
Time: time at which the sample was drawn (minutes post-injection).

Subject: a factor giving the subject from which the sample was drawn.
Models
Davidian and Giltinan (1995) use the biexponential model SSbiexp (C.4,
p. 514) with these data.
A.5 CO2Carbon Dioxide Uptake

Potvin et al. (1990) describe an experiment on the cold tolerance of a C4
grass species, Echinochloa crus-galli. The CO2 uptake of six plants from
Quebec and six plants from Mississippi was measured at several levels
of ambient CO2 concentration. Half the plants of each type were chilled
overnight before the experiment was conducted. The data are shown in
Figure 8.15 (p. 369).
Columns
uptake ~ conc | Plant
A.7 DNaseAssay Data for the Protein DNase
429

uptake: carbon dioxide uptake rate (mol/m2 sec).
conc: ambient concentration of carbon dioxide (mL/L).
Plant: a factor giving a unique identier for each plant.
Type: origin of the plant, Qu
ebec or Mississippi.
Treatment: treatment, chilled or nonchilled.
Models
Potvin et al. (1990) suggest using a modied form of the asymptotic regression model SSasymp (C.1, p. 511), which we have coded as SSasympOff
(C.2, p. 512).
A.6 DialyzerHigh-Flux Hemodialyzer

Vonesh and Carter (1992) describe data measured on high-ux hemodialyzers to assess their in vivo ultraltration characteristics. The ultraltration
rates (in mL/hr) of 20 high-ux dialyzers were measured at seven dierent
transmembrane pressures (in dmHg). The in vitro evaluation of the dialyzers used bovine blood at ow rates of either 200 dl/min or 300 dl/min.
The data, shown in Figure 5.1 (p. 215), are also analyzed in Littell et al.
(1996, 8.2).
Columns
rate ~ pressure | Subject

rate: hemodialyzer ultraltration rate (mL/hr).
pressure: transmembrane pressure (dmHg).
Subject: a factor giving a unique identier for each subject.
QB: bovine blood ow rate (dL/min)200 or 300.
index: index of observation within subject1 through 7.
A.7 DNaseAssay Data for the Protein DNase

Davidian and Giltinan (1995, 5.2.4, p. 134) describe data, shown in Figure 3.8 (p. 115), obtained during the development of an ELISA assay for
the recombinant protein DNase in rat serum.
430
Columns
density ~ conc | Run

density: the measured optical density in the assay. Duplicate optical
density measurements were obtained.

conc: the known concentration of the protein.
Run: a factor giving the run from which the data were obtained.
Models
Davidian and Giltinan (1995) use the four-parameter logistic model, SSfpl
(C.6, p. 517) with these data, modeling the optical density as a logistic
function of the logarithm of the concentration.
A.8 EarthquakeEarthquake Intensity

These data, shown in Figure A.3, are measurements recorded at available
seismometer locations for 23 large earthquakes in western North America
between 1940 and 1980. They were originally given in Joyner and Boore
(1981); are mentioned in Brillinger (1987); and are analyzed in 11.4 of
Davidian and Giltinan (1995).
Columns
accel ~ distance | Quake

accel: maximum horizontal acceleration observed (g).
distance: the distance from the seismological measuring station to the
epicenter of the earthquake (km).

Quake: a factor indicating the earthquake on which the measurements
were made.
Richter: the intensity of the earthquake on the Richter scale.
soil: soil condition at the measuring stationeither soil or rock.
A.9 ergoStoolErgometrics Experiment with Stool Types

1
17
10
431
100
11
10^-1
10^-2
12
19
10^-1
10^-2
10^-1
acceleration (g)
10^-2
18
15
13
21
10^-1
10^-2
10^-1
10^-2
23
22
10
10^-1
10^-2
10^-1
10^-2
20
16
14
10^-1
10^-2
1
10
100
10
100
Distance from epicenter (km)

FIGURE A.3. Lateral acceleration versus distance from the epicenter for 23 large
earthquakes in western North America. Both the acceleration and the distance
are on a logarithmic scale. Earthquakes of greatest intensity as measured on the
Richter scale are in the uppermost panels.
A.9 ergoStoolErgometrics Experiment with Stool

Types
Devore (2000, Exercise 11.9, p. 447) cites data from an article in Ergometrics (1993, pp. 519-535) on The Eects of a Pneumatic Stool and
a One-Legged Stool on Lower Limb Joint Load and Muscular Activity.
These data are shown in Figure 1.5 (p. 13).
effort ~ Type | Subject
432

1
2
8
Blood glucose level (mg/dl)
6
4
2
3
7
8
6
4
2
8
6
4
2
0
10
15
20
25
30
10
15
20
25
30
Time since alcohol ingestion (min/10)
FIGURE A.4. Blood glucose levels of seven subjects measured over a period of 5
hours on two dierent occasions. In both dates the subjects took alcohol at time
0, but on the second occasion a dietary additive was used.

effort: eort to arise from a stool
Type: a factor giving the stool type
Subject: a factor giving a unique identier for the subject in the exper-
iment
A.10 Glucose2Glucose Levels Following Alcohol

Ingestion
Hand and Crowder (1996, Table A.14, pp. 180181) describe data on the
blood glucose levels measured at 14 time points over 5 hours for 7 volunteers
who took alcohol at time 0. The same experiment was repeated on a second
date with the same subjects but with a dietary additive used for all subjects.
A plot of the data is presented in Figure A.4.
Columns
glucose ~ Time | Subject/Date
A.12 IndomethIndomethicin Kinetics
433

weight: blood glucose level (in mg/dl).
Time: time since alcohol ingestion (in min/10).
Subject: a factor identifying the subject whose glucose level is mea-
sured.
Date: a factor indicating the occasion in which the experiment was con-
ducted.
A.11 IGFRadioimmunoassay of IGF-I Protein

Davidian and Giltinan (1995, 3.2.1, p. 65) describe data, shown in Figure 4.6 (p. 144), obtained during quality control radioimmunoassays for ten
dierent lots of radioactive tracer used to calibrate the Insulin-like Growth
Factor (IGF-I) protein concentration measurements.
Columns
conc ~ age | Lot

conc: the estimated concentration of IGF-I protein, in ng/ml.
age: the age (in days) of the radioactive tracer.
Lot: a factor giving the radioactive tracer lot.
A.12 IndomethIndomethicin Kinetics

Kwan et al. (1976) present data on the plasma concentrations of indomethicin following intravenous injection. There are six dierent subjects in the
experiment. The sampling times, ranging from 15 minutes post-injection to
8 hours post-injection, are the same for each subject. The data, presented
in Figure 6.3 (p. 277), are analyzed in Davidian and Giltinan (1995, 2.1)
conc ~ time | Subject

conc: observed plasma concentration of indomethicin (mcg/ml).
time: time at which the sample was drawn (hours post-injection).
Subject: a factor indicating the subject from whom the sample is drawn.
434

5
10 15 20 25
10 15 20 25
323
309
303
305
311
315
321
319
60
50
40
30
Height of tree (ft)
20
10
301
60
50
40
30
20
10
329
327
325
307
331
60
50
40
30
20
10
5
10 15 20 25
10 15 20 25
10 15 20 25
Age of tree (yr)

FIGURE A.5. Height of Loblolly pine trees over time
Models
Davidian and Giltinan (1995) use the biexponential model SSbiexp (C.4,
p. 514) with these data.
A.13 LoblollyGrowth of Loblolly Pine Trees

Kung (1986) presents data, shown in Figure A.5, on the growth of Loblolly
pine trees.
height ~ age | Seed

height: height of the tree (ft).
age: age of the tree (yr).
Seed: a factor indicating the seed source for the tree.
A.15 OatsSplit-plot Experiment on Varieties of Oats
435
A.14 MachinesProductivity Scores for Machines

and Workers
Data on an experiment to compare three brands of machines used in an industrial process are presented in Milliken and Johnson (1992, 23.1, p. 285).
Six workers were chosen randomly among the employees of a factory to operate each machine three times. The response is an overall productivity
score taking into account the number and quality of components produced.
These data, shown in Figure 1.9 (p. 22), are analyzed in Milliken and Johnson (1992) with an ANOVA model.
score ~ Machine | Worker

score: productivity score.
Machine: a factor identifying the machine brandA, B, or C.
Worker: a factor giving the unique identier for each worker.
A.15 OatsSplit-plot Experiment on Varieties of

Oats
These data have been introduced by Yates (1935) as an example of a splitplot design. The treatment structure used in the experiment was a 34
full factorial, with three varieties of oats and four concentrations of nitrogen. The experimental units were arranged into six blocks, each with three
whole-plots subdivided into four subplots. The varieties of oats were assigned randomly to the whole-plots and the concentrations of nitrogen to
the subplots. All four concentrations of nitrogen were used on each wholeplot.
The data, presented in Figure 1.20 (p. 47), are analyzed in Venables and
Ripley (1999, 6.11).
yield ~ nitro | Block

yield: the subplot yield (bushels/acre).
nitro: nitrogen concentration (cwt/acre)0.0, 0.2, 0.4, or 0.6.
Block: a factor identifying the blockI through VI.
Variety: oats varietyGolden Rain, Marvellous, or Victory.
436
A.16 OrangeGrowth of Orange Trees

Draper and Smith (1998, Exercise 24.N, p. 559) present data on the growth
of a group of orange trees. These data are plotted in Figure 8.1 (p. 339).
circumference ~ age | Tree

circumference: circumference of the tree (mm)
age: time in days past the arbitrary origin of December 31, 1968.
Tree: a factor identifying the tree on which the measurement is made.
Models
The logistic growth model, SSlogis (C.7, p. 519) provides a reasonable t
to these data.
A.17 OrthodontOrthodontic Growth Data

Investigators at the University of North Carolina Dental School followed
the growth of 27 children (16 males, 11 females) from age 8 until age 14.
Every two years they measured the distance between the pituitary and
the pterygomaxillary ssure, two points that are easily identied on x-ray
exposures of the side of the head. These data are reported in Pottho and
Roy (1964) and plotted in Figure 1.11 (p. 31).

distance: the distance from the center of the pituitary to the pterygo-
maxillary ssure (mm).

age: the age of the subject when the measurement is made (years).
Subject: a factor identifying the subject on whom the measurement was
made.
Sex: a factor indicating if the subject is male or female.
Models:
Based on the relationship shown in Figure 1.11 we begin with a simple
linear relationship between distance and age
A.20 OxideVariability in Semiconductor Manufacturing
437
A.18 OvaryCounts of Ovarian Follicles

Pierson and Ginther (1987) report on a study of the number of large ovarian
follicles detected in dierent mares at several times in their estrus cycles.
These data are shown in Figure 5.10 (p. 240).
follicles ~ Time | Mare

follicles: the number of ovarian follicles greater than 10 mm in diam-
eter.
Time: time in the estrus cycle. The data were recorded daily from 3 days
before ovulation until 3 days after the next ovulation. The measurement times for each mare are scaled so that the ovulations for each
mare occur at times 0 and 1.
Mare: a factor indicating the mare on which the measurement is made.
A.19 OxboysHeights of Boys in Oxford

These data are described in Goldstein (1987) as data on the height of
a selection of boys from Oxford, England versus a standardized age. We
display the data in Figure 3.1 (p. 99).
height ~ age | Subject

height: height of the boy (cm)
age: standardized age (dimensionless)
Subject: a factor giving a unique identier for each boy in the experi-
ment
Occasion: an ordered factorthe result of converting age from a con-
tinuous variable to a count so these slightly unbalanced data can be

analyzed as balanced.
A.20 OxideVariability in Semiconductor

Manufacturing
These data are described in Littell et al. (1996, 4.4, p. 155) as coming from
a passive data collection study in the semiconductor industry where the objective is to estimate the variance components to determine the assignable
438
causes of the observed variability. The observed response is the thickness

of the oxide layer on silicon wafers, measured at three dierent sites of each
of three wafers selected from each of eight lots sampled from the population
of lots. We display the data in Figure 4.14 (p. 168).
Thickness ~ 1 | Lot/Wafer

Thickness: thickness of the oxide layer.
Lot: a factor giving a unique identier for each lot.
Wafer: a factor giving a unique identier for each wafer within a lot.
A.21 PBGEect of Phenylbiguanide on Blood

Pressure
Data on an experiment to examine the eect of a antagonist MDL 72222 on
the change in blood pressure experienced with increasing dosage of phenylbiguanide are described in Ludbrook (1994) and analyzed in Venables and
Ripley (1999, 8.8). Each of ve rabbits was exposed to increasing doses of
phenylbiguanide after having either a placebo or the HD5 -antagonist MDL
72222 administered. The data are shown in Figure 3.4 (p. 107).
deltaBP ~ dose | Rabbit

deltaBP: change in blood pressure (mmHg).
dose: dose of phenylbiguanide (g).
Rabbit: a factor identifying the test animal.
Treatment: a factor identifying whether the observation was made after
administration of placebo or the HD5 -antagonist MDL 72222.
Models
The form of the response suggests a logistic model SSlogis (C.7, p. 519)
for the change in blood pressure as function of the logarithm of the concentration of PBG.
A.22 PBIBA Partially Balanced Incomplete Block Design
Block
+
>
5
13
6
4
15
7
14
10
9
3
2
12
8
11
1
s
w
#
1
2
3
4
5
6
>
s
w
7
8
9
w
s
s
2.0
>
> s
{
{ s
#
> # w #
+
s
w
#
+{
>
w {
+ + #
>
w
# {
+{
> >
+
2.5
#
{
13
14
15
{
>
+
10
11
12
439
#
+
w
s
s
w
{
3.0
3.5
4.0
response
FIGURE A.6. Data on the response in an experiment conducted using fteen

treatments in fteen blocks of size four. The responses are shown by block with
dierent characters indicating dierent treatments.
A.22 PBIBA Partially Balanced Incomplete

Block Design
Data from a partially balanced incomplete block design in which there
were fteen treatments used in fteen blocks of size four. The blocking is
incomplete in that only a subset of the treatments can be used in each block.
It is partially balanced in that every pair of treatments occurs together in
a block the same number of times.
These data were described in Cochran and Cox (1957, p. 456). They are
also used as data set 1.5.1 in Littell et al. (1996, 1.5.1). The data are
shown in Figure A.6.
response ~ Treatment | Block

response: the continuous response in the experiment
Treatment: the treatment factor
Block: the block
440
A.23 PhenobarbPhenobarbitol Kinetics

Data from a pharmacokinetics study of phenobarbital in neonatal infants.
During the rst few days of life the infants receive multiple doses of phenobarbital for prevention of seizures. At irregular intervals blood samples are
drawn and serum phenobarbital concentrations are determined. The data,
displayed in Figure 6.15 (p. 296), were originally given in Grasela and Donn
(1985) and are analyzed in Boeckmann et al. (1994) and in Davidian and
Giltinan (1995, 6.6).

conc: phenobarbital concentration in the serum (g/L).
time: time when the sample is drawn or drug administered (hr).
Subject: a factor identifying the infant.
Wt: birth weight of the infant (kg).
Apgar: the 5-minute Apgar score for the infant. This is an indication of
health of the newborn infant. The scale is 1 10.

ApgarInd: a factor indicating whether the 5-minute Apgar score is < 5
or 5.
dose: dose of drug administered (g/kg).
Models
A one-compartments open model with intravenous administration and rstorder elimination, described in 6.4, is used for these data
A.24 PixelPixel Intensity in Lymphnodes

These data are from an experiment conducted by Deborah Darien, Department of Medical Sciences, School of Veterinary Medicine, University of
Wisconsin, Madison. The mean pixel intensity of the right and left lymphnodes in the axillary region obtained from CT scans of 10 dogs were recorded
over a period of 14 days after intravenous application of a contrast. The
data are shown in Figure 1.17 (p. 42).
pixel ~ day | Dog
A.25 QuinidineQuinidine Kinetics
441
pixel: mean pixel intensity of lymphnode in the CT scan.

day: number of days since contrast administration.
Dog: a factor giving the unique identier for each dog.
Side: a factor indicating the side on which the measurement was made.
A.25 QuinidineQuinidine Kinetics

Verme, Ludden, Clementi and Harris (1992) analyze routine clinical data on
patients receiving the drug quinidine as a treatment for cardiac arrythmia
(atrial brillation of ventricular arrythmias). All patients were receiving
oral quinidine doses. At irregular intervals blood samples were drawn and
serum concentrations of quinidine were determined. These data, shown in
Figure A.7, are analyzed in several publications, including Davidian and
Giltinan (1995, 9.3).

conc: serum quinidine concentration (mg/L).
time: time (hr) at which the drug was administered or the blood sample
drawn. This is measured from the time the patient entered the study.
Subject: a factor identifying the patient on whom the data were col-
lected.
dose: dose of drug administered (mg). Although there were two dif-
ferent forms of quinidine administered, the doses were adjusted for

dierences in salt content by conversion to milligrams of quinidine
base.
interval: when the drug has been given at regular intervals for a suf-
ciently long period of time to assume steady state behavior, the

interval is recorded.
Age: age of the subject on entry to the study (yr).
Height: height of the subject on entry to the study (in.).
Weight: body weight of the subject (kg).
Race: a factor identifying the raceCaucasian, Black, or Latin.
442

0
2000
6000
2000
6000
2000
6000
2000
6000
69
81
54
41
74
28
51
78
25
61
64
60
59
10
76
94
58
113
50
39
98
30
82
93
108
119
32
133
128
136
21
43
90
102
40
84
107
130
66
139
33
80
125
110
79
95
114
135
105
116
62
65
112
127
55
68
124
35
47
20
96
99
134
12
49
67
85
72
100
71
14
26
75
87
103
138
45
44
97
36
37
104
118
137
17
29
34
46
73
223
19
38
42
52
56
63
83
57
77
115
121
123
11
48
126
53
122
129
132
16
106
15
22
88
91
117
120
13
89
27
109
70
23
92
111
18
24
6
2
6
2
6
2
6
2
Serum quinidine concentration (mg/L)
6
2
6
2
6
2
6
2
6
2
6
2
6
2
6
2
6
2
6
2
6
2
6
2
0
2000
6000
2000
6000
2000
6000
2000
6000
Time from patient entering study (hr.)
FIGURE A.7. Serum concentrations of quinidine in 136 hospitalized patients

under varying dosage regimens versus time since entering the study.
Smoke: a factor giving smoking status at the time of the measurement
no or yes.
Ethanol: a factor giving ethanol (alcohol) abuse status at the time of
the measurementnone, current, or former.

Heart: a factor indicating congestive heart failure for the subject
none/mild, moderate, or severe.

Creatinine: a factor in eight levels coding the creatinine clearance and
other measurements. Creatinine clearance is divided into those greater

than 50 mg/min and those less than 50 mg/min.
A.27 SoybeanSoybean Leaf Weight over Time
443
glyco: alpha-1 acid glycoprotein concentration (mg/dL). Often mea-
sured at the same time as the quinidine concentration.

Models
A model for these data is described in 8.2.2.
A.26 RailEvaluation of Stress in Rails

Devore (2000, Example 10.10, p. 427) cites data from an article in Materials Evaluation on a study of travel time for a certain type of wave that
results from longitudinal stress of rails used for railroad track. The data
are displayed in Figure 1.1 (p. 4).
travel ~ 1 | Rail

travel: travel time for ultrasonic head-waves in the rail (nanoseconds).
The value given is the original travel time minus 36,100 nanoseconds.
Rail: a factor giving the number of the rail on which the measurement
was made.
A.27 SoybeanSoybean Leaf Weight over Time

These data, shown in Figure 6.10 (p. 288), are described in Davidian and
Giltinan (1995, 1.1.3, p. 7) as Data from an experiment to compare
growth patterns of two genotypes of soybeans: Plant Introduction #416937
(P), an experimental strain, and Forrest (F), a commercial variety.
weight ~ Time | Plot

weight: average leaf weight per plant (g).
Time: time the sample was taken (days after planting).
Plot: a factor giving a unique identier for each plot.
Variety: a factor indicating the variety; Forrest (F) or Plant Introduc-
tion #416937 (P)

Year: the year the plot was planted.
444
Models
The form of the response suggests a logistic model, SSlogis ( C.7, p. 519).
A.28 SpruceGrowth of Spruce Trees

Diggle et al. (1994, Example 1.3, page 5) describe data on the growth of
spruce trees that have been exposed to an ozone-rich atmosphere or to a
normal atmosphere. These data are plotted in Figures A.8A.10. The
display formula for these data is
logSize ~ days | Tree

logSize: the logarithm of an estimate of the volume of the tree trunk
days: number of days since the beginning of the experiment
Tree: a factor giving a unique identier for each tree
Plot: a factor identifying the plot in which the tree was grown. The
levels of this factor are Ozone1, Ozone2, Normal1, and Normal2.

Treatment a factor indicating whether the tree was grown in an ozone-
rich atmosphere or a normal atmosphere.
A.29 TheophTheophylline Kinetics

Boeckmann et al. (1994) report data from a study by Dr. Robert Upton of
the kinetics of the anti-asthmatic drug theophylline. Twelve subjects were
given oral doses of theophylline then serum concentrations were measured
at 11 time points over the next 25 hours. Davidian and Giltinan (1995) also
analyze these data, shown in Figure 8.6 (p. 352).
conc ~ Time | Subject

conc: theophylline concentration in the sample (mg/L).
Time: time since drug administration when the sample was drawn (hr).
Subject: a factor identifying the subject.
Wt: weight of the subject (kg).
Dose: dose administered to the subject (mg/kg).
200
400
600
200
400
600
200
400
445
600
O1T07
O1T01
O1T06
O1T17
O1T05
O1T09
O1T04
O1T25
O1T12
O1T08
O1T13
O1T23
O1T03
O1T02
O1T11
O1T21
O1T20
O1T27
O1T14
O1T22
6
5
log(Size)
4
3
7
6
5
4
O1T24
O1T18
O1T19
O1T15
O1T10
O1T26
O1T16
7
6
4
3
200
400
600
200
400
600
200
400
600
200
400
600

FIGURE A.8. Growth measures in the logarithm of an estimate of the volume of
the spruce tree trunk versus time. These 27 trees were in the rst plot that was
exposed to an ozone-rich atmosphere throughout the experiment
446
200
400
600
200
400
600
200
400
600
O2T03
O2T12
O2T14
O2T02
O2T10
O2T08
O2T17
O2T05
O2T07
O2T16
O2T13
O2T19
O2T11
O2T27
O2T21
O2T24
O2T15
O2T25
O2T01
O2T20
6
5
log(Size)
4
3
7
6
5
4
O2T18
O2T23
O2T09
O2T26
O2T22
O2T04
O2T06
7
6
4
3
200
400
600
200
400
600
200
400
600
200
400
600

FIGURE A.9. Growth measures in the logarithm of an estimate of the volume
of the spruce tree trunk versus time. These 27 trees were in the second plot that
was exposed to an ozone-rich atmosphere throughout the experiment
200
N2T06
400
600
200
N2T07
N2T04
400
600
200
N2T01
N2T12
400
447
600
N2T05
7
6
5
4
N2T11
N2T10
N2T13
N2T09
N2T02
N1T05
N1T04
N1T06
N1T02
N1T07
N1T10
N1T09
N1T12
N1T08
N1T03
N2T03
N2T08
N1T01
N1T11
6
5
log(Size)
4
3
7
6
4
3
200
400
600
200
400
600
200
400
600
200
400
600

FIGURE A.10. Growth measures in the logarithm of an estimate of the volume
of the spruce tree trunk versus time. These 25 trees were in the rst and second
plots that were exposed to an normal atmosphere throughout the experiment
448
Models:
Both Boeckmann et al. (1994) and Davidian and Giltinan (1995) use a twocompartment open pharmacokinetic model, which we code as SSfol (C.5,
p. 516), for these data.
A.30 WaferModeling of Analog MOS Circuits

In an experiment conducted at the Microelectronics Division of Lucent
Technologies to study the variability in the manufacturing of analog MOS
circuits, the intensities of the current at ve ascending voltages were collected on n-channel devices. Measurements were made on eight sites of each
of ten wafers. Figure 3.11 (p. 118) shows the response curves for each site,
by wafer.
current ~ voltage | Wafer/Site

current: the intensity of current (mA).
voltage: the voltage applied to the device (V).
Wafer: a factor giving a unique identier for each wafer.
Site: a factor giving an identier for each site within a wafer.
A.31 Wheat2Wheat Yield Trials

Stroup and Baenziger (1994) report data on an agronomic yield trial to
compare 56 dierent varieties of wheat. The experimental units were organized according to a randomized complete block design with four blocks.
All 56 varieties of wheat were used in each block. The latitude and longitude of each experimental unit in the trial were also recorded. The data,
shown in Figure 5.22 (p. 261), are also analyzed in Littell et al. (1996,
9.6.2).
Columns
yield ~ variety | Block

yield: wheat yield.
A.31 Wheat2Wheat Yield Trials
449
variety: a factor giving the unique identier for each wheat variety.
Block: a factor giving a unique identier for each block in the experi-
ment.
latitude: latitude of the experimental unit.
longitude: longitude of the experimental unit.
Appendix B
S Functions and Classes
There are over 300 dierent functions and classes dened in the nlme library.
In this appendix we reproduce the on-line documentation for those functions and classes that are most frequently used in the examples in the
text. The documentation for all the functions and classes in the library is
available with the library.
Autocorrelation Function
ACF
ACF(object, maxLag, ...)
Arguments
object
maxLag
...
Any object from which an autocorrelation function

can be obtained. Generally an object resulting from
a model t, from which residuals can be extracted.
Maximum lag for which the autocorrelation should
be calculated.
Some methods for this generic require additional arguments.
Description
This function is generic; method functions can be written to handle
specic classes of objects. Classes that already have methods for this
function include gls and lme.
452
Appendix B. S Functions and Classes
Value
Will depend on the method function used; see the appropriate documentation.
See Also
ACF.gls, ACF.lme
ACF.lme
Autocorrelation Function for lme Residuals
ACF(object, maxLag, resType)

Arguments
object
An object inheriting from class lme, representing a

tted linear mixed-eects model.
maxLag
An optional integer giving the maximum lag for which

the autocorrelation should be calculated. Defaults to
maximum lag in the within-group residuals.
resType
An optional character string specifying the type of

residuals to be used. If "response", the raw residuals (observed tted) are used; else, if "pearson",
the standardized residuals (raw residuals divided by
the corresponding standard errors) are used; else, if
"normalized", the normalized residuals (standardized residuals premultiplied by the inverse squareroot factor of the estimated error correlation matrix)
are used. Partial matching of arguments is used, so
only the rst character needs to be provided. Defaults
to "pearson".
Description
This method function calculates the empirical autocorrelation function (Box et al., 1994) for the within-group residuals from an lme t.
The autocorrelation values are calculated using pairs of residuals within
the innermost group level. The autocorrelation function is useful for investigating serial correlation models for equally spaced data.
Value
A data frame with columns lag and ACF representing, respectively, the
lag between residuals within a pair and the corresponding empirical
autocorrelation. The returned value inherits from class ACF.
anova.lme
453
See Also
ACF.gls, plot.ACF
Examples
fm1 <- lme(follicles ~ sin(2*pi*Time) + cos(2*pi*Time), Ovary,
random = ~ sin(2*pi*Time) | Mare)
ACF(fm1, maxLag = 11)
anova.lme
Compare Likelihoods of Fitted Objects
anova(object, ..., test, type, adjustSigma, Terms, L,

verbose)
Arguments
object
A tted model object inheriting from class lme,

representing a mixed-eects model.
...
Other optional tted model objects inheriting from

classes gls, gnls, lm, lme, lmList, nlme, nlsList,
or nls.
test
An optional logical value controlling whether likelihood ratio tests should be used to compare the tted
models represented by object and the objects in ....
Defaults to TRUE.
type

sum of squares to be used in F-tests for the terms in
the model. If "sequential", the sequential sum of
squares obtained by including the terms in the order
they appear in the model is used; else, if "marginal",
the marginal sum of squares obtained by deleting a
term from the model at a time is used. This argument is only used when a single tted object is passed
to the function. Partial matching of arguments is
used, so only the rst character needs to be provided.
Defaults to "sequential".
adjustSigma
An optional logical value. If TRUE and the estimation

method used to obtain object was maximum likelihood,
the residual standard error is multiplied by

nobs /(nobs npar ), converting it to a REML-like
estimate. This argument is only used when a single tted object is passed to the function. Default
is TRUE.
454
Terms
An optional integer or character vector specifying

which terms in the model should be jointly tested to
be zero using a Wald F-test. If given as a character
vector, its elements must correspond to term names;
else, if given as an integer vector, its elements must
correspond to the order in which terms are included
in the model. This argument is only used when a single tted object is passed to the function. Default is
NULL.
An optional numeric vector or array specifying linear

combinations of the coecients in the model that
should be tested to be zero. If given as an array,
its rows dene the linear combinations to be tested.
If names are assigned to the vector elements (array
columns), they must correspond to names of the coecients and will be used to map the linear combination(s) to the coecients; else, if no names are
available, the vector elements (array columns) are assumed in the same order as the coecients appear in
the model. This argument is only used when a single tted object is passed to the function. Default is
NULL.
verbose
An optional logical value. If TRUE, the calling sequences for each tted model object are printed with
the rest of the output, being omitted if verbose =
FALSE. Defaults to FALSE.
Description
When only one tted model object is present, a data frame with the
sums of squares, numerator degrees of freedom, denominator degrees
of freedom, F-values, and p-values for Wald tests for the terms in the
model (when Terms and L are NULL), a combination of model terms
(when Terms in not NULL), or linear combinations of the model coefcients (when L is not NULL). Otherwise, when multiple tted objects
are being compared, a data frame with the degrees of freedom, the (restricted) log-likelihood, the Akaike Information Criterion (AIC), and
the Bayesian Information Criterion (BIC) of each object is returned.
If test=TRUE, whenever two consecutive objects have dierent number
of degrees of freedom, a likelihood ratio statistic, with the associated
p-value is included in the returned data frame.
Value
A data frame inheriting from class anova.lme.
coef.lme
455
Note
Likelihood comparisons are not meaningful for objects t using
restricted maximum likelihood and with dierent xed eects.
See Also
gls, gnls, nlme, lme, AIC, BIC, print.anova.lme
Examples
fm1 <- lme(distance ~ age, Orthodont, random = ~ age | Subject)
anova(fm1)
fm2 <- update(fm1, random = pdDiag(~age))
anova(fm1, fm2)
coef.lme
Extract lme Coecients
coef(object, augFrame, level, data, which, FUN,

omitGroupingFactor)
Arguments
object

augFrame
An optional logical value. If TRUE, the returned data

frame is augmented with variables dened in data;
else, if FALSE, only the coecients are returned.
Defaults to FALSE.
level
An optional positive integer giving the level of grouping to be used in extracting the coecients from an
object with multiple nested grouping levels. Defaults
to the highest or innermost level of grouping.
data
An optional data frame with the variables to be used

for augmenting the returned data frame when
augFrame = TRUE. Defaults to the data frame used
to t object.
which
An optional positive integer or character vector specifying which columns of data should be used in the
augmentation of the returned data frame. Defaults
to all columns in data.
456
An optional summary function or a list of summary

functions to be applied to group-varying variables,
when collapsing data by groups. Group-invariant variables are always summarized by the unique value that
they assume within that group. If FUN is a single function it will be applied to each noninvariant variable
by group to produce the summary for that variable. If
FUN is a list of functions, the names in the list should
designate classes of variables in the frame such as
ordered, factor, or numeric. The indicated function will be applied to any group-varying variables of
that class. The default functions to be used are mean
for numeric factors, and Mode for both factor and
ordered. The Mode function, dened internally in
gsummary, returns the modal or most popular value
of the variable. It is dierent from the mode function
that returns the S-language mode of the variable.
omitGroupingFactor
An optional logical value. When TRUE the grouping
factor itself will be omitted from the groupwise summary of data, but the levels of the grouping factor
will continue to be used as the row names for the
returned data frame. Defaults to FALSE.
FUN
Description
The estimated coecients at level i are obtained by adding together the
xed-eects estimates and the corresponding random-eects estimates
at grouping levels less or equal to i. The resulting estimates are returned
as a data frame, with rows corresponding to groups and columns to
coecients. Optionally, the returned data frame may be augmented
with covariates summarized over groups.
Value
A data frame inheriting from class coef.lme with the estimated coecients at level level and, optionally, other covariates summarized over
groups. The returned object also inherits from classes ranef.lme and
data.frame.
See Also
lme, fixef.lme, ranef.lme, plot.ranef.lme, gsummary
Examples
coef.lmList
457
coef(fm1)
coef(fm1, augFrame = TRUE)
coef.lmList
Extract lmList Coecients
coef(object, augFrame, data, which, FUN,

omitGroupingFactor)
Arguments
object
augFrame
data
which
FUN
An object inheriting from class lmList, representing

a list of lm objects with a common model.
frame is augmented with variables dened in the data
frame used to produce object; else, if FALSE, only
the coecients are returned. Defaults to FALSE.
to t object.
An optional positive integer or character vector specifying which columns of the data frame used to produce object should be used in the augmentation of
the returned data frame. Defaults to all variables in
the data.
when collapsing the data by groups. Group-invariant
variables are always summarized by the unique value
that they assume within that group. If FUN is a single function it will be applied to each noninvariant
variable by group to produce the summary for that
variable. If FUN is a list of functions, the names in
the list should designate classes of variables in the
frame such as ordered, factor, or numeric. The indicated function will be applied to any group-varying
variables of that class. The default functions to be
used are mean for numeric factors, and Mode for both
factor and ordered. The Mode function, dened internally in gsummary, returns the modal or most popular value of the variable. It is dierent from the
mode function that returns the S-language mode of
the variable.
458
omitGroupingFactor
factor itself will be omitted from the groupwise summary of data but the levels of the grouping factor
Description
The coecients of each lm object in the object list are extracted and
organized into a data frame, with rows corresponding to the lm components and columns corresponding to the coecients. Optionally, the
returned data frame may be augmented with covariates summarized
over the groups associated with the lm components.
Value
A data frame inheriting from class coef.lmList with the estimated coecients for each lm component of object and, optionally, other covariates summarized over the groups corresponding to the lm components. The returned object also inherits from classes ranef.lmList and
data.frame.
See Also
lmList, fixed.effects.lmList, ranef.lmList,
plot.ranef.lmList, gsummary
Examples
fm1 <- lmList(distance ~ age|Subject, data = Orthodont)
coef(fm1)
coef(fm1, augFrame = TRUE)
fitted.lme
Extract lme Fitted Values
fitted(object, level, asList)

Arguments
object

level
An optional integer vector giving the level(s) of grouping to be used in extracting the tted values from
object. Level values increase from outermost to innermost grouping, with level zero corresponding to
fixef
459
the population tted values. Defaults to the highest

or innermost level of grouping.
asList
An optional logical value. If TRUE and a single value

is given in level, the returned object is a list with
the tted values split by groups; else the returned
value is either a vector or a data frame, according to
the length of level. Defaults to FALSE.
Description
The tted values at level i are obtained by adding together the population-tted values (based only on the xed-eects estimates) and the
estimated contributions of the random eects to the tted values at
grouping levels less or equal to i. The resulting values estimate the
best linear unbiased predictions (BLUPs) at level i.
Value
If a single level of grouping is specied in level, the returned value is
either a list with the tted values split by groups (asList = TRUE) or
a vector with the tted values (asList = FALSE); else, when multiple
grouping levels are specied in level, the returned object is a data
frame with columns given by the tted values at dierent levels and
the grouping factors.
See Also
lme, residuals.lme
Examples
fm1 <- lme(distance ~ age + Sex, data = Orthodont, random = ~ 1)
fitted(fm1, level = 0:1)
Extract Fixed Eects
fixef
fixef(object, ...)
fixed.effects(object, ...)
Arguments
object
Any tted model object from which xed-eects

estimates can be extracted.
...
Some methods for this generic function require additional arguments.
460
Description
function include lmList and lme.
Value
See Also
fixef.lmList, fixef.lme
gapply
Apply a Function by Groups
gapply(object, which, FUN, form, level, groups, ...)

Arguments
object
An object to which the function will be applied, usually a groupedData object or a data.frame. Must
inherit from class data.frame.
which
An optional character or positive integer vector specifying which columns of object should be used with
FUN. Defaults to all columns in object.
FUN
Function to apply to the distinct sets of rows of the

data frame object dened by the values of groups.
form
An optional one-sided formula that denes the groups.

When this formula is given the right-hand side is evaluated in object, converted to a factor if necessary,
and the unique levels are used to dene the groups.
Defaults to formula(object).
level
An optional positive integer giving the level of grouping to be used in an object with multiple nested
grouping levels. Defaults to the highest or innermost
level of grouping.
groups
An optional factor that will be used to split the rows

into groups. Defaults to getGroups(object, form,
level).
...
Optional additional arguments to the summary function FUN. Often it is helpful to specify na.rm = TRUE.
getGroups
461
Description
Applies the function to the distinct sets of rows of the data frame
dened by groups.
Value
Returns a data frame with as many rows as there are levels in the
groups argument.
See Also
gsummary
Examples
## Find number of nonmissing "conc" observations for each Subject
gapply( Quinidine, FUN = function(x) sum(!is.na(x$conc)) )
getGroups
Extract Grouping Factors from an Object
getGroups(object, form, level, data)

Arguments
object
form
level
data
Any object.
An optional formula with a conditioning expression
on its right hand side (i.e., an expression involving
the | operator). Defaults to formula(object).
A positive integer vector with the level(s) of grouping
to be used when multiple nested levels of grouping are
present. This argument is optional for most methods
of this generic function and defaults to all levels of
nesting.
A data frame in which to interpret the variables named
in form. Optional for most methods.
Description
function include corStruct, data.frame, gls, lme, lmList, and varFunc.
Value
462
See Also
getGroupsFormula, getGroups.data.frame, getGroups.gls,
getGroups.lmList, getGroups.lme
Fit Linear Model Using Generalized Least Squares
gls
gls(model, data, correlation, weights, subset, method,

na.action, control, verbose)
Arguments
model
A two-sided linear formula object describing the

model, with the response on the left of a operator and the terms, separated by + operators, on the
right.
data
An optional data frame containing the variables

named in model, correlation, weights, and subset.
By default the variables are taken from the environment from which gls is called.
correlation
An optional corStruct object describing the withingroup correlation structure. See the documentation
of corClasses for a description of the available
corStruct classes. If a grouping variable is to be used,
it must be specied in the form argument to the
corStruct constructor. Defaults to NULL, corresponding to uncorrelated errors.
weights
An optional varFunc object or one-sided formula describing the within-group heteroscedasticity structure.
If given as a formula, it is used as the argument to
varFixed, corresponding to xed variance weights.
See the documentation on varClasses for a description of the available varFunc classes. Defaults to NULL,
corresponding to homoscesdatic errors.
subset
An optional expression indicating which subset of the

rows of data should be used in the t. This can be a
logical vector, or a numeric vector indicating which
observation numbers are to be included, or a character vector of the row names to be included. All
observations are included by default.
method
A character string. If "REML" the model is t by maximizing the restricted log-likelihood. If "ML" the loglikelihood is maximized. Defaults to "REML".
gls
463
na.action
A function that indicates what should happen when

the data contain NAs. The default action (na.fail)
causes gls to print an error message and terminate
if there are any incomplete observations.
control
A list of control values for the estimation algorithm

to replace the default values returned by the function
glsControl. Defaults to an empty list.
verbose
An optional logical value. If TRUE information on the

evolution of the iterative algorithm is printed. Default is FALSE.
Description
This function ts a linear model using generalized least squares. The
errors are allowed to be correlated and/or have unequal variances.
Value
An object of class gls representing the linear model t. Generic functions such as print, plot, and summary have methods to show the
results of the t. See glsObject for the components of the t. The
functions resid, coef, and fitted can be used to extract some of its
components.
References
The dierent correlation structures available for the correlation argument are described in Box et al. (1994), Littell et al. (1996), and
Venables and Ripley (1999). The use of variance functions for linear
and nonlinear models is presented in detail in Carroll and Ruppert
(1988) and Davidian and Giltinan (1995).
See Also
glsControl, glsObject, varFunc, corClasses, varClasses
Examples
# AR(1) errors within each Mare
fm1 <- gls(follicles ~ sin(2*pi*Time) + cos(2*pi*Time), Ovary,
correlation = corAR1(form = ~ 1 | Mare))
# variance increases as a power of the absolute fitted values
fm2 <- gls(follicles ~ sin(2*pi*Time) + cos(2*pi*Time), Ovary,
weights = varPower())
464
gnls
Fit Nonlinear Model Using Generalized Least

Squares
gnls(model, data, params, start, correlation, weights,

subset, na.action, naPattern, control, verbose)
Arguments
model
A two-sided formula object describing the model, with

the response on the left of a ~ operator and a nonlinear expression involving parameters and covariates
on the right. If data is given, all names used in the
formula should be dened as parameters or variables
in the data frame.
data
An optional data frame containing the variables used

in model, correlation, weights, subset, and naPattern. By default the variables are taken from the
environment from which gnls is called.
params
An optional two-sided linear formula of the form

p1+ +pn~x1+ +xm, or list of two-sided formulas of
the form p1~x1+ +xm, with possibly dierent models for each parameter. The p1,...,pn represent parameters included on the right-hand side of model
and x1+ +xm dene a linear model for the parameters (when the left-hand side of the formula contains
several parameters, they are all assumed to follow
the same linear model described by the right-hand
side expression). A 1 on the right-hand side of the
formula(s) indicates a single xed eect for the corresponding parameter(s). By default, the parameters
are obtained from the names of start.
start
An optional named list, or numeric vector, with the

initial values for the parameters in model. It can be
omitted when a selfStarting function is used in
model, in which case the starting estimates will be
obtained from a single call to the nls function.
correlation
corStruct classes. If a grouping variable is to be used,
it must be specied in the form argument to the
corStruct constructor. Defaults to NULL, corresponding to uncorrelated errors.
gnls
465
weights
An optional varFunc object or one-sided formula describing the within-group heteroscedastic structure.
corresponding to homoscesdatic errors.
subset
An optional expression indicating which subset of the

rows of data should be used in the t. This can be a
logical vector, or a numeric vector indicating which
observation numbers are to be included, or a character vector of the row names to be included. All
na.action

causes gnls to print an error message and terminate
naPattern
An expression or formula object, specifying which returned values are to be regarded as missing.
control

gnlsControl. Defaults to an empty list.
verbose

Description
This function ts a nonlinear model using generalized least squares.
The errors are allowed to be correlated and/or have unequal variances.
Value
An object of class gnls, also inheriting from class gls, representing the
nonlinear model t. Generic functions such as print, plot and summary
have methods to show the results of the t. See gnlsObject for the
components of the t. The functions resid, coef, and fitted can be
used to extract some of its components.
References
The dierent correlation structures available for the correlation argument are described in Box et al. (1994), Littell et al. (1996), and
Venables and Ripley (1999). The use of variance functions for linear
466
and nonlinear models is presented in detail in Carroll and Ruppert

(1988) and Davidian and Giltinan (1995).
See Also
gnlsControl, gnlsObject, varFunc, corClasses, varClasses
Examples
# variance increases with a power of the absolute fitted values
fm1 <- gnls(weight ~ SSlogis(Time, Asym, xmid, scal), Soybean,
weights = varPower())
# errors follow an auto-regressive process of order 1
fm2 <- gnls(weight ~ SSlogis(Time, Asym, xmid, scal), Soybean,
correlation = corAR1())
groupedData
Construct a groupedData Object
groupedData(formula, data, order.groups, FUN, outer, inner,

labels, units)
Arguments
formula
A formula of the form resp cov | group where

resp is the response, cov is the primary covariate,
and group is the grouping factor. The expression 1
can be used for the primary covariate when there is
no other suitable candidate. Multiple nested grouping factors can be listed separated by the / symbol as
in fact1/fact2. In an expression like this the fact2
factor is nested within the fact1 factor.
data
A data frame in which the expressions in formula

can be evaluated. The resulting groupedData object
will consist of the same data values in the same order,
but with additional attributes.
order.groups
An optional logical value, or list of logical values, indicating if the grouping factors should be converted
to ordered factors according to the function FUN applied to the response from each group. If multiple
levels of grouping are present, this argument can be
either a single logical value (which will be repeated
for all grouping levels) or a list of logical values. If
no names are assigned to the list elements, they are
groupedData
467
assumed in the same order as the group levels (outermost to innermost grouping). Ordering within a level
of grouping is done within the levels of the grouping
factors which are outer to it. Changing the grouping
factor to an ordered factor does not aect the ordering of the rows in the data frame, but it does aect
the order of the panels in a trellis display of the data
or models tted to the data. Defaults to TRUE.
FUN
An optional summary function that will be applied

to the values of the response for each level of the
grouping factor, when order.groups = TRUE, to determine the ordering. Defaults to the max function.
outer
An optional one-sided formula, or list of one-sided

formulas, indicating covariates that are outer to the
grouping factor(s). If multiple levels of grouping are
present, this argument can be either a single onesided formula, or a list of one-sided formulas. If no
names are assigned to the list elements, they are assumed in the same order as the group levels (outermost to innermost grouping). An outer covariate
is invariant within the sets of rows dened by the
grouping factor. Ordering of the groups is done in
such a way as to preserve adjacency of groups with
the same value of the outer variables. When plotting
a groupedData object, the argument outer = TRUE
causes the panels to be determined by the outer formula. The points within the panels are associated by
level of the grouping factor. Defaults to NULL, meaning that no outer covariates are present.
inner
An optional one-sided formula, or list of one-sided

formulas, indicating covariates that are inner to the
grouping factor(s). If multiple levels of grouping are
present, this argument can be either a single onesided formula, or a list of one-sided formulas. If no
names are assigned to the list elements, they are assumed in the same order as the group levels (outermost to innermost grouping). An inner covariate can
change within the sets of rows dened by the grouping factor. An inner formula can be used to associate
points in a plot of a groupedData object. Defaults to
NULL, meaning that no inner covariates are present.
labels
An optional list of character strings giving labels for

the response and the primary covariate. The label
468
for the primary covariate is named x and that for the

response is named y. Either label can be omitted.
units
An optional list of character strings giving the units

for the response and the primary covariate. The units
string for the primary covariate is named x and that
for the response is named y. Either units string can
be omitted.
Description
An object of the groupedData class is constructed from the formula and
data by attaching the formula as an attribute of the data, along with
any of outer, inner, labels, and units that are given. If
order.groups is TRUE the grouping factor is converted to an ordered
factor with the ordering determined by FUN. Depending on the number of grouping levels and the type of primary covariate, the returned
object will be of one of three classes: nfnGroupedDatanumeric covariate, single level of nesting; nGroupedDatafactor covariate, single
level of nesting; and nmGroupedDatamultiple levels of nesting. Several modeling and plotting functions can use the formula stored with
a groupedData object to construct default plots and models.
Value
An object of one of the classes nfnGroupedData, nGroupedData, or nmGroupedData, also inheriting from classes groupedData and data.frame.
See Also
formula, gapply, gsummary, lme
Examples
Orth.new <- # create a new copy of the groupedData object
groupedData( distance ~ age | Subject,
data = as.data.frame( Orthodont ),
FUN = mean,
outer = ~ Sex,
labels = list(x = "Age",
y = "Distance from pituitary to pterygomaxillary fissure"),
units = list( x = "(yr)", y = "(mm)") )
plot( Orth.new )
# trellis plot by Subject
formula( Orth.new )
# extractor for the formula
gsummary( Orth.new )
# apply summary by Subject
fm1 <- lme( Orth.new )
# fixed and groups formulae extracted
# from object
gsummary
gsummary
469
Summarize by Groups
gsummary(object, FUN, omitGroupingFactor, form, level,

groups, invariantsOnly, ...)
Arguments
object
An object to be summarized, usually a groupedData

object or a data.frame.

functions to be applied to each variable in the frame.
The function or functions are applied only to variables in object that vary within the groups dened
by groups. Invariant variables are always summarized by group using the unique value that they assume within that group. If FUN is a single function
it will be applied to each noninvariant variable by
group to produce the summary for that variable. If
ordered, factor, or numeric. The indicated function will be applied to any non-invariant variables of
omitGroupingFactor
factor itself will be omitted from the groupwise summary, but the levels of the grouping factor will continue to be used as the row names for the data frame
that is produced by the summary. Defaults to FALSE.
FUN
form
An optional one-sided formula that denes the groups.

When this formula is given, the right-hand side is
evaluated in object, converted to a factor if necessary, and the unique levels are used to dene the
groups. Defaults to formula(object).
level
An optional positive integer giving the level of grouping to be used in an object with multiple nested
470
grouping levels. Defaults to the highest or innermost

level of grouping.
groups
An optional factor that will be used to split the rows

into groups. Defaults to getGroups(object, form,
level).
invariantsOnly
An optional logical value. When TRUE only those covariates that are invariant within each group will
be summarized. The summary value for the group
is always the unique value taken on by that covariate within the group. The columns in the summary
are of the same class as the corresponding columns
in object. By denition, the grouping factor itself
must be an invariant. When combined with omitGroupingFactor = TRUE, this option can be used to
discover is there are invariant covariates in the data
frame. Defaults to FALSE.
...
Optional additional arguments to the summary functions that are invoked on the variables by group.
Often it is helpful to specify na.rm = TRUE.
Description
Provide a summary of the variables in a data frame by groups of rows.
This is most useful with a groupedData object to examine the variables
by group.
Value
A data.frame with one row for each level of the grouping factor. The
number of columns is at most the number of columns in object.
See Also
summary, groupedData, getGroups
Examples
gsummary( Orthodont ) # default summary by Subject
## gsummary with invariantsOnly = TRUE and
## omitGroupingFactor = TRUE determines whether there
## are covariates like Sex that are invariant within
## the repeated observations on the same Subject.
gsummary( Orthodont, inv = TRUE, omit = TRUE )
intervals.lme
intervals
471
Condence Intervals on Coecients
intervals(object, level, ...)

Arguments
object
A tted model object from which parameter estimates can be extracted.
level
An optional numeric value for the interval condence

level. Defaults to 0.95.
...
Some methods for the generic may require additional

arguments.
Description
Condence intervals on the parameters associated with the model represented by object are obtained. This function is generic; method functions can be written to handle specic classes of objects. Classes which
already have methods for this function include: gls, lme, and lmList.
Value
See Also
intervals.gls, intervals.lme, intervals.lmList
intervals.lme
Condence Intervals on lme Parameters
intervals(object, level, which)

Arguments
object

level
An optional numeric value with the condence level

for the intervals. Defaults to 0.95.
472
which
An optional character string specifying the subset of

parameters for which to construct the condence intervals. Possible values are "all" for all parameters,
"var-cov" for the variancecovariance parameters
only, and "fixed" for the xed eects only. Defaults
to "all".
Description
Approximate condence intervals for the parameters in the linear mixedeects model represented by object are obtained, using a normal approximation to the distribution of the (restricted) maximum likelihood
estimators (the estimators are assumed to have a normal distribution
centered at the true parameter values and with covariance matrix equal
to the negative inverse Hessian matrix of the (restricted) log-likelihood
evaluated at the estimated parameters). Condence intervals are obtained in an unconstrained scale rst, using the normal approximation,
and, if necessary, transformed to the constrained scale. The pdNatural
parametrization is used for general positive-denite matrices.
Value
A list with components given by data frames with rows corresponding to parameters and columns lower, est., and upper representing,
respectively, lower condence limits, the estimated values, and upper
condence limits for the parameters. Possible components are:
fixed
reStruct
corStruct
varFunc
sigma
Fixed eects, only present when which is not equal

to "var-cov".
Random-eects variancecovariance parameters, only
present when which is not equal to "fixed".
Within-group correlation parameters, only present
when which is not equal to "fixed" and a correlation
structure is used in object.
Within-group variance function parameters, only present when which is not equal to "fixed" and a variance function structure is used in object.
Within-group standard deviation.
See Also
lme, print.intervals.lme, pdNatural
Examples
intervals(fm1)
intervals.lmList
intervals.lmList
473
Condence Intervals on lmList Coecients
intervals(object, level, pool)

Arguments
object

level
An optional numeric value with the condence level

for the intervals. Defaults to 0.95.
pool
An optional logical value indicating whether a pooled

estimate of the residual standard error should be
used. Default is attr(object, "pool").
Description
Condence intervals on the linear model coecients are obtained for
each lm component of object and organized into a three-dimensional
array. The rst dimension corresponding to the names of the object
components. The second dimension is given by lower, est., and upper
corresponding, respectively, to the lower condence limit, estimated
coecient, and upper condence limit. The third dimension is given
by the coecients names.
Value
A three-dimensional array with the condence intervals and estimates
for the coecients of each lm component of object.
See Also
lmList, plot.intervals.lmList
Examples
fm1 <- lmList(distance ~ age | Subject, Orthodont)
intervals(fm1)
474
Linear Mixed-Eects Models
lme
lme(fixed, data, random, correlation, weights, subset,

method, na.action, control)
Arguments
fixed
A two-sided linear formula object describing the xedeects part of the model, with the response on the left
of a operator and the terms, separated by + operators, on the right, an lmList object, or a groupedData object. The method functions lme.lmList and
lme.groupedData are documented separately.
data

named in fixed, random, correlation, weights, and
subset. By default the variables are taken from the
environment from which lme is called.
random
Optionally, any of the following: (i) a one-sided formula of the form ~x1+ +xn | g1/ /gm, with
x1+ +xn specifying the model for the random effects and g1/ /gm the grouping structure (m may
be equal to 1, in which case no / is required). The
random-eects formula will be repeated for all levels
of grouping, in the case of multiple levels of grouping;
(ii) a list of one-sided formulas of the form ~x1+ +xn
| g, with possibly dierent random-eects models for
each grouping level. The order of nesting will be assumed the same as the order of the elements in the
list; (iii) a one-sided formula of the form ~x1+ +xn,
or a pdMat object with a formula (i.e., a non-NULL
value for formula(object)), or a list of such formulas or pdMat objects. In this case, the grouping structure formula will be derived from the data used to t
the linear mixed-eects model, which should inherit
from class groupedData; (iv) a named list of formulas
or pdMat objects as in (iii), with the grouping factors as names. The order of nesting will be assumed
the same as the order of the order of the elements in
the list; (v) an reStruct object. See the documentation on pdClasses for a description of the available
pdMat classes. Defaults to a formula consisting of the
right-hand side of fixed.
lme
475
correlation
corStruct classes. Defaults to NULL, corresponding to
no within-group correlations.
weights
corresponding to homocesdatic within-group errors.
subset
An optional expression indicating the subset of the

rows of data that should be used in the t. This can
be a logical vector, or a numeric vector indicating
which observation numbers are to be included, or a
character vector of the row names to be included. All
method
A character string. If "REML" the model is t by maximizing the restricted log-likelihood. If "ML" the loglikelihood is maximized. Defaults to "REML".
na.action

causes lme to print an error message and terminate
control

lmeControl. Defaults to an empty list.
Description
This generic function ts a linear mixed-eects model in the formulation described in Laird and Ware (1982), but allowing for nested
random eects. The within-group errors are allowed to be correlated
and/or have unequal variances.
Value
An object of class lme representing the linear mixed-eects model t.
Generic functions such as print, plot and summary have methods to
show the results of the t. See lmeObject for the components of the
t. The functions resid, coef, fitted, fixef, and ranef can be used
to extract some of its components.
476
See Also
lmeControl, lme.lmList, lme.groupedData, lmeObject, lmList,
reStruct, reStruct, varFunc, pdClasses, corClasses, varClasses
Examples
fm1 <- lme(distance ~ age, data = Orthodont) # random is ~ age
lmeControl
Control Values for lme Fit
lmeControl(maxIter, msMaxIter, tolerance, niterEM, msTol,

msScale, msVerbose, returnObject, gradHess,
apVar, .relStep, natural)
Arguments
maxIter
Maximum number of iterations for the lme optimization algorithm. Default is 50.
msMaxIter
Maximum number of iterations for the ms optimization step inside the lme optimization. Default is 50.
tolerance
Tolerance for the convergence criterion in the lme

algorithm. Default is 1e-6.
niterEM
Number of iterations for the EM algorithm used to

rene the initial estimates of the random-eects variancecovariance coecients. Default is 25.
msTol
Tolerance for the convergence criterion in ms, passed

as the rel.tolerance argument to the function (see
documentation on ms). Default is 1e-7.
msScale
Scale function passed as the scale argument to the

ms function (see documentation on that function).
Default is lmeScale.
msVerbose
A logical value passed as the trace argument to

ms (see documentation on that function). Default is
FALSE.
returnObject
A logical value indicating whether the tted object

should be returned when the maximum number of
iterations is reached without convergence of the algorithm. Default is FALSE.
lmeControl
477
gradHess
A logical value indicating whether numerical gradient vectors and Hessian matrices of the log-likelihood
function should be used in the ms optimization. This
option is only available when the correlation structure (corStruct) and the variance function structure (varFunc) have no varying parameters and
the pdMat classes used in the random eects structure are pdSymm (general positive-denite), pdDiag
(diagonal), pdIdent (multiple of the identity), or
pdCompSymm (compound symmetry). Default is TRUE.
apVar
A logical value indicating whether the approximate

covariance matrix of the variancecovariance parameters should be calculated. Default is TRUE.
.relStep
Relative step for numerical derivatives calculations.

Default is .Machine$double.eps(1/3).
natural
A logical value indicating whether the pdNatural

parameterization should be used for general positivedenite matrices (pdSymm) in reStruct, when the
approximate covariance matrix of the estimators is
calculated. Default is TRUE.
Description
The values supplied in the function call replace the defaults and a list
with all possible arguments is returned. The returned list is used as
the control argument to the lme function.
Value
A list with components for each of the possible arguments.
See Also
lme, ms, lmeScale
Examples
# decrease the maximum number iterations in the ms call and
# request that information on the evolution of the ms iterations
# be printed
lmeControl(msMaxIter = 20, msVerbose = TRUE)
478
lmList
List of lm Objects with a Common Model
lmList(object, data, level, na.action, pool)

Arguments
object
Either a linear formula object of the form

y ~ x1+ +xn | g or a groupedData object. In
the formula, y represents the response, x1,...,xn
the covariates, and g the grouping factor specifying
the partitioning of the data according to which dierent lm ts should be performed. The grouping factor
g may be omitted from the formula, in which case the
grouping structure will be obtained from data, which
must inherit from class groupedData. The method
function lmList.groupedData is documented separately.
data
A data frame in which to interpret the variables named

in object.
level
An optional integer specifying the level of grouping

to be used when multiple nested levels of grouping
are present.
na.action

causes lmList to print an error message and terminate if there are any incomplete observations.
pool
An optional logical value that is preserved as an attribute of the returned value. This will be used as
the default for pool in calculations of standard deviations or standard errors for summaries.
Description
Data is partitioned according to the levels of the grouping factor g and
individual lm ts are obtained for each data partition, using the model
dened in object.
Value
A list of lm objects with as many components as the number of groups
dened by the grouping factor. Generic functions such as coef, fixef,
lme, pairs, plot, predict, ranef, summary, and update have methods
that can be applied to an lmList object.
nlme
479
See Also
lm, lme.lmList.
Examples
Extract Log-Likelihood
logLik
logLik(object, ...)
Arguments
object
Any object from which a log-likelihood, or a contribution to a log-likelihood, can be extracted.
...
Description
specic classes of objects. Classes which already have methods for this
function include: corStruct, gls, lm, lme, lmList, lmeStruct, reStruct, and
varFunc.
Value
nlme
Nonlinear Mixed-Eects Models
nlme(model, data, fixed, random, groups, start,

correlation, weights, subset, method, na.action,
naPattern, control, verbose)
480
Arguments
model
data
fixed
random
A nonlinear model formula, with the response on the

left of a operator and an expression involving parameters and covariates on the right, or an nlsList
object. If data is given, all names used in the formula
should be dened as parameters or variables in the
data frame. The method function nlme.nlsList is
documented separately.
named in model, fixed, random, correlation,
weights, subset, and naPattern. By default the
variables are taken from the environment from which
nlme is called.
A two-sided linear formula of the form
f1+ +fn ~ x1+ +xm, or a list of two-sided formulas of the form f1 ~ x1+ +xm, with possibly
dierent models for dierent parameters. The names
of the parameters, f1,...,fn, are included on the
right-hand side of model and the x1+ +xm expressions dene linear models for these parameters (when
the left-hand side of the formula contains several parameters, they all are assumed to follow the same linear model, described by the right-hand side expression). A 1 on the right-hand side of the formula(s)
indicates a single xed eects for the corresponding
parameter(s).
Optionally, any of the following: (i) a two-sided formula of the form r1+ +rn ~ x1+ +xm|g1/
/gQ, with r1,...,rn naming parameters included on
the right-hand side of model, x1+ +xm specifying
the random-eects model for these parameters and
g1/ /gQ the grouping structure (Q may be equal to
1, in which case no / is required). The random-eects
formula will be repeated for all levels of grouping, in
the case of multiple levels of grouping; (ii) a two-sided
formula of the form r1+ +rn ~ x1+ +xm, a
list of two-sided formulas of the form r1~x1+ +xm,
with possibly dierent random-eects models for different parameters, a pdMat object with a two-sided
formula, or list of two-sided formulas (i.e., a non-NULL
value for formula(random)), or a list of pdMat objects with two-sided formulas, or lists of two-sided
formulas. In this case, the grouping structure formula will be given in groups, or derived from the
nlme
481
data used to t the nonlinear mixed-eects model,

which should inherit from class groupedData; (iii) a
named list of formulas, lists of formulas, or pdMat objects as in (ii), with the grouping factors as names.
The order of nesting will be assumed the same as
the order of the order of the elements in the list;
(iv) an reStruct object. See the documentation on
pdClasses for a description of the available pdMat
classes. Defaults to fixed, resulting in all xed eects
having also random eects.
groups
An optional one-sided formula of the form ~g1 (single level of nesting) or ~g1/ /gQ (multiple levels
of nesting), specifying the partitions of the data over
which the random eects vary. g1,...,gQ must evaluate to factors in data. The order of nesting, when
multiple levels are present, is taken from left to right
(i.e., g1 is the rst level, g2 the second, etc.).
start
An optional numeric vector, or list of initial estimates

for the xed eects and random eects. If declared as
a numeric vector, it is converted internally to a list
with a single component fixed, given by the vector.
The fixed component is required, unless the model
function inherits from class selfStart, in which case
initial values will be derived from a call to nlsList.
An optional random component is used to specify initial values for the random eects and should consist
of a matrix, or a list of matrices with length equal to
the number of grouping levels. Each matrix should
have as many rows as the number of groups at the
corresponding level and as many columns as the number of random eects in that level.
correlation
corStruct classes. Defaults to NULL, corresponding to
no within-group correlations.
weights
corresponding to homoscesdatic within-group errors.
482
subset
An optional expression indicating the subset of the

rows of data that should be used in the t. This can
be a logical vector, or a numeric vector indicating
which observation numbers are to be included, or a
character vector of the row names to be included. All
method
A character string. If "REML" the model is t by maximizing the restricted log-likelihood. If "ML" the loglikelihood is maximized. Defaults to "ML".
na.action

causes nlme to print an error message and terminate
naPattern
An expression or formula object, specifying which returned values are to be regarded as missing.
control

nlmeControl. Defaults to an empty list.
verbose

Description
This generic function ts a nonlinear mixed-eects model in the formulation described in Lindstrom and Bates (1990), but allowing for nested
random eects. The within-group errors are allowed to be correlated
and/or have unequal variances.
Value
An object of class nlme representing the nonlinear mixed-eects model
t. Generic functions such as print, plot and summary have methods
to show the results of the t. See nlmeObject for the components of
the t. The functions resid, coef, fitted, fixef, and ranef can be
used to extract some of its components.
See Also
nlmeControl, nlme.nlsList, nlmeObject, nlsList, reStruct,
varFunc, pdClasses, corClasses, varClasses
Examples
## all parameters as fixed and random effects
nlmeControl
483
fm1 <- nlme(weight ~ SSlogis(Time, Asym, xmid, scal),

data = Soybean, fixed = Asym + xmid + scal ~ 1,
start = c(18, 52, 7.5))
## only Asym and xmid as random, with a diagonal covariance
fm2 <- nlme(weight ~ SSlogis(Time, Asym, xmid, scal),
data = Soybean, fixed = Asym + xmid + scal ~ 1,
random = pdDiag(Asym + xmid ~ 1),
start = c(18, 52, 7.5))
nlmeControl
Control Values for nlme Fit
nlmeControl(maxIter, pnlsMaxIter, msMaxIter, minScale,

tolerance, niterEM, pnlsTol, msTol, msScale,
returnObject, msVerbose, gradHess, apVar,
.relStep, natural)
Arguments
maxIter
Maximum number of iterations for the nlme optimization algorithm. Default is 50.
pnlsMaxIter
Maximum number of iterations for the PNLS optimization step inside the nlme optimization. Default
is 7.
msMaxIter
Maximum number of iterations for the ms optimization step inside the nlme optimization. Default is 50.
minScale
Minimum factor by which to shrink the default step

size in an attempt to decrease the sum of squares in
the PNLS step. Default 0.001.
tolerance
Tolerance for the convergence criterion in the nlme

algorithm. Default is 1e-6.
niterEM
Number of iterations for the EM algorithm used to

rene the initial estimates of the random-eects variancecovariance coecients. Default is 25.
pnlsTol
Tolerance for the convergence criterion in PNLS step.

Default is 1e-3.
msTol
Tolerance for the convergence criterion in ms, passed

as the rel.tolerance argument to the function (see
documentation on ms). Default is 1e-7.
msScale
Scale function passed as the scale argument to the

ms function (see documentation on that function).
Default is lmeScale.
484
returnObject
A logical value indicating whether the tted object

should be returned when the maximum number of
iterations is reached without convergence of the algorithm. Default is FALSE.
msVerbose
A logical value passed as the trace argument to

ms (see documentation on that function). Default is
FALSE.
gradHess
A logical value indicating whether numerical gradient vectors and Hessian matrices of the log-likelihood
function should be used in the ms optimization. This
option is only available when the correlation structure (corStruct) and the variance function structure (varFunc) have no varying parameters and
the pdMat classes used in the random eects structure are pdSymm (general positive-denite), pdDiag
(diagonal), pdIdent (multiple of the identity), or
pdCompSymm (compound symmetry). Default is TRUE.
apVar
A logical value indicating whether the approximate

covariance matrix of the variancecovariance parameters should be calculated. Default is TRUE.
.relStep
Relative step for numerical derivatives calculations.

Default is .Machine$double.eps(1/3).
natural
A logical value indicating whether the pdNatural parameterization should be used for general positivedenite matrices (pdSymm) in reStruct, when the
approximate covariance matrix of the estimators is
calculated. Default is TRUE.
Description
The values supplied in the function call replace the defaults and a list
with all possible arguments is returned. The returned list is used as
the control argument to the nlme function.
Value
A list with components for each of the possible arguments.
See Also
nlme, ms, nlmeStruct
Examples
# decrease the maximum number iterations in the ms call and
nlsList
485
# request that information on the evolution of the ms iterations

# be printed
nlmeControl(msMaxIter = 20, msVerbose = TRUE)
nlsList
List of nls Objects with a Common Model
nlsList(model, data, start, control, level, na.action,

pool)
Arguments
model
data
start
control
level
na.action
pool
Either a nonlinear model formula, with the response

on the left of a operator and an expression involving
parameters, covariates, and a grouping factor separated by the | operator on the right, or a selfStart
function. The method function nlsList.selfStart
is documented separately.
A data frame in which to interpret the variables
named in model.
An optional named list with initial values for the parameters to be estimated in model. It is passed as
the start argument to each nls call and is required
when the nonlinear function in model does not inherit
from class selfStart.
A list of control values passed as the control argument to nls. Defaults to an empty list.
An optional integer specifying the level of grouping
to be used when multiple nested levels of grouping
are present.
causes nlsList to print an error message and terminate if there are any incomplete observations.
An optional logical value that is preserved as an attribute of the returned value. This will be used as
the default for pool in calculations of standard deviations or standard errors for summaries.
Description
Data is partitioned according to the levels of the grouping factor dened
in model and individual nls ts are obtained for each data partition,
using the model dened in model.
486
Value
A list of nls objects with as many components as the number of groups
dened by the grouping factor. Generic functions such as coef, fixef,
lme, pairs, plot, predict, ranef, summary, and update have methods
that can be applied to an nlsList object.
See Also
nls, nlme.nlsList.
Examples
fm1 <- nlsList(uptake ~ SSasympOff(conc, Asym, lrc, c0),
data = CO2, start = c(Asym = 30, lrc = -4.5, c0 = 52))
fm1
pairs.lme
Pairs Plot of an lme Object
pairs(object, form, label, id, idLabels, grid, ...)

Arguments
object

form
An optional one-sided formula specifying the desired

type of plot. Any variable present in the original data
frame used to obtain object can be referenced. In
addition, object itself can be referenced in the formula using the symbol ".". Conditional expressions
on the right of a | operator can be used to dene
separate panels in a trellis display. The expression
on the right-hand side of form, and to the left of
the | operator, must evaluate to a data frame with
at least two columns. Default is coef(.) , corresponding to a pairs plot of the coecients evaluated
at the innermost level of nesting.
id
An optional numeric value, or one-sided formula. If

given as a value, it is used as a signicance level for
an outlier test based on the Mahalanobis distances of
the estimated random eects. Groups with random
eects distances greater than the 1 value percentile
of the appropriate chi-square distribution are identied in the plot using idLabels. If given as a onesided formula, its right-hand side must evaluate to a
pairs.lme
487
logical, integer, or character vector which is used to

identify points in the plot. If missing, no points are
identied.
idLabels
An optional vector, or one-sided formula. If given as a

vector, it is converted to character and used to label
the points identied according to id. If given as a
one-sided formula, its right-hand side must evaluate
to a vector which is converted to character and used
to label the identied points. Default is the innermost
grouping factor.
grid
An optional logical value indicating whether a grid

should be added to plot. Default is FALSE.
...
Optional arguments passed to the trellis plot function.
Description
Diagnostic plots for the linear mixed-eects t are obtained. The form
argument gives considerable exibility in the type of plot specication.
A conditioning expression (on the right side of a | operator) always
implies that dierent panels are used for each level of the conditioning
factor, according to a trellis display. The expression on the right-hand
side of the formula, before a | operator, must evaluate to a data frame
with at least two columns. If the data frame has two columns, a scatter
plot of the two variables is displayed (the trellis function xyplot is
used). Otherwise, if more than two columns are present, a scatter plot
matrix with pairwise scatter plots of the columns in the data frame is
displayed (the trellis function splom is used).
Value
A diagnostic trellis plot.
See Also
lme, xyplot, splom
Examples
# scatter plot of coefficients by gender, identifying
# unusual subjects
pairs(fm1, ~coef(., augFrame = T) | Sex, id = 0.1, adj = -0.5)
# scatter plot of estimated random effects
pairs(fm1, ~ranef(.))
488
plot.lme
Plot an lme Object
plot(object, form, abline, id, idLabels, idResType, grid,

...)
Arguments
object

form
An optional formula specifying the desired type of

plot. Any variable present in the original data frame
used to obtain object can be referenced. In addition,
object itself can be referenced in the formula using
the symbol ".". Conditional expressions on the right
of a | operator can be used to dene separate panels in a trellis display. Default is resid(., type =
"p") fitted(.) , corresponding to a plot of the
standardized residuals versus tted values, both evaluated at the innermost level of nesting.
abline
An optional numeric value, or numeric vector of length

two. If given as a single value, a horizontal line will
be added to the plot at that coordinate; else, if given
as a vector, its values are used as the intercept and
slope for a line added to the plot. If missing, no lines
are added to the plot.
id

given as a value, it is used as a signicance level for
a two-sided outlier test for the standardized, or normalized residuals. Observations with absolute standardized (normalized) residuals greater than the 1
value/2 quantile of the standard normal distribution
are identied in the plot using idLabels. If given as
a one-sided formula, its right-hand side must evaluate to a logical, integer, or character vector which is
used to identify observations in the plot. If missing,
no observations are identied.
idLabels
An optional vector, or one-sided formula. If given as

a vector, it is converted to character and used to label
the observations identied according to id. If given
as a one-sided formula, its right-hand side must evaluate to a vector that is converted to character and
plot.lme
489
used to label the identied observations. Default is

the innermost grouping factor.
idResType

residuals to be used in identifying outliers, when id
is a numeric value. If "pearson", the standardized
residuals (raw residuals divided by the corresponding standard errors) are used; else, if "normalized",
the normalized residuals (standardized residuals premultiplied by the inverse square-root factor of the
estimated error correlation matrix) are used. Partial
matching of arguments is used, so only the rst character needs to be provided. Defaults to "pearson".
Description
Diagnostic plots for the linear mixed-eects t are obtained. The form
argument gives considerable exibility in the type of plot specication.
A conditioning expression (on the right side of a | operator) always
implies that dierent panels are used for each level of the conditioning
factor, according to a trellis display. If form is a one-sided formula,
histograms of the variable on the right-hand side of the formula, before
a | operator, are displayed (the trellis function histogram is used).
If form is two-sided and both its left- and right-hand side variables
are numeric, scatter plots are displayed (the trellis function xyplot
is used). Finally, if form is two-sided and its left-hand side variable
is a factor, boxplots of the right-hand side variable by the levels of
the left-hand side variable are displayed (the trellis function bwplot is
used).
Value
A diagnostic trellis plot.
See Also
lme, xyplot, bwplot, histogram
Examples
# standardized residuals versus fitted values by gender
plot(fm1, resid(., type = "p") ~ fitted(.) | Sex, abline = 0)
# box-plots of residuals by Subject
plot(fm1, Subject ~ resid(.))
# observed versus fitted values by Subject
plot(fm1, distance ~ fitted(.) | Subject, abline = c(0,1))
490
plot.nfnGroupedData
Plot an nfnGroupedData Object
plot(x, outer, inner, innerGroups, xlab, ylab, strip,

aspect, panel, key, grid, ...)
Arguments
x
An object inheriting from class nfnGroupedData, representing a groupedData object with a numeric primary covariate and a single grouping level.
outer
An optional logical value or one-sided formula, indicating covariates that are outer to the grouping factor, which are used to determine the panels of the
trellis plot. If equal to TRUE, attr(object, "outer")
is used to indicate the outer covariates. An outer covariate is invariant within the sets of rows dened
by the grouping factor. Ordering of the groups is
done in such a way as to preserve adjacency of groups
with the same value of the outer variables. Defaults
to NULL, meaning that no outer covariates are to be
used.
inner
An optional logical value or one-sided formula, indicating a covariate that is inner to the grouping factor,
which is used to associate points within each panel
of the trellis plot. If equal to TRUE, attr(object,
"inner") is used to indicate the inner covariate. An
inner covariate can change within the sets of rows dened by the grouping factor. Defaults to NULL, meaning that no inner covariate is present.
innerGroups
An optional one-sided formula specifying a factor to

be used for grouping the levels of the inner covariate.
Dierent colors, or line types, are used for each level
of the innerGroups factor. Default is NULL, meaning
that no innerGroups covariate is present.
xlab, ylab
Optional character strings with the labels for the

plot. Default is the corresponding elements of attr(
object, "labels") and attr(object, "units")
pasted together.
strip
An optional function passed as the strip argument

to the xyplot function. Default is strip.default(
..., style = 1) (see trellis.args).
plot.nfnGroupedData
491
aspect
An optional character string indicating the aspect ratio for the plot passed as the aspect argument to the
xyplot function. Default is "xy" (see trellis.args).
panel
An optional function used to generate the individual panels in the trellis display, passed as the panel
argument to the xyplot function.
key
An optional logical function or function. If TRUE and

innerGroups is non-NULL, a legend for the dierent
innerGroups levels is included at the top of the plot.
If given as a function, it is passed as the key argument to the xyplot function. Default is TRUE if
innerGroups is non-NULL and FALSE otherwise.
grid

should be added to plot. Default is TRUE.
...
Optional arguments passed to the xyplot function.
Description
A trellis plot of the response versus the primary covariate is generated. If outer variables are specied, the combination of their levels
are used to determine the panels of the trellis display. Otherwise, the
levels of the grouping variable determine the panels. A scatter plot of
the response versus the primary covariate is displayed in each panel,
with observations corresponding to same inner group joined by line
segments. The trellis function xyplot is used.
Value
A trellis plot of the response versus the primary covariate.
See Also
groupedData, xyplot
Examples
# different panels per Subject
plot(Orthodont)
# different panels per gender
plot(Orthodont, outer = TRUE)
492
plot.nmGroupedData
Plot an nmGroupedData Object
plot(x, collapseLevel, displayLevel, outer, inner,

preserve, FUN, subset, grid, ...)
Arguments
x
An object inheriting from class nmGroupedData, representing a groupedData object with multiple grouping factors.
collapseLevel An optional positive integer or character string indicating the grouping level to use when collapsing the
data. Level values increase from outermost to innermost grouping. Default is the highest or innermost
level of grouping.
displayLevel
An optional positive integer or character string indicating the grouping level to use for determining the
panels in the trellis display, when outer is missing.
Default is collapseLevel.
outer
An optional logical value or one-sided formula, indicating covariates that are outer to the displayLevel
grouping factor, which are used to determine the panels of the trellis plot. If equal to TRUE, the displayLevel element attr(object, "outer") is used to
indicate the outer covariates. An outer covariate is
invariant within the sets of rows dened by the grouping factor. Ordering of the groups is done in such a
way as to preserve adjacency of groups with the same
value of the outer variables. Defaults to NULL, meaning that no outer covariates are to be used.
inner
An optional logical value or one-sided formula, indicating a covariate that is inner to the displayLevel
grouping factor, which is used to associate points
within each panel of the trellis plot. If equal to TRUE,
attr(object, "outer") is used to indicate the inner covariate. An inner covariate can change within
the sets of rows dened by the grouping factor. Defaults to NULL, meaning that no inner covariate is
present.
preserve
An optional one-sided formula indicating a covariate whose levels should be preserved when collapsing
plot.nmGroupedData
FUN
subset
grid
...
493
the data according to the collapseLevel grouping

factor. The collapsing factor is obtained by pasting
together the levels of the collapseLevel grouping
factor and the values of the covariate to be preserved.
Default is NULL, meaning that no covariates need to
be preserved.
functions to be used for collapsing the data. The
function or functions are applied only to variables
in object that vary within the groups dened by
collapseLevel. Invariant variables are always summarized by group using the unique value that they
assume within that group. If FUN is a single function it will be applied to each noninvariant variable
by group to produce the summary for that variable.
If FUN is a list of functions, the names in the list
should designate classes of variables in the data such
as ordered, factor, or numeric. The indicated function will be applied to any noninvariant variables of
An optional named list. Names can be either positive integers representing grouping levels, or names
of grouping factors. Each element in the list is a vector indicating the levels of the corresponding grouping factor to be used for plotting the data. Default is
NULL, meaning that all levels are used.
should be added to plot. Default is TRUE.
Description
The groupedData object is summarized by the values of the displayLevel grouping factor (or the combination of its values and the values
of the covariate indicated in preserve, if any is present). The collapsed
data is used to produce a new groupedData object, with grouping
factor given by the displayLevel factor, which is plotted using the
appropriate plot method for groupedData objects with single level of
grouping.
494
Value
A trellis display of the data collapsed over the values of the collapseLevel grouping factor and grouped according to the displayLevel
grouping factor.
See Also
groupedData, collapse.groupedData, plot.nfnGroupedData,
plot.nffGroupedData
Examples
# no collapsing, panels by Dog
plot(Pixel, display = "Dog", inner = ~Side)
# collapsing by Dog, preserving day
plot(Pixel, collapse = "Dog", preserve = ~day)
plot.Variogram
Plot a Variogram Object
plot(object, smooth, showModel, sigma, span, xlab, ylab,

type, ylim, ...)
Arguments
object
An object inheriting from class Variogram, consisting

of a data frame with two columns named variog and
dist, representing the semivariogram values and the
corresponding distances.
smooth
An optional logical value controlling whether a loess

smoother should be added to the plot. Defaults to
TRUE, when showModel is FALSE.
showModel
An optional logical value controlling whether the semivariogram corresponding to an "modelVariog" attribute of object, if any is present, should be added
to the plot. Defaults to TRUE, when the "modelVariog"
attribute is present.
sigma
An optional numeric value used as the height of a

horizontal line displayed in the plot. Can be used to
represent the process standard deviation. Default is
NULL, implying that no horizontal line is drawn.
span
An optional numeric value with the smoothing parameter for the loess t. Default is 0.6.
predict.lme
495
xlab,ylab
Optional character strings with the x- and y-axis labels. Default respectively to "Distance" and "Semivariogram".
type
An optional character indicating the type of plot. Defaults to "p".
ylim
An optional numeric vector with the limits for the

y-axis. Defaults to c(0, max(object$variog)).
...
Optional arguments passed to the trellis xyplot function.
Description
An xyplot of the semivariogram versus the distances is produced. If
smooth = TRUE, a loess smoother is added to the plot. If showModel
= TRUE and object includes an "modelVariog" attribute, the corresponding semivariogram is added to the plot.
Value
An xyplot trellis plot.
See Also
Variogram, xyplot, loess
Examples
fm1 <- lme(follicles ~ sin(2*pi*Time) + cos(2*pi*Time), Ovary)
plot(Variogram(fm1, form = ~ Time | Mare, maxDist = 0.7))
predict.lme
Predictions from an lme Object
predict(object, newdata, level, asList, na.action)

Arguments
object

newdata
An optional data frame to be used for obtaining the

predictions. All variables used in the xed- and
random-eects models, as well as the grouping factors, must be present in the data frame. If missing,
the tted values are returned.
496
level
An optional integer vector giving the level(s) of grouping to be used in obtaining the predictions. Level values increase from outermost to innermost grouping,
with level zero corresponding to the population predictions. Defaults to the highest or innermost level of
grouping.
asList
An optional logical value. If TRUE and a single value is

given in level, the returned object is a list with the
predictions split by groups; else the returned value
is either a vector or a data frame, according to the
length of level.
na.action

newdata contains NAs. The default action (na.fail)
causes the function to print an error message and
terminate if there are any incomplete observations.
Description
The predictions at level i are obtained by adding together the population predictions (based only on the xed-eects estimates) and the
estimated contributions of the random eects to the predictions at
grouping levels less or equal to i. The resulting values estimate the
best linear unbiased predictions (BLUPs) at level i. If group values not
included in the original grouping factors are present in newdata, the
corresponding predictions will be set to NA for levels greater or equal
to the level at which the unknown groups occur.
Value
If a single level of grouping is specied in level, the returned value is
either a list with the predictions split by groups (asList = TRUE) or
a vector with the predictions (asList = FALSE); else, when multiple
frame with columns given by the predictions at dierent levels and the
grouping factors.
See Also
lme, fitted.lme
Examples
newOrth <- data.frame(Sex = c("Male","Male","Female","Female",
"Male","Male"),
age = c(15, 20, 10, 12, 2, 4),
Subject = c("M01","M01","F30","F30","M04",
qqnorm.lme
497
"M04"))
predict(fm1, newOrth, level = 0:1)
qqnorm.lme
Normal Plot of Residuals or Random Eects from

an lme Object
qqnorm(object, form, abline, id, idLabels, grid, ...)

Arguments
object

form
An optional one-sided formula specifying the desired

type of plot. Any variable present in the original data
frame used to obtain object can be referenced. In
addition, object itself can be referenced in the formula using the symbol ".". Conditional expressions
on the right of a | operator can be used to dene
separate panels in a trellis display. The expression
on the right-hand side of form and to the left of a
| operator must evaluate to a residuals vector, or a
random eects matrix. Default is resid(., type
= "p"), corresponding to a normal plot of the standardized residuals evaluated at the innermost level of
nesting.
abline
An optional numeric value, or numeric vector of length

two. If given as a single value, a horizontal line will
be added to the plot at that coordinate; else, if given
as a vector, its values are used as the intercept and
slope for a line added to the plot. If missing, no lines
are added to the plot.
id

given as a value, it is used as a signicance level for a
two-sided outlier test for the standardized residuals
(random eects). Observations with absolute standardized residuals (random eects) greater than the
1 value/2 quantile of the standard normal distribution are identied in the plot using idLabels. If given
as a one-sided formula, its right-hand side must evaluate to a logical, integer, or character vector which is
used to identify observations in the plot. If missing,
no observations are identied.
498
idLabels
An optional vector, or one-sided formula. If given as

a vector, it is converted to character and used to label
the observations identied according to id. If given
as a one-sided formula, its right-hand side must evaluate to a vector that is converted to character and
used to label the identied observations. Default is
the innermost grouping factor.
grid

should be added to plot. Default is FALSE.
...
Description
Diagnostic plots for assessing the normality of residuals and random
eects in the linear mixed-eects t are obtained. The form argument
gives considerable exibility in the type of plot specication. A conditioning expression (on the right side of a | operator) always implies
that dierent panels are used for each level of the conditioning factor,
according to a trellis display.
Value
A diagnostic trellis plot for assessing normality of residuals or random
eects.
See Also
lme, plot.lme
Examples
# normal plot of standardized residuals by gender
qqnorm(fm1, ~ resid(., type = "p") | Sex, abline = c(0, 1))
# normal plots of random effects
qqnorm(fm1, ~ranef(.))
ranef
ranef(object, ...)
Extract Random Eects
ranef.lme
499
Arguments
object
Any tted model object from which random eects

estimates can be extracted.
...
Description
function include lmList and lme.
Value
See Also
ranef.lmList, ranef.lme
ranef.lme
Extract lme Random Eects
ranef(object, augFrame, level, data, which, FUN, standard,

omitGroupingFactor)
Arguments
object

augFrame

frame is augmented with variables dened in data;
else, if FALSE, only the coecients are returned. Defaults to FALSE.
level
An optional vector of positive integers giving the levels of grouping to be used in extracting the random
eects from an object with multiple nested grouping
levels. Defaults to all levels of grouping.
data

to t object.
500
which
An optional positive integer vector specifying which

columns of data should be used in the augmentation
of the returned data frame. Defaults to all columns
in data.
FUN

when collapsing data by groups. Group-invariant variables are always summarized by the unique value that
they assume within that group. If FUN is a single function it will be applied to each noninvariant variable
by group to produce the summary for that variable. If
ordered, factor, or numeric. The indicated function will be applied to any group-varying variables of
An optional logical value indicating whether the estimated random eects should be standardized(i.e.,
divided by the corresponding estimated standard error). Defaults to FALSE.
omitGroupingFactor
An optional logical value. When TRUE, the grouping
standard
Description
The estimated random eects at level i are represented as a data frame
with rows given by the dierent groups at that level and columns given
by the random eects. If a single level of grouping is specied, the
returned object is a data frame; else, the returned object is a list of such
data frames. Optionally, the returned data frame(s) may be augmented
with covariates summarized over groups.
Value
A data frame, or list of data frames, with the estimated random effects at the grouping level(s) specied in level and, optionally, other
ranef.lmList
501
covariates summarized over groups. The returned object inherits from

classes ranef.lme and data.frame.
See Also
lme, fixed.effects.lme, coef.lme, plot.ranef.lme, gsummary
Examples
ranef(fm1)
ranef(fm1, augFrame = TRUE)
ranef.lmList
Extract lmList Random Eects
ranef(object, augFrame, data, which, FUN, standard,

omitGroupingFactor)
Arguments
object

augFrame

frame is augmented with variables dened in the data
frame used to produce object; else, if FALSE, only
the random eects are returned. Defaults to FALSE.
data

to t object.
which
An optional positive integer or character vector specifying which columns of the data frame used to produce object should be used in the augmentation of
the returned data frame. Defaults to all variables in
the data.
FUN

when collapsing the data by groups. Group-invariant
variables are always summarized by the unique value
that they assume within that group. If FUN is a single function it will be applied to each noninvariant
variable by group to produce the summary for that
variable. If FUN is a list of functions, the names in
502
the list should designate classes of variables in the

frame such as ordered, factor, or numeric. The indicated function will be applied to any group-varying
variables of that class. The default functions to be
used are mean for numeric factors, and Mode for both
factor and ordered. The Mode function, dened internally in gsummary, returns the modal or most popular value of the variable. It is dierent from the
mode function that returns the S-language mode of
the variable.
standard
An optional logical value indicating whether the estimated random eects should be standardized (i.e.,
divided by the corresponding estimated standard error). Defaults to FALSE.
omitGroupingFactor
An optional logical value. When TRUE, the grouping
Description
A data frame containing the dierences between the coecients of the
individual lm ts and the average coecients.
Value
A data frame with the dierences between the individual lm coecients
in object and their average. Optionally, the returned data frame may
be augmented with covariates summarized over groups or the dierences may be standardized.
See Also
lmList, fixef.lmList
Examples
ranef(fm1)
ranef(fm1, standard = TRUE)
ranef(fm1, augFrame = TRUE)
residuals.lme
residuals.lme
503
Extract lme Residuals
residuals(object, level, type, asList)

Arguments
object

level
An optional integer vector giving the level(s) of grouping to be used in extracting the residuals from object.
Level values increase from outermost to innermost
grouping, with level zero corresponding to the population residuals. Defaults to the highest or innermost
level of grouping.
type

"normalized", the normalized residuals (standardized residuals premultiplied by the inverse squareroot factor of the estimated error correlation matrix)
are used. Partial matching of arguments is used, so
only the rst character needs to be provided. Defaults
to "pearson".
asList
An optional logical value. If TRUE and a single value

is given in level, the returned object is a list with
the residuals split by groups; else the returned value
is either a vector or a data frame, according to the
length of level. Defaults to FALSE.
Description
The residuals at level i are obtained by subtracting the tted levels
at that level from the response vector (and dividing by the estimated
within-group standard error, if type="pearson"). The tted values at
level i are obtained by adding together the population-tted values
(based only on the xed-eects estimates) and the estimated contributions of the random eects to the tted values at grouping levels less
or equal to i.
504
Value
If a single level of grouping is specied in level, the returned value
is either a list with the residuals split by groups (asList = TRUE)
or a vector with the residuals (asList = FALSE); else, when multiple
frame with columns given by the residuals at dierent levels and the
grouping factors.
See Also
lme, fitted.lme
Examples
residuals(fm1, level = 0:1)
selfStart
Construct Self-Starting Nonlinear Models
selfStart(model, initial, parameters, template)

Description
This function is generic; methods functions can be written to handle
specic classes of objects. Available methods include selfStart.default and selfStart.formula. See the documentation on the appropriate method function.
Value
A function object of the selfStart class.
See Also
selfStart.default, selfStart.formula
selfStart.default
505
selfStart.default

Arguments
model
A function object dening a nonlinear model.
A function object, taking three arguments: mCall,

data, and LHS, representing, respectively, a matched
call to the function model, a data frame in which to
interpret the variables in mCall, and the expression
from the left-hand side of the model formula in the
call to nls. This function should return initial values
for the parameters in model.
parameters, template
These arguments are included for consistency with
the generic function, but are not used in the default
method. See the documentation on selfStart.formula.
initial
Description
A method for the generic function selfStart for formula objects.
Value
A function object of class selfStart, corresponding to a self-starting nonlinear model function. An initial attribute (dened by the initial
argument) is added to the function to calculate starting estimates for
the parameters in the model automatically.
See Also
selfStart.formula
Examples
# first.order.log.model is a function object defining a first
# order compartment model
# first.order.log.initial is a function object which calculates
# initial values for the parameters in first.order.log.model
# self-starting first order compartment model
SSfol <- selfStart(first.order.log.model,
first.order.log.initial)
506
selfStart.formula

Arguments
model
A nonlinear formula object of the form expression.
initial
A function object, taking three arguments: mCall,

data, and LHS, representing, respectively, a matched
call to the function model, a data frame in which to
interpret the variables in mCall, and the expression
from the left-hand side of the model formula in the
call to nls. This function should return initial values
for the parameters in model.
parameters
A character vector specifying the terms on the righthand side of model for which initial estimates should
be calculated. Passed as the namevec argument to
the deriv function.
template
An optional prototype for the calling sequence of the

returned object, passed as the function.arg argument to the deriv function. By default, a template is
generated with the covariates in model coming rst
and the parameters in model coming last in the calling sequence.
Description
A method for the generic function selfStart for formula objects.
Value
A function object of class selfStart, obtained by applying deriv to the
right-hand side of the model formula. An initial attribute (dened by
the initial argument) is added to the function to calculate starting
estimates for the parameters in the model automatically.
See Also
selfStart.default, deriv
Examples
## self-starting logistic model
SSlogis <- selfStart(~ Asym/(1 + exp((xmid - x)/scal)),
Variogram
507
function(mCall, data, LHS)

{
xy <- sortedXyData(mCall[["x"]], LHS, data)
if(nrow(xy) < 4) {
stop("Too few distinct x values to fit a logistic")
}
z <- xy[["y"]]
if (min(z) <= 0) { z <- z + 0.05 * max(z) } # avoid zeroes
z <- z/(1.05 * max(z))
# scale to within unit height
xy[["z"]] <- log(z/(1 - z))
# logit transformation
aux <- coef(lm(x ~ z, xy))
parameters(xy) <- list(xmid = aux[1], scal = aux[2])
pars <- as.vector(coef(nls(y ~ 1/(1 + exp((xmid - x)/scal)),
data = xy, algorithm = "plinear")))
value <- c(pars[3], pars[1], pars[2])
names(value) <- mCall[c("Asym", "xmid", "scal")]
value
}, c("Asym", "xmid", "scal"))
Variogram
Calculate Semivariogram
Variogram(object, distance, ...)

Description
function include default, gls and lme. See the appropriate method documentation for a description of the arguments.
Value
See Also
Variogram.default,Variogram.gls, Variogram.lme,
plot.Variogram
508
Variogram.lme
Calculate Semivariogram for Residuals from an

lme Object
Variogram(object, distance, form, resType, data,

na.action, maxDist, length.out, collapse, nint,
breaks, robust, metric)
Arguments
object

distance
An optional numeric vector with the distances between residual pairs. If a grouping variable is present,
only the distances between residual pairs within the
same group should be given. If missing, the distances
are calculated based on the values of the arguments
form, data, and metric, unless object includes a
corSpatial element, in which case the associated covariate (obtained with the getCovariate method) is
used.
form
An optional one-sided formula specifying the covariate(s) to be used for calculating the distances between residual pairs and, optionally, a grouping factor for partitioning the residuals (which must appear
to the right of a | operator in form). Default is 1, implying that the observation order within the groups
is used to obtain the distances.
resType

"normalized", the normalized residuals (standardized residuals premultiplied by the inverse squareroot
factor of the estimated error correlation matrix) are
used. Partial matching of arguments is used, so only
the rst character needs to be provided. Defaults to
"pearson".
data
An optional data frame in which to interpret the variables in form. By default, the same data used to t
object is used.
na.action

Variogram.lme
509
causes an error message to be printed and the function to terminate, if there are any incomplete observations.
maxDist
An optional numeric value for the maximum distance

used for calculating the semivariogram between two
residuals. By default all residual pairs are included.
length.out
An optional integer value. When object includes a

corSpatial element, its semivariogram values are
calculated and this argument is used as the
length.out argument to the corresponding
Variogram method. Defaults to 50.
collapse

collapsing to be applied to the individual semivariogram values. If equal to "quantiles", the semivariogram values are split according to quantiles of
the distance distribution, with equal number of observations per group, with possibly varying distance
interval lengths. Else, if "fixed", the semivariogram
values are divided according to distance intervals of
equal lengths, with possibly dierent number of observations per interval. Else, if "none", no collapsing
is used and the individual semivariogram values are
returned. Defaults to "quantiles".
nint
An optional integer with the number of intervals to

be used when collapsing the semivariogram values.
Defaults to 20.
robust
An optional logical value specifying if a robust semivariogram estimator should be used when collapsing
the individual values. If TRUE the robust estimator is
used. Defaults to FALSE.
breaks
An optional numeric vector with the breakpoints for

the distance intervals to be used in collapsing the
semivariogram values. If not missing, the option specied in collapse is ignored.
metric
An optional character string specifying the distance

metric to be used. The currently available options
are "euclidean" for the root sum-of-squares of distances; "maximum" for the maximum dierence; and
"manhattan" for the sum of the absolute dierences.
Partial matching of arguments is used, so only the
rst three characters need to be provided. Defaults
to "euclidean".
510
Description
This method function calculates the semivariogram for the withingroup residuals from an lme t. The semivariogram values are calculated for pairs of residuals within the same group. If collapse is different from "none", the individual semivariogram values are collapsed
using either a robust estimator (robust = TRUE) dened in Cressie
(1993), or the average of the values within the same distance interval.
The semivariogram is useful for modeling the error term correlation
structure.
Value
A data frame with columns variog and dist representing, respectively,
the semivariogram values and the corresponding distances. If the semivariogram values are collapsed, an extra column, n.pairs, with the
number of residual pairs used in each semivariogram calculation, is included in the returned data frame. If object includes a corSpatial
element, a data frame with its corresponding semivariogram is included
in the returned value, as an attribute "modelVariog". The returned
value inherits from class Variogram.
See Also
lme, Variogram.default, Variogram.gls, plot.Variogram
Examples
fm1 <- lme(weight ~ Time * Diet, BodyWeight, ~ Time | Rat)
Variogram(fm1, form = ~ Time | Rat, nint = 10, robust = TRUE)
Appendix C
A Collection of Self-Starting Nonlinear
Regression Models
We have mentioned several self-starting nonlinear regression models in the

text. In this appendix we describe each of the self-starting models included
with the nlme library. For each model we give the model formula, a description of the parameters, and the strategy used to obtain starting estimates.
C.1 SSasympThe Asymptotic Regression Model

The asymptotic regression model is used to model a response y that approaches a horizontal asymptote as x . We write it as
y(x) = 1 + (2 1 ) exp[ exp(3 )x],
(C.1)
so that 1 is the asymptote as x and 2 is y(0). These parameters

are shown in Figure C.1. The parameter 3 is the logarithm of the rate
constant. We use the logarithm to enforce positivity of the rate constant so
the model does approach an asymptote. The corresponding half-life t0.5 =
log 2/ exp(3 ) is illustrated in Figure C.1.
C.1.1
Starting Estimates for SSasymp
Starting values for the asymptotic regression model are obtained by:
(0)
1. Using NLSstRtAsymptote to get an estimate 1

(0)
of the asymptote.
(0)
2. Regressing log(|y 1 |) on t. The estimated slope is exp(3 ).
512
Appendix C. A Collection of Self-Starting Nonlinear RegressionModels
t0.5
1
FIGURE C.1. The asymptotic regression model showing the parameters 1 , the
asymptotic response as x , 2 , the response at x = 0, and t0.5 , the half-life.
3. Using an algorithm for partially linear models (Bates and Chambers,

1992, 10.2.5) to rene estimates of 1 , 2 , and 3 in
y(x) = 1 + (2 1 ) exp[exp(3 )x].
Because 1 and 2 occur linearly in the model expression, the least
squares t iterates over a single parameter.
These estimates are the nal nonlinear regression estimates.
C.2 SSasympOffAsymptotic Regression with an

Oset
This is an alternative form of the asymptotic regression model that provides
a more stable parameterization for the CO2 data. It is written
y(x) = 1 {1 exp[ exp(2 ) (x 3 )]}.
(C.2)
As in SSasymp, 1 is the asymptote as x . In this formulation 2 is

the logarithm of the rate constant, corresponding to a half-life of t0.5 =
log 2/ exp(2 ), and 3 is the value of x at which y = 0. The parameters 1 ,
t0.5 , and 3 are shown in Figure C.2.
C.2.1
Starting Estimates for SSasympOff
First we t SSasymp then we transform the parameters to the formulation

used in SSasympOff. If omega is the vector of parameters from SSasymp and
C.3 SSasympOrigAsymptotic Regression Through the Origin
513
3
1
t0.5
x
FIGURE C.2. The asymptotic regression model with an oset showing the parameters 1 , the asymptote as x , t0.5 , the half-life, and 3 , the value of x
for which y = 0.
is the vector of parameters for SSasympOff, the correspondence is

1 = 1 ,
2 = 3 ,
3 = exp(3 ) log[(2 1 )/1 ].
C.3 SSasympOrigAsymptotic Regression

Through the Origin
This form of the asymptotic regression model is constrained to pass through
the origin. It is called the BOD model in Bates and Watts (1988) where it is
used to model Biochemical Oxygen Demand curves. The model is written
y(x) = 1 [1 exp( exp(2 )x].
(C.3)
As in SSasympOff, 1 is the asymptote as x and 2 is the logarithm of

the rate constant, corresponding to a half-life of t0.5 = log 2/ exp(2 ). The
parameters 1 and t0.5 are shown in Figure C.3.
C.3.1
Starting Estimates for SSasympOrig
Starting values for this regression model are obtained by:
514
t0.5
1
x
FIGURE C.3. The asymptotic regression model through the origin showing the
parameters 1 , the asymptote as x and t0.5 , the half-life.
(0)
1. Using NLSstRtAsymptote to get an estimate 1
of the asymptote.
2. Obtaining an initial estimate of 2 as

(0)
2 = log abs
n

"
(0)
log(1 yi /1 )/xi /n.
i=1
3. Using an algorithm for partially linear models to rene the estimates

of 1 and 2 . Because 1 occurs linearly in the model expression, the
least squares t iterates over a single parameter.
C.4 SSbiexpBiexponential Model

The biexponential model is a linear combination of two negative exponential terms
y(x) = 1 exp [ exp(2 )x] + 3 exp [ exp(4 )x] .
(C.4)
The parameters 1 and 3 are the coecients of the linear combination,

and the parameters 2 and 4 are the logarithms of the rate constants.
The two sets of parameters (1 , 2 ) and (3 , 4 ) are exchangeable, meaning
that the values of the pairs can be exchanged without changing the value of
y(x). We create an identiable parameterization by requiring that 2 > 4 .
A representative biexponential model, along with its constituent exponential curves, is shown in Figure C.4.
C.4 SSbiexpBiexponential Model
515
y
5
x
1
FIGURE C.4. A biexponential model showing the linear combination of the exponentials (solid line) and its constituent exponential curves (dashed line and
dotted line). The dashed line is 3.5 exp(4x) and the dotted line is 1.5 exp(x).
C.4.1
Starting Estimates for SSbiexp
The starting estimates for the biexponential model are determined by curve
peeling, which involves:
1. Choosing half the data with the largest x values and tting the simple
linear regression model
log abs(y) = a + bx.
(0)
(0)
2. Setting 3 = exp a and 4 = log abs(b) and calculating the residu(0)

(0)
als ri = yi 3 exp[ exp(4 )xi ] for the half of the data with the
smallest x values. Fit the simple linear regression model
log abs(r) = a + bx.
(0)
3. Setting 2 = log abs(b) and using an algorithm for partially linear

models to rene the estimates of 1 , 2 , 3 , and 4 . Because the
model is linear in 1 and 3 , the only starting estimates used in this
step are those for 2 and 4 and the iterations are with respect to
these two parameters.
The estimates obtained this way are the nal nonlinear regression estimates.
516
y
5
x
1
FIGURE C.5. A sample response curve from a rst-order open-compartment

model. The parameters correspond to an elimination rate constant of 1, an absorption rate constant of 3, and a clearance of 0.1. The dose is 1.
C.5 SSfolFirst-Order Compartment Model

This model is derived from a compartment model in pharmacokinetics describing the concentration of a drug in the serum following a single oral
dose. The model is based on rst-order kinetics for the absorption of the
drug from the digestive system and for the elimination of the drug from
the circulatory system. Because the drug is eliminated from the circulatory system, the system of compartments is called an open system, and the
model is a rst-order open compartment model. It is written
y(x) =
D exp(1 ) exp(2 )
{exp [ exp(1 )x] exp [ exp(2 )x]} ,
exp(3 ) [exp(2 ) exp(1 )]
(C.5)
where D is the dose, 1 is the logarithm of the elimination rate constant,

2 is the logarithm of the absorption rate constant, and 3 is the logarithm
of the clearance.
A sample response curve from a rst-order open compartment model is
shown in Figure C.5
C.5.1
Starting Estimates for SSfol
The starting estimates for the SSfol model are also determined by curve
peeling. The steps are:
C.6 SSfplFour-Parameter Logistic Model
517
1. Determine the position of the maximum response. Fit the simple

linear regression model
log(y) = a + bx
to the data with x values greater than or equal to the position of the
(0)
(0)
(0)
maximum response. Set 1 = log abs(b) and 2 = 1 + 1.
2. Use an algorithm for partially linear models to t the nonlinear regression model
y(x) = k{exp[ exp(1 )x] exp[ exp(2 )x]}
rening the estimates of 1 and 2 .
3. Use the current estimates of 1 and 2 and an algorithm for partially
linear models to t
y(x) = kD
exp[ exp(1 )x] exp[exp(2 )x)]

.
exp(1 ) exp(2 )
Set 3 = 1 + 2 log k.
C.6 SSfplFour-Parameter Logistic Model

The four-parameter logistic model relates a response y to an input x via a
sigmoidal or S-shaped function. We write it as
y(x) = 1 +
2 1
.
1 + exp [(3 x) /4 ]
(C.6)
We require that 4 > 0 so the parameters are:

1 the horizontal asymptote as x
2 the horizontal asymptote as x
3 the x value at the inection point. At this value of x the response
is midway between the asymptotes.
4 a scale parameter on the x-axis. When x = 3 + 4 the response
is 1 + (2 1 )/(1 + e1 ) or roughly three-quarters of the distance
from 1 to 2 .
These parameters are shown in Figure C.6
518
3
4
FIGURE C.6. The four-parameter logistic model. The parameters are the horizontal asymptote 1 as x , the horizontal asymptote 2 as x , the x
value at the inection point (3 ), and a scale parameter 4 .
C.6.1
Starting Estimates for SSfpl
The steps in determining starting estimates for the SSfpl model are:
(0)
1. Use NLSstClosestX to determine 3 as the x value corresponding a

response at the midpoint of the range of the responses.
2. Use an algorithm for partially linear models to t A, B, and while
holding 3 xed in the nonlinear regression model
y(x) = A +
B
.
1 + exp[(3 x)/ exp ]
The purpose of this t is to rene the estimate of , the logarithm of

the scale parameter 4 . We start at zero.
3. Use the rened estimate of and an algorithm for partially linear
models to t
y(x) = A +
B
1 + exp[(3 x)/ exp ]
with respect to A, B, 3 and . The estimates are then 1 = A,

2 = A + B, 4 = exp and 3 .
C.7 SSlogisSimple Logistic Model
519
x
FIGURE C.7. The simple logistic model showing the parameters 1 , the horizontal asymptote as x , 2 , the value of x for which y = 1 /2, and 3 , a scale
parameter on the x-axis. If 3 < 0 the curve will be monotone decreasing instead
of monotone increasing and 1 will be the horizontal asymptote as x .
C.7 SSlogisSimple Logistic Model

The simple logistic model is a special case of the four-parameter logistic
model in which one of the horizontal asymptotes is zero. We write it as
y(x) =
1
.
1 + exp [(2 x)/3 ]
(C.7)
For this model we do not require that the scale parameter 3 be positive.
If 3 > 0 then 1 is the horizontal asymptote as x and 0 is the
horizontal asymptote as x . If 3 < 0, these roles are reversed. The
parameter 2 is the x value at which the response is 1 /2. It is the inection
point of the curve. The scale parameter 3 represents the distance on the
x-axis
this inection point and the point where the response is
between
1 / 1 + e1 0.731 . These parameters are shown in Figure C.7.
C.7.1
Starting Estimates for SSlogis
The starting estimates are determined by:

1. Scaling and, if necessary, shifting the responses y so the transformed
responses y are strictly within the interval (0, 1).
2. Taking the logistic transformation
z = log[y /(1 y )]
520
x
FIGURE C.8. The MichaelisMenten model used in enzyme kinetics. The parameters are 1 , the horizontal asymptote as x and 2 , the value of x at
which the response is 1 /2.
and tting the simple linear regression model

x = a + bz.
(0)
(0)
3. Use 2 = a and 3 = b and an algorithm for partially linear models

to t
y=
1
.
1 + exp[(2 x)/3 ]
The resulting estimates are the nal nonlinear regression estimates.
C.8 SSmicmenMichaelisMenten Model

The MichaelisMenten model is used in enzyme kinetics to relate the initial
rate of an enzymatic reaction to the concentration of the substrate. It is
written
y(x) =
1 x
,
2 + x
(C.8)
where 1 is the horizontal asymptote as x and 2 , the Michaelis

parameter, is the value of x at which the response is 1 /2.
These parameters are shown in Figure C.8
C.8 SSmicmenMichaelisMenten Model
C.8.1
521
Starting Estimates for SSmicmen
The starting estimates are obtained by:

1. Fitting a simple linear regression model
1
1
=a+b
y
x
for the inverse response as a function of the inverse of x.
(0)
2. Setting 2 = abs(b/a) and using an algorithm for partially linear

models to t
y=
1 x
.
2 + x
The resulting estimates are the nal nonlinear regression estimates.
Index
ACF, see autocorrelation function

ACF S function, 241, 257, 397, 451
lme objects, 452
AIC, 10, 83
Akaike Information Criterion, see AIC
Alfalfa, see datasets
alternating algorithm, see Lindstrom
and Bates algorithm
analysis of covariance model, see
ANCOVA
ANCOVA, 3040
anova, 12, 24, 453
with single argument, 90
apparent volume of distribution, 379
AR, see autoregressive model
AR(1), 229
ARMA, see autoregressivemoving
average model
Assay, see datasets
asymptotic regression model, 301, 511
through the origin, 513
with an oset, 368, 512
augPred, 39, 361
autocorrelation function, 227
autoregressive model, 228
autoregressivemoving average model,
230
autoregressive-moving average model,

228
Bayesian Information Criterion, see
BIC
best linear unbiased predictor, see
predictions
BIC, 10, 83
biexponential model, 278, 514
block-diagonal matrix, 162
BLUP, see predictions
BodyWeight, see datasets
CAR(1), see continuous AR(1)
Cefamandole, see datasets
clearance, 295, 351
CO2, see datasets
coef
lmList objects, 457
lme objects, 455
compartment model
rst-order open-compartment, 351,
516
one-compartment, 295
one-compartment open with rstorder absorption, 378
two-compartment, 278
524
Index
compound symmetry, 161, 227

condence intervals, 92
correlation coecient, 93
xed eects, 92
standard deviation, 93
variancecovariance components, 93
continuous AR(1), 229
contrasts, 14
Helmert, 14
polynomial, 46
treatment, 16
correlation structures, 205, 226
AR, 228
ARMA, 228
compound symmetry, 227
general, 228
MA, 229
spatial, 230
with gls, 251
with lme, 239
with nlme, 395
corStruct, 232249
classes, 234
corAR1, 235, 397
corARMA, 236, 397
corCAR1, 236
corCompSymm, 233
corExp, 238
corRatio, 263
corSpher, 262
corSymm, 234
spatial correlation classes, 237
datasets
Alfalfa, 55, 425
Assay, 163, 425
BodyWeight, 105, 221, 244, 427
Cefamandole, 302, 427
CO2, 111, 130, 368, 428
Dialyzer, 214, 255, 401, 429
DNase, 131, 410, 429
Earthquake, 303, 430
ergoStool, 12, 431
Glucose2, 412, 432
IGF, 144, 155, 433
Indometh, 277, 433
Loblolly, 301, 434
Machines, 21, 435
Oats, 45, 160, 435
Orange, 338, 356, 436

Orthodont, 30, 147, 250, 436
Ovary, 239, 395, 437
Oxboys, 53, 100, 437
Oxide, 167, 437
PBG, 108, 411, 438
PBIB, 53, 439
Phenobarb, 294, 440
Pixel, 40, 440
Quinidine, 123, 378, 393, 441
Rail, 4, 443
Soybean, 287, 443
Spruce, 112, 444
Theoph, 120, 350, 363, 392, 413,
444
Wafer, 116, 170, 386, 448
Wheat2, 260, 448
deriv, 339
design plot, 13
diagnostic plots
lme objects, 11
nlme objects, 361
random eects, 188196
within-group error, 174187
Dialyzer, see datasets
DNase, see datasets
dotplot, 372
Earthquake, see datasets
EM algorithm, 79
empirical model, 274
ergoStool, see datasets
extended nonlinear regression model,
332
rst-order open-compartment model,
351, 516
fitted, 458
xed eects, 58
degrees-of-freedom, 91
hypothesis tests, 87
fixef, 459
four-parameter logistic model, 410,
517
gapply, 120, 460
GaussNewton algorithm, 325, 331
GaussSeidel algorithm, 334
generalized least squares, 201
Index
generalized nonlinear least squares,
333
getGroups, 100, 461
getInitial, 345
gls, 205, 249267, 462
methods, 250
with correlation structures, 251
with variance functions, 251
GLS model, 203205
Glucose2, see datasets
gnls, 332, 401409, 464
methods, 402
GNLS model, 333
approximate distributions of
estimates, 335
gradient attribute, 339
grouped data, 97
balanced, 99
groupedData
balancedGrouped, 109
constructor, 101, 108
display formula, 98
inner factor, 107
outer factor, 104
groupedData, 466
growth curve data, 30
gsummary, 106, 121, 469
heteroscedasticity, 178, 201, 291
IGF, see datasets
Indometh, see datasets
information matrix, 82, 323
initial attribute, 344
intervals, 471
gnls objects, 403
lmList objects, 142, 473
lme objects, 156, 471
nlme objects, 363
nlsList objects, 281, 350
isotropic correlation, 226, 231
Laird-Ware model, see linear mixedeects model
likelihood
components of, 71
extended LME model, 203
extended NLME model
525
adaptive Gaussian approximation, 332

Laplacian approximation, 331
LME approximation, 331
GLS model, 204
multilevel LME model, 77
multilevel NLME model
optimization, 79
proled, 65
pseudo, 207
QR decompositions with, 68
restricted, 75
single-level LME model, 62
single-level NLME model
likelihood ratio tests, 83
Lindstrom and Bates algorithm, 313,
330
LME step, 313
PNLS step, 313
Lindstrom and Bates model, see nonlinear mixed-eects model
linear mixed-eects model
extended, 202203
multilevel, 40, 60
single-level, 28, 58
lm, 5, 134138
lme, 8, 146174, 474
fixed and random, 146
maximum likelihood estimation, 150
methods, 147
multilevel, 167174
coecients, 170
predictions, 174
REML estimation, 150
single-level, 146166
split-plot analysis, 160
with lmList object, 147
LME model, see linear mixed-eects
model
526
Index
lmeControl, 476
lmList, 32, 139146, 478
methods, 140
Loblolly, see datasets
logistic model, 274, 338, 519
logLik, 479
MA, see moving average model
Machines, see datasets
Manhattan distance, 230
matrix logarithm, 78
maximum likelihood estimators
approximate distribution, 81
LME model, 66
mechanistic model, 274
MichaelisMenten model, 520
MLE, see maximum likelihood
estimators
model
asymptotic regression, 301, 511
with an oset, 368
asymptotic regression through the
origin, 513
asymptotic regression with an
oset, 512
biexponential, 278, 514
empirical, 274
rst-order open-compartment, 351,
516
four-parameter logistic, 410, 517
logistic, 274, 289, 338
logistic regression, 519
mechanistic, 274
MichaelisMenten, 520
one-compartment, 295
one-compartment open with rstorder absorption, 378
two-compartment, 278
moving average model, 229
multilevel model
likelihood, 77
linear mixed-eects, 40, 60
lme t, 167
nonlinear mixed-eects, 309
naPattern argument to nlme, 298,
380
NewtonRaphson algorithm, 79
nlme library
obtaining, viii
nlme, 283, 479
fixed and random, 355
covariate modeling, 365385
extended NLME model, 391
maximum likelihood estimation, 358
methods, 357
multilevel, 385391
REML estimation, 387
with nlsList object, 357
NLME model, see nonlinear mixedeects model
nlmeControl, 483
nls, 279, 338342
nlsList, 280, 347354, 485
methods, 349
nonidentiability, 204
nonlinear least squares, 278
nonlinear mixed-eects model, 282
approximate distributions of
estimates, 322
Bayesian hierarchical, 311
compared to LME model, 273277
extended, 328332
likelihood estimation, 312
multilevel, 309310
nonparametric maximum likelihood,
311
nonlinear regression model, 278
NONMEM software, 310
normal plot
of random eects, 188
of residuals, 179, 180
nugget eect, 231
Oats, see datasets
one-compartment open model with
rst-order absorption, 378
one-way ANOVA, 411
xed-eects model, 6
random-eects model, 7
Orange, see datasets
Orthodont, see datasets
orthogonal-triangular decomposition,
see QR decomposition
Index
Ovary, see datasets
Oxboys, see datasets
Oxide, see datasets
pairs
lmList objects, 141
lme objects, 188, 190, 486
nlme objects, 359
partially linear models, 342
PBG, see datasets
PBIB, see datasets
pdMat, 157
classes, 158
pdBlocked, 162
pdCompSymm, 161
pdDiag, 158, 283, 364
pdIdent, 164
peeling, 278
penalized nonlinear least squares, 313
Phenobarb, see datasets
Pixel, see datasets
plot
Variogram objects, 494
gnls objects, 404
groupedData object, 492
groupedData objects, 105, 490
lm objects, 135
nls objects, 341
nlsList objects, 350
positive-denite matrix, see variance
covariance
precision factor, 313
predictions
augmented, 39, 361
BLUP, 37, 71, 94
multilevel model, 174
NLME model, 323
random eects, 37
response, 37, 94
pseudo-likelihood, 207
qqnorm
gls objects, 253
lme objects, 179, 180, 497
nlme objects, 361
random eects, 188
QR decomposition, 66, 326
527
QuasiNewton algorithm, 79
Quinidine, see datasets
Rail, see datasets
random eects
crossed, 163
multilevel, 60
overparameterization, 156
single-level, 58
randomized block design, 1221
ranef, 498
lmList objects, 501
lme objects, 499
rate constant, 278, 351, 379
relative precision factor, 59
parameterization for, 78
REML, see restricted maximum
likelihood
residuals
normalized, 239
Pearson, 149
response, 149
residuals S function, 503
restricted maximum likelihood
LME model, 75
NLME model, 314
SBC, see BIC
scatter-plot matrix, 359
Schwarzs Bayesian Criterion, see BIC
self-starting models, 342347
available in nlme library, 346
SSasymp, 301, 511
SSasympOff, 369, 512
SSasympOrig, 513
SSbiexp, 279, 514
SSfol, 352, 516
SSfpl, 410, 517
SSlogis, 288, 347, 519
SSmicmen, 520
selfStart, 343
constructor, 504, 505
formula objects, 506
functions, 346
semivariogram, 230
robust, 231
serial correlation, 226230
shrinkage estimates, 152
Soybean, see datasets
528
Index
spatial correlation, 230232

exponential, 232
Gaussian, 232
linear, 232
rational quadratic, 232
spherical, 232
spatial data, 226
split-plot, 4552
Spruce, see datasets
starting estimates, 340
Theoph, see datasets
time-series data, 226
time-varying covariate, 378
trellis
aspect ratio, 111
display of grouped data, 110
display of multilevel data, 116
panel function, 114
plot layout, 110
two-stage model, 309, 333
unbalanced data, 25
varFunc, 208225
classes, 208
varComb, 213
varConstPower, 212, 220, 392
varExp, 211
varFixed, 208
varIdent, 209
varPower, 210, 217, 290
varReg, 268
variance covariate, 206
variance functions, 206225
with gls, 251
with lme, 214
with nlme, 391
variance weights, 208
variancecovariance
components, 93
of random eects, 58
of response, 66
of within-group error, 202
pdMat classes, 157
Variogram, 245, 264, 507
lme objects, 508
varWeights, 208
volume of distribution, 295
Wafer, see datasets
Wheat2, see datasets
within-group error
assumptions, 58
correlation, 202
heteroscedasticity, 202

Mixed Effects Models in S and S-Plus PDF

Uploaded by

Copyright:

Available Formats

Mixed Effects Models in S and S-Plus PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Mixed Effects Models in S and S-Plus PDF

Uploaded by

Copyright:

Available Formats

To Elisa and Laura

To Mary Ellen, Barbara, and Michael

nesses of mixed-eects models, we emphasize the applied aspects of these.

The default settings are for 80 and 7, respectively.

Linear Mixed-Eects Models

1 Linear Mixed-Eects Models

2 Theory and Computational Methods for LME Models

3 Describing the Structure of Grouped Data

4 Fitting Linear Mixed-Eects Models

4.3.1 Assessing Assumptions

5 Extending the Basic Linear Mixed-Eects Model

Nonlinear Mixed-Eects Models

6 NLME Models: Basic Concepts and Motivating

7 Theory and Computational Methods for NLME

7.4.1 General model formulation . . . . . . .

8 Fitting Nonlinear Mixed-Eects Models

A.18 OvaryCounts of Ovarian Follicles . . . . . . . . . . .

C A Collection of Self-Starting Nonlinear Regression

Many common statistical models can be expressed as linear models that

1. Linear Mixed-Eects Models

1.1 A Simple Example of Random Eects

1.1 A Simple Example of Random Eects

As would be expected, the structure of the data is quite simpleeach row

As is typical with S, we do not produce output directly from the tting

1. Linear Mixed-Eects Models

A -1 is used in the model formula to prevent the default inclusion of an

1.1 A Simple Example of Random Eects

1. Linear Mixed-Eects Models

To complete the statistical model, we must specify the distribution of

This model may be modied if it does not seem appropriate. As described

1.1.1 Fitting the Random-Eects Model With lme

1.1 A Simple Example of Random Eects

Many variations in the specications of linear mixed-eects models for

the possible arguments and their forms are given in Appendix B.

corresponding to a log-restricted-likelihood of 61.089. The estimated mean

1. Linear Mixed-Eects Models

and the estimated within-rail standard deviation

1.1 A Simple Example of Random Eects

Notice that the ML estimate of is 4.0208, the same as the REML

1.1.2 Assessing the Fitted Model

# produces Figure 1.4

1. Linear Mixed-Eects Models

We can see that there is considerable imprecision in the estimates of all

1.2 A Randomized Block Design

1.2 A Randomized Block Design

Effort required to arise (Borg scale)

# produces Figure 1.6

1. Linear Mixed-Eects Models

1.2.1 Choosing Contrasts for Fixed-Eects Terms

This form of xed-eects matrix X i is sometimes called the cell means

1.2 A Randomized Block Design

> contrasts( ergoStool$Type )

Using the Helmert contrasts shown above, the components of represent:

1. Linear Mixed-Eects Models

Standardized Within-Group Residuals:

By convention, the coecient corresponding to the rst column in the

On some occasions we may want to switch to other contrasts that provide

1.2 A Randomized Block Design

> fm2Stool <+