SIXTH EDITION

Applied Multivariate Statistical Analysis

RICHARD A. JOHNSON
University of WisconsinMadison

DEAN W. WICHERN
Texas A&M University

Pearson Education International

Contents

PREFACE 1 ASPECTS OF MULTIVARIATE ANALYSIS

xv 1

1.1 1.2 1.3

Introduction 1 Applications of Multivariate Techniques 3 The Organization of Data 5

Arrays, 5 Descriptive Statistics, 6 Graphical Techniques, 11

1.4

Data Displays and Pictorial Representations 19

Linking Multiple Two-Dimensional Scatter Plots, 20 Graphs of Growth Curves, 24 Stars, 26 Chernoff Faces, 27

1.5 1.6

Distance 30 Final Comments 37 Exercises 37 References 47

49

MATRIX ALGEBRA AND RANDOM VECTORS

2.1 2.2 2.3 2.4 2.5 2.6

Introduction 49 Some Basics of Matrix and Vector Algebra 49 Vectors, 49 Matrices, 54 Positive Definite Matrices 60 ** A Square-Root Matrix 65 Random Vectors and Matrices 66 Mean Vectors and Covariance Matrices 68
Partitioning the Covariance Matrix, 73 The Mean Vector and Covariance Matrix for Linear Combinations of Random Variables, 75 Partitioning the Sample Mean Vector and Covariance Matrix, 77 Matrix Inequalities and Maximization 78 vii

2.7

viii

Contents

Supplement 2A: Vectors and Matrices: Basic Concepts Vectors, 82 Matrices, 87 Exercises 103 References 110 3 SAMPLE GEOMETRY AND RANDOM SAMPLING 3.1 3.2 3.3 3.4

82

111

3.5 3.6

Introduction 111 The Geometry of the Sample 111 Random Samples and the Expected Values of the Sample Mean and Covariance Matrix 119 Generalized Variance 123 Situations in which the Generalized Sample Variance Is Zero, 129 Generalized Variance Determined by\R\ and Its Geometrical Interpretation, 134 Another Generalization of Variance, 137 Sample Mean, Covariance, and Correlation As Matrix Operations 137 Sample Values of Linear Combinations of Variables 140 Exercises 144 References 148 149

THE MULTIVARIATE NORMAL DISTRIBUTION 4.1 4.2 Introduction 149 The Multivariate Normal Density and Its Properties 149 Additional Properties of the Multivariate Normal Distribution, 156 Sampling from a Multivariate Normal Distribution and Maximum Likelihood Estimation 168 The Multivariate Normal Likelihood, 168 Maximum Likelihood Estimation offi and X, 170 Sufficient Statistics, 173 The Sampling Distribution of X and S 173 Properties of the Wishart Distribution, 174 Large-Sample Behavior of X and S 175 Assessing the Assumption of Normality 177 Evaluating the Normality of the Univariate Marginal Distributions, 177 Evaluating Bivariate Normality, 182 Detecting Outliers and Cleaning Data 187 Steps for Detecting Outliers, 189 Transformations to Near Normality 192 Transforming Multivariate Observations, 195 Exercises 200 References 208

4.3

4.4 4.5 4.6

4.7 4.8

Contents 5 INFERENCES ABOUT A MEAN VECTOR 5.1 5.2 5.3 5.4 Introduction 210 The Plausibility of /JL0 as a Value for a Normal Population Mean 210 Hotelling's T2 and Likelihood Ratio Tests General Likelihood Ratio Method, 219 216

ix 210

Confidence Regions and Simultaneous Comparisons of Component Means 220 Simultaneous Confidence Statements, 223 A Comparison of Simultaneous Confidence Intervals with One-at-a-Time Intervals, 229 The Bonferroni Method of Multiple Comparisons, 232 Large Sample Inferences about a Population Mean Vector 234 Multivariate Quality Control Charts 239 Charts for Monitoring a Sample of Individual Multivariate Observations for Stability, 241 Control Regions for Future Individual Observations, 247 Control Ellipse for Future Observations, 248 T2-Chart for Future Observations, 248 Control Charts Based on Subsample Means, 249 Control Regions for Future Subsample Observations, 251 Inferences about Mean Vectors when Some Observations Are Missing 251

5.5 5.6

5.7 5.8

Difficulties Due to Time Dependence in Multivariate Observations 256 Supplement 5A: Simultaneous Confidence Intervals and Ellipses as Shadows of the p-Dimensional Ellipsoids 258 Exercises 261 References 272 273

COMPARISONS OF SEVERAL MULTIVARIATE MEANS 6.1 6.2 Introduction 273 Paired Comparisons and a Repeated Measures Design 273 Paired Comparisons, 273 A Repeated Measures Design for CompaTing Treatments, 279 Comparing Mean Vectors from Two Populations 284 Assumptions Concerning the Structure of the Data, 284 Further Assumptions When n\ and n2Are Small, 285 Simultaneous Confidence Intervals, 288 The Two-Sample Situation When 2) = X2, 291 An Approximation to the Distribution of T2 for Normal Populations When Sample Sizes Are Not Large, 294 Comparing Several Multivariate Population Means (One-Way Manova) 296 Assumptions about the Structure of the Data for One-Way MANOVA, 296

6.3

6.4

Contents

6.5 6.6 6.7

6.8 6.9 6.10

A Summary of Univariate ANOVA, 297 Multivariate Analysis of Variance (MANOVA), 301 Simultaneous Confidence Intervals for Treatment Effects 308 Testing for Equality of Covariance Matrices 310 Two-Way Multivariate Analysis of Variance 312 Univariate Two-Way Fixed-Effects Model with Interaction, 312 Multivariate Two-Way Fixed-Effects Model with Interaction, 315 Profile Analysis 323 Repeated Measures Designs and Growth Curves 328 Perspectives and a Strategy for Analyzing Multivariate Models 332 Exercises 337 References 358

MULTIVARIATE LINEAR REGRESSION MODELS

360

7.1 7.2 7.3

Introduction 360 The Classical Linear Regression Model 360 Least Squares Estimation 364
Sum-of-Squares Decomposition, 366 Geometry of Least Squares, 367 Sampling Properties of Classical Least Squares Estimators, 369 Inferences About the Regression Model 370 Inferences Concerning the Regression Parameters, 370 Likelihood Ratio Tests for the Regression Parameters, 374 Inferences from the Estimated Regression Function 378 Estimating the Regression Function at z0, 378 Forecasting a New Observation at z0,379 Model Checking and Other Aspects of Regression 381 Does the Model Fit?, 381 Leverage and Influence, 384 Additional Problems in Linear Regression, 384 Multivariate Multiple Regression 387 Likelihood Ratio Tests for Regression Parameters, 395 Other Multivariate Test Statistics, 398 Predictions from Multivariate Multiple Regressions, 399 The Concept of Linear Regression 4Q1 Prediction of Several Variables, 406 Partial Correlation Coefficient, 409 Comparing the Two Formulations of the Regression Model 410 Mean Corrected Form of the Regression Model, 410 Relating the Formulations, 412

7.4

7.5

7.6

7.7

7.8

7.9

7.10

Multiple Regression Models with Time Dependent Errors 413 Supplement 7A: The Distribution of the Likelihood Ratio for the Multivariate Multiple Regression Model 418 Exercises 420 References 428

Contents 8 PRINCIPAL COMPONENTS

xi 430

8.1 8.2

Introduction 430 Population Principal Components 430

Principal Components Obtained from Standardized Variables, 436 Principal Components for Covariance Matrices with Special Structures, 439 Summarizing Sample Variation by Principal Components 441 The Number of Principal Components, 444Interpretation of the Sample Principal Components, 448 Standardizing the Sample Principal Components, 449 Graphing the Principal Components 454 Large Sample Inferences^ 456 Large Sample Properties o/A/ and e,-, 456 Testing for the Equal Correlation Structure, 457 Monitoring Quality with Principal Components 459 Checking.a Given Set of Measurements for Stability, 459 Controlling Future Values, 463

8.3

8.4 8.5

8.6

Supplement 8A: The Geometry of the Sample Principal Component Approximation 466
The p-Dimensional Geometrical Interpretation, 468 The n-Dimensional Geometrical Interpretation, 469

Exercises 470 References 480

9 FACTOR ANALYSIS AND INFERENCE FOR STRUCTURED COVARIANCE MATRICES

481

9.1 9.2 9.3

Introduction 481 The Orthogonal Factor Model 482 Methods of Estimation 488
The Principal Component (and Principal Factor) Method, 488 A Modified Approachthe Principal Factor Solution, 494 The Maximum Likelihood Method, 495 A Large Sample Test for the Number of Common Factors, 501 Factor Rotation 504 . Oblique Rotations, 512 Factor Scores 513 The Weighted Least Squares Method, 514 The Regression Method, 516 Perspectives and a Strategy for Factor Analysis 519 Supplement 9A: Some Computational Details for Maximum Likelihood Estimation 527 Recommended Computational Scheme, 528 Maximum Likelihood Estimators of p = LzLj + iftz 529

9.4 9.5

9.6

Exercises 530 References 538

xii

Contents 10 CANONICAL CORRELATION ANALYSIS 539

10.1 10.2 10.3

Introduction 539 Canonical Variates and Canonical Correlations 539 Interpreting the Population Canonical Variables 545
Identifying the Canonical Variables, 545 Canonical Correlations as Generalizations of Other Correlation Coefficients, 547 The First r Canonical Variables as a Summary of Variability, 548 A Geometrical Interpretation of the Population Canonical Correlation Analysis 549 The Sample Canonical Variates and Sample Canonical Correlations 550 Additional Sample Descriptive Measures 558 Matrices of Errors ofApproximations, 558 Proportions of Explained Sample Variance, 561 Large Sample Inferences 563 Exercises 567 References 574 575

10.4 10.5

10.6

11

DISCRIMINATION AND CLASSIFICATION

11.1 11.2 11.3

11.4 11.5

11.6

11.7

11.8

Classification of Normal Populations When X1 = 2 2 = % 584 Scaling, 589 Fisher's Approach to Classification with Two Populations, 590 Is Classification a Good Idea?, 592 Classification of Normal Populations When %i ^ X2, 593 Evaluating Classification Functions 596 Classification with Several Populations 606 The Minimum Expected Cost of Misclassification Method, 606 Classification with Normal Populations, 609 Fisher's Method for Discriminating among Several Populations 621 Using Fisher's Discriminants to Classify Objects, 628 Logistic Regression and Classification 634 Introduction, 634 The Logit Model, 634 Logistic Regression Analysis, 636 Classification, 638 Logistic Regression with Binomial Responses, 640 Final Comments 644 Including Qualitative Variables, 644 Classification Trees, 644 Neural Networks, 647 Selection ofVariables, 648

Introduction 575 Separation and Classification for Two Populations 576 Classification with Two Multivariate Normal Populations 584

Contents

xiii

Testing for Group Differences, 648 Graphics, 649 Practical Considerations Regarding Multivariate Normality, 649

Exercises 650 References 669

12 CLUSTERING, DISTANCE METHODS, AND ORDINATION 671

12.1 12.2

Introduction 671 Similarity Measures 673

Distances and Similarity Coefficients for Pairs of Items, 673 Similarities and Association Measures for Pairs of Variables, 677 Concluding Comments on Similarity, 678 Hierarchical Clustering Methods 680 Single Linkage, 682 Complete Linkage, 685 Average Linkage, 690 Ward's Hierarchical Clustering Method, 692 Final CommentsHierarchical Procedures, 695

12.3

VIA
12.5 12.6 12.7

Nonhierarchical Clustering Methods 696

K-means Method, 696 Final CommentsNonhierarchical Procedures, 701 Clustering Based on Statistical Models 703 Multidimensional Scaling 706 The Basic Algorithm, 708 Correspondence Analysis 716 Algebraic Development of Correspondence Analysis, 718 Inertia, 725 Interpretation in Two Dimensions, 726 Final Comments, 726 Biplots for Viewing Sampling Units and Variables 726 Constructing Biplots, 727 Procrustes Analysis: A Method for Comparing Configurations 732 Constructing the Procrustes Measure ofAgreement, 733

12.8 12.9

Supplement 12A: Data Mining 740

Introduction, 740 The Data Mining Process, 741 Model Assessment, 742 &

Exercises 747 References 755

APPENDIX DATA INDEX SUBJECT INDEX 757 764 767