DATAENG

(Engineering Data Analysis)

Lesson 11: Multiple Linear Regression

Multiple Linear Regression Model
Hypothesis test in multiple linear regression
Prediction of new observations
Adequacy of the regression model

1
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Applied Statistics and
Probability for Engineers

Sixth Edition
Douglas C. Montgomery George C. Runger

Chapter 12
Multiple Linear Regression

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model

12-1.1 Introduction
• Many applications of regression analysis
involve situations in which there are more than
one regressor variable.
• A regression model that contains more than
one regressor variable is called a multiple
regression model.

Sec 12-1 Multiple Linear Regression Model 3

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
12-1.1 Introduction

• For example, suppose that the effective life of a

cutting tool depends on the cutting speed and the
tool angle. A possible multiple regression model
could be
Y = b 0 + b1 x 1 + b2 x 2 +  (12-1)
where
Y – tool life
x1 – cutting speed
x2 – tool angle

Sec 12-1 Multiple Linear Regression Model 4

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.


12-1: Multiple Linear Regression Model

12-1.2 Least Squares Estimation of the Parameters
The method of least squares may be used to estimate the regression
coefficients in the multiple regression model,
Y = b0 + b1x1 + b2 x 2 +…+ bK x K + . Suppose that n > k observations are
available, and let xij denote the ith observation or level of variable xj. The
observations are
(xi1, xi2, , xik, yi), i = 1, 2, , n and n  k

It is customary to present the data for multiple regression in a table such as

Table 12-1.
Table 12-1 Data for Multiple Linear Regression
y x1 x2  xk
y1 x11 x12  x 1k
y2 x21 x22  x 2k

yn xn1 xn2  x nk
Sec 12-1 Multiple Linear Regression Model 5
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
12-1.2 Least Squares Estimation of the Parameters
• The least squares function is given by
2
n n  k 
L =  i2 =  y −b − b j xij 
 i 0  
(12-2)
i =1 i =1  j =1 
• The least squares estimates must satisfy

L n  k 
=−2  y − bˆ − bˆ x  = 0
0  j ij
(12-3)
b0 bˆ  i 
ˆ
0 , b1 ,, b k
ˆ i =1  j = 1 
and
L n  k 
=−2  y − bˆ − bˆ x  x = 0
 i 0 
j = 1, 2, , k (12-4)
b j 
j ij ij
bˆ 0 , bˆ 1 ,, bˆ k i =1  j =1 
Sec 12-1 Multiple Linear Regression Model 6
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
12-1.2 Least Squares Estimation of the Parameters

• The least squares normal equations are

n n n n
nbˆ 0 + bˆ 1  xi1 + bˆ   xi 2 +  + bˆ k  xik =  yi
i =1 i =1 i =1 i =1
n n n n n
bˆ 0  xi1 + bˆ   i1 + bˆ 
x 2
 xi1 xi 2 + + bˆ k  xi1 xik =  xi1 yi
i =1 i =1 i =1 i =1 i =1

   
n n n n n
bˆ 0  xik + bˆ 1  xik xi1 + bˆ 2  xik xi 2 +  + bˆ k  xik2 =  xik yi (12-5)
i =1 i =1 i =1 i =1 i =1

• The solution to the normal equations are the least

squares estimators of the regression coefficients.
Sec 12-1 Multiple Linear Regression Model 7
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
EXAMPLE 12-1 Wire Bond Strength In Chapter 1, we
used data on pull strength of a wire bond in a
semiconductor manufacturing process, wire length, and die
height to illustrate building an empirical model. We will use
the same data, repeated for convenience in Table 12-2,
and show the details of estimating the model parameters. A
three-dimensional scatter plot of the data is presented in
Fig. 1-15. Figure12-4 shows a matrix of two-dimensional
scatter plots of the data. These displays can be helpful in
visualizing the relationships among variables in a
multivariable data set. For example, the plot indicates that
there is a strong linear relationship between strength and
wire length.

Sec 12-1 Multiple Linear Regression Model 8

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
Example 12-1
Table 12-2 Wire Bond Data for Example 12-1
Observation Pull Wire Length Die Observation Pull Wire Die Height
Number Strength x1 Height Number Strength Length x2
y x2 y x1

1 9.95 2 50 14 11.66 2 360

2 24.45 8 110 15 21.65 4 205
3 31.75 11 120 16 17.89 4 400
4 35.00 10 550 17 69.00 20 600
5 25.02 8 295 18 10.30 1 585
6 16.86 4 200 19 34.93 10 540
7 14.38 2 375 20 46.59 15 250
8 9.60 2 52 21 44.88 15 290
9 24.35 9 100 22 54.12 16 510
10 27.50 8 300 23 56.63 17 590
11 17.08 4 412 24 22.13 6 100
12 37.00 11 400 25 21.15 5 400
13 41.95 12 500

Sec 12-1 Multiple Linear Regression Model 9

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model

Figure 12-4 Matrix of computer-generated scatter plots for the wire bond
pull strength data in Table 12-2.
Sec 12-1 Multiple Linear Regression Model 10
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
Example 12-1
Specifically, we will fit the multiple linear regression model

Y= b0 + b1x1 + b2x 2 + 

where Y = pull strength, x1 = wire length, and x2 = die height. From the
data in Table 12-2 we calculate

25 25 25
n = 25,  yi = 725.82,  xi1 = 206,  xi 2 = 8, 294
i =1 i =1 i =1
25 25 25

 i =1
x = 2, 396,
2
i1 x
i =1
2
i2 = 3, 531,848,  xi1 xi 2 = 77,177
i =1
25 25


i =1
xi1 yi = 8, 008.47,  xi 2 yi = 274,816.71
i =1

Sec 12-1 Multiple Linear Regression Model 11

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
Example 12-1
For the model Y = b0 + b1x1 + b2 x 2 + , the normal equations (12-5) are
n n n
1  xi1 + b 2  xi 2 =  yi
ˆ +b
nb ˆ ˆ
0
i =1 i =1 i =1
n n n n
ˆ
b   xi1 + bˆ   xi21 ˆ
+b   xi1xi 2 =  xi1 yi
i =1 i =1 i =1 i =1
n n n n
ˆ
b   xi 2 ˆ
+b   xi1 xi 2
ˆ
+b   xi22 =  xi 2 yi
i =1 i =1 i =1 i =1

Inserting the computed summations into the normal equations, we obtain

25bˆ 0 + 206bˆ 1 + 8294bˆ 2 = 725.82
206bˆ  + 2396bˆ  + bˆ  = 8,008.47
8294bˆ  + 77,177bˆ  + 3,531,848bˆ  = 274,816.71
Sec 12-1 Multiple Linear Regression Model 12
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
Example 12-1
The solution to this set of equations is

bˆ  = 2.26379, bˆ  = 2.74427, bˆ  = 0.01253

Therefore, the fitted regression equation is

yˆ = 2.26379 + 2.74427 x1 + 0.01253x2

Practical Interpretation: This equation can be used to

predict pull strength for pairs of values of the regressor
variables wire length (x1) and die height (x2). This is
essentially the same regression model given in Section 1-3.
Figure 1-16 shows a three-dimensional plot of the plane of
predicted values generated from this equation.
Sec 12-1 Multiple Linear Regression Model 13
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
EXCEL
Click

Click
Regression

Sec 12-1 Multiple Linear Regression Model 14

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
EXCEL

Sec 12-1 Multiple Linear Regression Model 15

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
EXCEL

Sec 12-1 Multiple Linear Regression Model 16

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
12-1.3 Matrix Approach to Multiple Linear Regression

Suppose the model relating the regressors

to the response is
yi = b0 + b1xi1 + b2 xi 2 +  + bk xik + i i = 1, 2, , n

In matrix notation this model can be written as

y = Xb +  (12-6)

Sec 12-1 Multiple Linear Regression Model 17

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model

Sec 12-1 Multiple Linear Regression Model 18

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
12-1.3 Matrix Approach to Multiple Linear Regression

where

 y1  1 x11 x12  x1k  b0   1 

y  1 x x  x  b   
y =  2 X= 21 22 2k 
b =  1  and  =  2 
       
       
 yn  1 x n1 xn2  x nk  b
 k  n 

Sec 12-1 Multiple Linear Regression Model 19

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
12-1.3 Matrix Approach to Multiple Linear Regression

We wish to find the vector of least squares

estimators that minimizes:
n
( ) (y − Xb )
L=  i
 2
=   = y − Xb
i =1

The resulting least squares estimate is

b̂ = (XX)−1 Xy (12-7)

Sec 12-1 Multiple Linear Regression Model 20

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
12-1.3 Matrix Approach to Multiple Linear Regression
The fitted regression model is
k
yˆi = bˆ 0 +  bˆ j xij i = 1, 2, , n (12-8)
j =1

In matrix notation, the fitted model is

yˆ = Xb ˆ

The difference between the observation yi and the fitted

value yˆ i is a residual, say, ei = yi − yˆi . The (n  1) vector of
residuals is denoted by

e = y − yˆ (12-9)

Sec 12-1 Multiple Linear Regression Model 21

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
Example 12-2
In Example 12-1, we illustrated fitting the multiple
regression model

y = b0 + b1x1 + b2x2 + 

where y is the observed pull strength for a wire

bond, x1 is the wire length, and x2 is the die height.
The 25 observations are in Table 12-2. We will now
use the matrix approach to fit the regression model
above to these data. The model matrix X and y
vector for this model are

Sec 12-1 Multiple Linear Regression Model 22

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
Example 12-2
1 2 50   9.95
1 8 
110   24.45
  
1 11 120   31.75
   
1 10 550   35.00 
1 8 295  25.02 
   
1 4 200   16.86 
1 2 375   14.38
   
1 2 52   9.60 
1 9 100   24.35
   
1 8 300   27.50 
   
1 4 412   17.08
1 11 400   37.00 
   
X = 1 12 500  y =  41.95
1 2 360   11.66 
   
1 4 205   21.65
1 4 400   17.89 
   
1 20 600   69.00 
   
1 1 585   10.30 
1 10 540   34.93
   
1 15 250   46.59 
1 15 290   44.88
   
1 16 510   54.12 
1 17 590   56.63
   
1 6 100   22.13
1   
 5 400   21.15
Sec 12-1 Multiple Linear Regression Model 23
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
Example 12-2
The XX matrix is 1 2 50 
1 1  1 
110 
XX =  2 8  5  
1 8
   
50 110  400  
1 5 400
 25 206 8,294 
=  206 2,396 77,177 
8,294 77,177 3,531,848

and the Xy vector is

 9.95 
1 1  1    725.82 
Xy =  2 8  5    =  8,008.47 
24.45
    
50 110  400   274,816.71
21.15
The least squares estimates are found from Equation 12-7 as

b̂ = (XX) −1 Xy
Sec 12-1 Multiple Linear Regression Model 24
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
Example 12-2
bˆ 0   25 −1
8, 294   725.82 
or   
206
 bˆ 1  =  206 2,396 77,177   8,008.37 
 ˆ  8, 294 77,177 3,531,848  274,811.31
b2     
 0.214653 −0.007491 −0.000340   725.82 
=  −0.007491 0.001671 −0.000019   8,008.47 
 −0.000340 −0.000019 +0.0000015  274,811.31
 2.26379143
=  2.74426964 
 0.01252781

Therefore, the fitted regression model with the regression coefficients

rounded to five decimal places is
yˆ = 2.26379 + 2.74427 x1 + 0.01253x2
This is identical to the results obtained in Example 12-1.

Sec 12-1 Multiple Linear Regression Model 25

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
Example 12-2
This regression model can be used to predict values of pull strength for various
values of wire length (x1) and die height (x2). We can also obtain the fitted values
by substituting each observation (xi1, xi2), i = 1, 2, , n, into the equation. For
example, the first observation has x11 = 2 and
x12 = 50, and the fitted value is
yˆ1 = 2.26379 + 2.74427 x11 + 0.01253x12
= 2.26379 + 2.74427(2) + 0.01253(50)
= 8.38
The corresponding observed value is y1 = 9.95. The residual corresponding to the
first observation is
e1 = y1 − yˆ1
= 9.95 − 8.38
= 1.57

Table 12-3 displays all 25 fitted values and the corresponding residuals. The
fitted values and residuals are calculated to the same accuracy as the original
data.
Sec 12-1 Multiple Linear Regression Model 26
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
Example 12-2
Table 12-3 Observations, Fitted Values, and Residuals for Example 12-2

Observation Observation
Number yi ŷi ei = yi − yˆi Number yi ŷi ei = yi − yˆi
1 9.95 8.38 1.57 14 11.66 12.26 –0.60
2 24.45 25.60 –1.15 15 21.65 15.81 5.84
3 31.75 33.95 –2.20 16 17.89 18.25 –0.36
4 35.00 36.60 –1.60 17 69.00 64.67 4.33
5 25.02 27.91 –2.89 18 10.30 12.34 –2.04
6 16.86 15.75 1.11 19 34.93 36.47 –1.54
7 14.38 12.45 1.93 20 46.59 46.56 0.03
8 9.60 8.40 1.20 21 44.88 47.06 –2.18
9 24.35 28.21 –3.86 22 54.12 52.56 1.56
10 27.50 27.98 –0.48 23 56.63 56.31 0.32
11 17.08 18.40 –1.32 24 22.13 19.98 2.15
12 37.00 37.46 –0.46 25 21.15 21.00 0.15
13 41.95 41.46 0.49

Sec 12-1 Multiple Linear Regression Model 27

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
Table 12-4 Minitab Multiple Regression Output for the Wire Bond Pull Strength Data
Regression Analysis: Strength versus :Length, Height

The regression equation is

Strength = 2.26 + 2.74 Length + 0.0125 Height

Predictor b̂ 0 Coef SE Coef T P VIF

Constant 2.264 1.060 2.14 0.044
Length
b̂ 1 2.74427 0.09352 29.34 0.000 1.2
Height b̂ 2 0.012528 0.002798 4.48 0.000 1.2

S = 2.288 R-Sq = 98.1% R-Sq (adj) = 97.9%

PRESS = 156.163 R-Sq (pred) = 97.44%

Analysis of Variance

Source DF SS MS F P
Regression 2 5990.8 2995.4 572.17 0.000
Residual Error 22 115.2 5.2 ̂ 2
Total 24 6105.9

Source DF Seq SS
Length 1 5885.9
Height 1 104.9

Predicted Values for New Observations

New Obs Fit SE Fit 95.0% CI 95.0% PI

1 27.663 0.482 (26.663, 28.663) (22.814, 32.512)

Values of Predictors for New Observations

News Obs Length Height

1 8.00 275

Sec 12-1 Multiple Linear Regression Model 28

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
Estimating 2

An unbiased estimator of 2 is
n
 i
e 2
SS E

ˆ2 = i =1= (1210)
n− p n− p

Sec 12-1 Multiple Linear Regression Model 29

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
12-1.4 Properties of the Least Squares Estimators

Unbiased estimators:

E (bˆ ) = E (XX )−1 XY 
= E (XX ) −1
X(Xb +  ) 
= E (XX ) −1
XXb + (XX )−1 X 
=b
Covariance Matrix:
C00 C01 C02 
C = ( XX) −1 = C10 C11 C12 
C20 C21 C22 
Sec 12-1 Multiple Linear Regression Model 30
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-1: Multiple Linear Regression Model
12-1.4 Properties of the Least Squares Estimators

Individual variances and covariances:

V (bˆ j ) =  2 C jj , j = 0, 1, 2
cov(bˆ i , bˆ j ) =  2 Cij , i j

In general,

ˆ −1
2

cov(b ) =  ( X X) =  C
2

Sec 12-1 Multiple Linear Regression Model 31

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-2: Hypothesis Tests in Multiple Linear Regression

12-2.1 Test for Significance of Regression

The appropriate hypotheses are

H 0 : b1 = b 2 =  = b k = 0
H1: bj  0 for at least one j (1211)

The test statistic is

SS R /k MS R
F0 = = (1212)
SS E /(n − p ) MS E

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 32

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-2: Hypothesis Tests in Multiple Linear Regression

12-2.1 Test for Significance of Regression

Table 12-9 Analysis of Variance for Testing

Significance of Regression in Multiple Regression
Source of Degrees of
Variation Sum of Squares Freedom Mean Square F0
Regression SSR k MSR MSR/MSE
Error or SSE n–p MSE
residual
Total SST n–1

k = number of coefficient -1 (variable)

p = k + 1 (number of coefficient)
n = number of observation

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 33

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-2: Hypothesis Tests in Multiple Linear Regression

EXAMPLE 12-3 Wire Bond Strength ANOVA We will test for

significance of regression (with  = 0.05) using the wire bond pull
strength data from Example 12-1. The total sum of squares is

2
 n 
  yi 
  (725.82) 2
SST = y y −  i =1 
= 27,178.5316 −
n 25
= 6105.9447

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 34

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-2: Hypothesis Tests in Multiple Linear Regression

Example 12-3

The regression or model sum of squares is computed as follows:

2
  n
  yi 
  2
SS R = bˆ  X y −  i =1  = 27,063.3581 −
( 725.82)
n 25
= 5990.7712

and by subtraction

SS E = SST − SS R = y y − b X y = 115.1716

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 35

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-3
The analysis of variance is shown in Table 12-10. To test H0: b1 = b2 = 0,
we calculate the statistic
MS R 2995.3856
f0 = = = 572.17
MS E 5.2352

Since f0 > f0.05,2,22= 3.44 (or since the Pvalue is considerably smaller than
 = 0.05), we reject the null hypothesis and conclude that pull strength is
linearly related to either wire length or die height, or both.

Practical Interpretation: Rejection of H0 does not necessarily imply that

the relationship found is an appropriate model for predicting pull strength
as a function of wire length and die height. Further tests of model
adequacy are required before we can be comfortable using this model in
practice.

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 36

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-2: Hypothesis Tests in Multiple Linear Regression

Example 12-3

Table 1210 Test for Significance of' Regression for Example 12-3

Source of Sum of Degrees of Mean

Variation Squares Freedom Square f0 Pvalue
Regression 5990.7712 2 2995.3856 572.17 1.08E-19
Error or residual 115.1735 22 5.2352
Total 6105.9447 24

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 37

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-2: Hypothesis Tests in Multiple Linear Regression

R2 and Adjusted R2
The coefficient of multiple determination

SS R SS E
R = 2
=1− (1213)
SST SST

• For the wire bond pull strength data, we find that

R2 = SSR/SST = 5990.7712/6105.9447 = 0.9811.
• Thus, the model accounts for about 98% of the
variability in the pull strength response.

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 38

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-2: Hypothesis Tests in Multiple Linear Regression

R2 and Adjusted R2
The adjusted R2 is

SS E / (n − p )
2
Radj =1− (1214)
SST / (n − 1)

• The adjusted R2 statistic penalizes the analyst

for adding terms to the model.
• It can help guard against overfitting (including
regressors that are not really useful)

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 39

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-2: Hypothesis Tests in Multiple Linear Regression

12-2.2 Tests on Individual Regression Coefficients

and Subsets of Coefficients

The hypotheses for testing the significance of any

individual regression coefficient:

H0: bj = bj0
H1: bj  bj0 (12-15)

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 40

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-2: Hypothesis Tests in Multiple Linear Regression

12-2.2 Tests on Individual Regression Coefficients and

Subsets of Coefficients

The test statistic is

ˆ −b
b ˆ −b
b
j j0 j j0
T0 = = (12-16)
ˆ )
se(b
 2C jj j

• Reject H0 if |t0| > t/2,n-p.

• This is called a partial or marginal test

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 41

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-2: Hypothesis Tests in Multiple Linear Regression

EXAMPLE 12-4 Wire Bond Strength Coefficient Test

Consider the wire bond pull strength data, and suppose that we
want to test the hypothesis that the regression coefficient for x2
(die height) is zero. The hypotheses are

H0: b2 = 0
H1: b2  0

The main diagonal element of the (XX)−1 matrix corresponding to b̂

is C22 = 0.0000015, so the t-statistic in Equation 12-16 is

bˆ  0.01253
t0 = = = 4.477
ˆ 2C22 (5.2352)(0.0000015)

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 42

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-4
Note that we have used the estimate of 2 reported to four decimal
places in Table 12-10. Since t0.025,22 = 2.074, we reject H0: b2 = 0 and
conclude that the variable x2 (die height) contributes significantly to the
model. We could also have used a P-value to draw conclusions. The
P-value for t0 = 4.477 is P = 0.0002, so with  = 0.05 we would reject
the null hypothesis.

Practical Interpretation: Note that this test measures the

marginal or partial contribution of x2 given that x1 is in the model. That is,
the t-test measures the contribution of adding the variable x2 = die
height to a model that already contains x1 = wire length. Table 12-4
shows the value of the t-test computed by Minitab. The Minitab t-test
statistic is reported to two decimal places. Note that the computer
produces a t-test for each regression coefficient in the model. These
t-tests indicate that both regressors contribute to the model.

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 43

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-2: Hypothesis Tests in Multiple Linear Regression

The general regression significance test or the extra

sum of squares method:

y = Xb + 
 b1 
b= 
b 2 

We wish to test the hypotheses:

H0: b1 = 0
H1: b1  0 (12-17)

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 44

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-2: Hypothesis Tests in Multiple Linear Regression

A general form of the model can be written:

y = Xb + = X1b1 + X2b2 +  (12-18)

where X1 represents the columns of X associated with

b1 and X2 represents the columns of X associated with
b2

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 45

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-2: Hypothesis Tests in Multiple Linear Regression

For the full model:

SS R (b) = bˆ  X y ( p = k + 1 degrees of freedom)

y  y − b  X y
MS E =
n− p

If H0 is true, the reduced model is

y = X2b2 +  (12-19)

SS R (b 2 ) = bˆ 2 X2 y ( p − r degrees of freedom ) (12-20)

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 46

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-2: Hypothesis Tests in Multiple Linear Regression

The test statistic is:

SS R (b1 | b 2 )/r
F0 = (12-21)
MS E

Reject H0 if f0 > f,r,n-p

The test in Equation (12-21) is often referred to as
a partial F-test

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 47

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-6 Wire Bond Strength General Regression Test

Consider the wire bond pull-strength data in Example 12-1. We will

investigate the contribution of two new variables, x3 and x4, to the
model using the partial F-test approach. The new variables are
explained at the end of this example. That is, we wish to test

H 0: b3 = b4 = 0 H1: b3  0 or b4  0

To test this hypothesis, we need the extra sum of squares due to b3 and
b4 or

SSR(b4, b3 b2, b1, b0) = SSR (b4, b3, b2, b1, b0) − SSR(b2, b1, b0)
= SSR(b4, b3, b2, b1b0) − SSR (b2, b1b0)

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 48

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-2: Hypothesis Tests in Multiple Linear Regression

Example 12-6
In Example 12-3 we calculated
2
 n 
  yi 
 
 
SS R (b2 , b1 |b0 ) = b X y −  i =1  = 5990.7712 ( two degrees of freedom)
n

Also, Table 12-4 shows the Minitab output for the model with only x1 and x2 as predictors. In
the analysis of variance table, we can see that SSR = 5990.8 and this agrees with our
calculation. In practice, the computer output would be used to obtain this sum of squares.

If we fit the model Y = b0 + b1x1 + b2x2 + b3x3 + b4x4, we can use the same matrix formula.
Alternatively, we can look at SSR from computer output for this model. The analysis of
variance table for this model is shown in Table 12-11 and we see that

SSR(b4, b3, b2, b1b0) = 6024.0 (four degrees of freedom)

Therefore,

SSR(b4, b3b2, b1, b0) = 6024.0 − 5990.8 = 33.2 (two degrees of freedom)

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 49

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-2: Hypothesis Tests in Multiple Linear Regression
Example 12-6
This is the increase in the regression sum of squares due to adding x3 and x4 to a model
already containing x1 and x2. To test H0, calculate the test statistic

SS R (b4 , b3 |b2 , b1, b0 )/2 33.2/2

f0 = = = 4.05
MS E 4.1

Note that MSE from the full model using x1, x2, x3 and x4 is used in the denominator of the
test statistic. Because f0.05, 2, 20 = 3.49, we reject H0 and conclude that at least one of the
new variables contributes significantly to the model. Further analysis and tests will be
needed to refine the model and determine if one or both of x3 and x4 are important.

The mystery of the new variables can now be explained. These are quadratic powers of the
original predictors wire length and wire height. That is, x3 = x12 and x4 = x22 . A test for
quadratic terms is a common use of partial F-tests. With this information and the original
data for x1 and x2, you can use computer software to reproduce these calculations. Multiple
regression allows models to be extended in such a simple manner that the real meaning of
x3 and x4 did not even enter into the test procedure. Polynomial models such as this are
discussed further in Section 12-6.

Sec 12-2 Hypothesis Tests in Multiple Linear Regression 50

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-3: Confidence Intervals in Multiple Linear Regression

12-3.1 Confidence Intervals on Individual

Regression Coefficients
Definition
A 100(1 - )% confidence interval on the regression
coefficient bj, j = 0, 1, , k in the multiple linear
regression model is given by

bˆ j − t/2, n − p 
ˆ 2C jj  b j  bˆ j + t/2, n − p 
ˆ 2C jj (12-22)

Sec 12-3 Confidence Intervals in Multiple Linear Regression 51

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-3: Confidence Intervals in Multiple Linear Regression
Example 12-7 Wire Bond Strength Confidence Interval
We will construct a 95% confidence interval on the parameter b1 in the wire bond
pull strength problem. The point estimate of b1 isbˆ 1 = 2.74427 and the diagonal
element of (XX)-1 corresponding to b1 is C11 = 0.001671. The estimate of 2 is
ˆ  = 5.2352 , and t0.025,22 = 2.074. Therefore, the 95% CI on b1 is computed from
Equation 12-22 as
2.74427 − (2.074) (5.2352)(.001671)  b1  2.74427
+ (2.074) (5.2352)(.001671)
which reduces to

2.55029  b1  2.93825

Also, computer software such as Minitab can be used to help calculate this
confidence interval. From the regression output in Table 10-4,bˆ 1 = 2.74427 and the
standard error of bˆ 1 = 0.0935 . This standard error is the multiplier of the t-table
constant in the confidence interval. That is, 0.0935 = (5.2352)(0.001671) .
Consequently, all the numbers are available from the computer output to
construct the interval and this is the typical method used in practice.
Sec 12-3 Confidence Intervals in Multiple Linear Regression 52
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-3: Confidence Intervals in Multiple Linear Regression

12-3.2 Confidence Interval on the Mean Response

The mean response at a point x0 is estimated by

ˆ
ˆ Y |x0 = x 0 b
 (12-23)

The variance of the estimated mean response is

( )
V ˆ Y | x0 =  2 x0 (XX )−1 x 0 (12-24)

Sec 12-3 Confidence Intervals in Multiple Linear Regression 53

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-3: Confidence Intervals in Multiple Linear Regression

12-3.2 Confidence Interval on the Mean Response

Definition
For the multiple linear regression model, a 100(1 - )%
confidence interval on the mean response at the point
x01, x02, , x0k is

ˆ Y |x 0 − t /2,n− p ˆ 2 x0 ( X X) −1 x 0
(12-25)
  Y |x 0  ˆ Y |x 0 + t /2,n− p ˆ 2 x0 ( X X) −1 x 0

Sec 12-3 Confidence Intervals in Multiple Linear Regression 54

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-3: Confidence Intervals in Multiple Linear Regression
Example 12-8 Wire Bond Strength Confidence Interval on the
Mean Response
The engineer in Example 12-1 would like to construct a 95% CI on the
mean pull strength for a wire bond with wire length x1 = 8 and die height
x2 = 275. Therefore,
1 
x0 =  8 
275

The estimated mean response at this point is found from Equation 12-23
as

 2.26379
ˆ Y |x0 = x 0bˆ = 1 8 275 2.74427 = 27.66
 0.01253

Sec 12-3 Confidence Intervals in Multiple Linear Regression 55

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
12-3: Confidence Intervals in Multiple Linear Regression
Example 12-8

The variance of ̂ Y |x0 is estimated by

 .214653 −.007491 −.000340   1

ˆ 2 x0 (XX)−1 x 0 = 5.2352 [1 8 275]  − .007491 .001671 − .000019   8
 − .000340 − .000019 .0000015  275
= 5.2352 (0.0444) = 0.23244

Therefore, a 95% CI on the mean pull strength at this point is found

from Equation 12-25 as

27.66 − 2.074 0.23244  Y |x0  27.66 + 2.074 0.23244

which reduces to
26.66  Y | x 0  28.66

Sec 12-3 Confidence Intervals in Multiple Linear Regression 56

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-4: Prediction of New Observations
A point estimate of the future observation Y0 is

yˆ 0 = x0b̂
A 100(1-)% prediction interval for this future observation
is

yˆ 0 − t /2, n− p ˆ 2 (1 + x0 ( X X) −1 x 0 ) (12-26)

 Y0  yˆ 0 − t /2, n− p ˆ 2 (1 + x0 ( X X) −1 x 0 )

Sec 12-4 Prediction of New Observations 57

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-4: Prediction of New Observations
Example 12-9 Wire Bond Strength Confidence Interval
Suppose that the engineer in Example 12-1 wishes to construct a 95%
prediction interval on the wire bond pull strength when the wire length is
x1 = 8 and the die height is x2 = 275. Note that x0 = [1 8 275] , and the point
estimate of the pull strength is yˆ 0 = x0bˆ = 27.66. Also, in Example 12-8 we
calculated x0 ( X X) −1 x0 = 0.04444 . Therefore, from Equation 12-26 we have

27.66 − 2.074 5.2352(1 + 0.0444)  Y0  27.66

+ 2.074 5.2352(1 + 0.0444)

and the 95% prediction interval is

22.81  Y0  32.51

Notice that the prediction interval is wider than the confidence interval on
the mean response at the same point, calculated in Example 12-8. The
Minitab output in Table 12-4 also displays this prediction interval.
Sec 12-4 Prediction of New Observations 58
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-5: Model Adequacy Checking
12-5.1 Residual Analysis
Example 12-10
The residuals for the model from Example 12-1 are shown in Table 12-3. A
normal probability plot of these residuals is shown in Fig. 12-6. No severe
deviations from normality are obviously apparent, although the two largest
residuals (e15 = 5.84 and e17 = 4.33) do not fall extremely close to a
straight line drawn through the remaining residuals.

Figure 12-6 Normal probability plot

of residuals.

Sec 12-5 Model Adequacy Checking 59

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-5: Model Adequacy Checking
12-5.1 Residual Analysis
Example 12-10
The standardized residuals
ei ei
di = = (12-27)
MS E ̂ 2
are often more useful than the ordinary residuals when assessing
residual magnitude. For the wire bond strength example, the
standardized residuals corresponding to e15 and e17 are
d15 = 5.84/ 5.2352 = 2.55 and d17 = 4.33/ 5.2352 = 1.89 , and they do not seem
unusually large. Inspection of the data does not reveal any error in
collecting observations 15 and 17, nor does it produce any other
reason to discard or modify these two points.
Sec 12-5 Model Adequacy Checking 60
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-5: Model Adequacy Checking
12-5.1 Residual Analysis
Example 12-10
The residuals are plotted against ŷ in Fig. 12-7, and against x1 and x2 in Figs. 12-8 and 12-9,
respectively.* The two largest residuals, e15 and e17, are apparent. Figure 12-8 gives some
indication that the model underpredicts the pull strength for assemblies with short wire length
(x1  6) and long wire length (x1  15) and overpredicts the strength for assemblies with
intermediate wire length (7  x1  14). The same impression is obtained from Fig. 12-7. Either
the relationship between strength and wire length is not linear (requiring that a term involving
x12 , say, be added to the model), or other regressor variables not presently in the model
affected the response.

Figure 12-7 Plot of residuals against

Sec 12-5 Model Adequacy Checking 61

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-5: Model Adequacy Checking
12-5.1 Residual Analysis
Example 12-10
Either the relationship between strength and wire length is not
linear (requiring that a term involvingx12 , say, be added to the
model), or other regressor variables not presently in the model
affected the response.

Figure 12-8 Plot of residuals

against x1.

Sec 12-5 Model Adequacy Checking 62

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-5: Model Adequacy Checking
12-5.1 Residual Analysis
Example 12-10
Figure 12-9 Plot of residuals against x2.

Sec 12-5 Model Adequacy Checking 63

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-5: Model Adequacy Checking
12-5.1 Residual Analysis

ei
ri = i = 1, 2, , n (12-28)
ˆ 2 (1 − hii )

where hii is the ith diagonal element of the matrix

H = X(XX)-1X

The H matrix is sometimes called the “hat” matrix, since

yˆ = Xbˆ = X(X X)−1 X y = Hy

Sec 12-5 Model Adequacy Checking 64

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-5: Model Adequacy Checking
12-5.1 Residual Analysis

Since each row of the matrix X corresponds to a vector,

say xi = 1, xi1 , xi 2 , , xik  , another way to write the diagonal
elements of the hat matrix is

hii = xi (X X )−1 xi (12-29)

The variance of the ith residual is

V(ei) = 2(1 - hii), i = 1, 2, , n

Sec 12-5 Model Adequacy Checking 65

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-5: Model Adequacy Checking
12-5.1 Residual Analysis
To illustrate, consider the two observations identified in the wire bond strength
data (Example 12-10) as having residuals that might be unusually large,
observations 15 and 17. The standardized residuals are
e 5.84 e 4.33
d15 = 15 = = 2.55 and d17 = 17 = = 1.89
ˆ 2 5 .2352 MS E 5.2352
Now h15,15 = 0.0737 and h17,17 = 0.2593, so the studentized residuals are
e15 5.84
r15 = = = 2.65
(
ˆ 2 1 − h15,15 ) 5.2352(1 − 0.0737 )
and e17 4.33
r17 = = = 2.20
(
ˆ 2 1 − h17,17 ) 5.2352(1 − 0.2593)

Notice that the studentized residuals are larger than the corresponding
standardized residuals. However, the studentized residuals are still not so large
as to cause us serious concern about possible outliers.

Sec 12-5 Model Adequacy Checking 66

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-5: Model Adequacy Checking
12-5.2 Influential Observations
Cook’s distance measure

(bˆ (i ) − bˆ )X X(bˆ (i ) − bˆ )

Di = i = 1, 2, , n
pˆ 2

ri2 hii
Di = i = 1, 2,  , n (12-30)
p (1 − hii )

Sec 12-5 Model Adequacy Checking 67

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-5: Model Adequacy Checking
EXAMPLE 12-11 Wire Bond Strength Cook’s Distances Table
12-12 lists the values of the hat matrix diagonals hii and Cook’s
distance measure Di for the wire bond pull strength data in Example
12-1. To illustrate the calculations, consider the first observation:
r12 h11
D1 = 
p (1 − h11 )
[e1/ MS E (1 − h11 ) ]2 h11
=− 
p (1 − h11 )
[1.57 / 5.2352(1 − 0.1573) ]2 0.1573
= 
3 (1 − 0.1573)
= 0.035
The Cook distance measure Di does not identify any potentially
influential observations in the data, for no value of Di exceeds unity.

Sec 12-5 Model Adequacy Checking 68

Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 12-5: Model Adequacy Checking
Example 12-11
TABLE • 12-8 Influence Diagnostics for the Wire Bond Pull Strength Data
Observations hii Cook’s Distance Measure Di Observations i hii Cook’s Distance
i Measure Di
1 0.1573 0.035 14 0.1129 0.003
2 0.1116 0.012 15 0.0737 0.187
3 0.1419 0.060 16 0.0879 0.001
4 0.1019 0.021 17 0.2593 0.565
5 0.0418 0.024 18 0.2929 0.155
6 0.0749 0.007 19 0.0962 0.018
7 0.1181 0.036 20 0.1473 0.000
8 0.1561 0.020 21 0.1296 0.052
9 0.1280 0.160 22 0.1358 0.028
10 0.0413 0.001 23 0.1824 0.002
11 0.0925 0.013 24 0.1091 0.040
12 0.0526 0.001 25 0.0729 0.000
13 0.0820 0.001
Sec 12-5 Model Adequacy Checking 69
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.
 Important Terms & Concepts of Chapter 12
All possible regressions Polynomial regression model
Analysis of variance test in multiple Prediction interval on a future
regression observation
Categorical variables PRESS statistic
Confidence intervals on the mean Residual analysis & model adequacy
response checking
Cp statistic Significance of regression
Extra sum of squares method Stepwise regression & related methods
Hidden extrapolation Variance Inflation Factor (VIF)
Indicator variables
Inference (test & intervals) on individual
model parameters
Influential observations
Model parameters & their interpretation
in multiple regression
Multicollinearity
Multiple regression
Outliers
Chapter 12 Summary 70
Copyright © 2014 John Wiley & Sons, Inc. All rights reserved.