03 Revisions L Regression

1
STAT2008/STAT6038
Revision of Simple Linear Regression
The Simple Linear Regression Model and the
Parameter Estimates
2
The dependent (or response) variable is the

variable we wish to understand or predict
The independent (or predictor) variable is the
variable we will use to understand or predict the
dependent variable
Regression analysis is a statistical technique that
uses observed data to relate the dependent
variable to one or more independent variables
Goal
3
The objective of regression analysis is to build a

regression model (or predictive equation) that can
be used to describe, predict and control the
dependent variable on the basis of the independent
variable
What regression does
4
Develops an equation which represents the relationship

between the variables.
Simple linear regression straight line relationship between y
and x (i.e. one explanatory variable)
Multiple linear regression straight line relationship between
y and x1, x2, .., xk where we have k explanatory variables
Non-linear regression relationship not a straight line (i.e. y
is related to some function of x, e.g. log(x))
Goal
5
To model the relationship between a response variable (Y) and an

explanatory variable (X) using a straight line model of the form.
Y 0 1X
where 0 is the intercept, 1 is the slope and is a random error

The interpretation of the intercept 0 is that it is the expected value of Y
when X is equal to zero (this may or may not be an interpretable quantity).
The interpretation of the slope parameter 1 is that when X increases by 1,
then Y increases by an amount 1 .
The quantities 0 and 1 are called the parameters of the regression model.
They are fixed, unknown quantities that need to be estimated
Assumptions
6
The errors are usually assumed to be

independent, zero-mean, constant variance
Normal random variables
so is distributed as a Normal random variate

with mean 0 and spread (standard
deviation)
The key features are:
Bivariate Fit of Y By X. A scatterplot an X-Y plot graphically

representing the relationship between X and Y. Here, we are looking for a
straight-line relationship
Key Features
8
Linear Fit. The fitted modeI is of the form

Y 0 1 x
with the two numbers 0 and 1 given being the estimates of 0 and, 1
respectively. Note that the fitted line contains no error term since the error
random variable is expected to be zero.
Note
9
In reality predicted values wont be exact

So, more reasonable is to ask what would be the
expected value of the dependent variable Y when
X=x
So in general we use
10
E(Y|X=x)
If we add in assumption of linearity we get
E Y | X x 0 1x
0 and 1 are coefficients that determine the straight line

In general, for any pair of observations
11
E Yi | X xi 0 1xi
In practice, the observed value will (almost
always) differ from the expected value.
Denote difference by greek epsilon, i
Mean of i will be zero
i Yi E Yi | X xi Yi 0 1 xi
Yi 0 1 xi i
Yi 0 1 xi i
12
Called the population or true regression line

0 and 1 are constants to be estimated
i is a random variable with mean = 0
Interpretation will be in two parts
an expectation (0 + 1 xi) which reflects the
systematic relationship, and a discrepancy (i)
which represents all the other many factors (apart
from X) which may affect Y.
Estimation
13
Is done using a process called Least Squares

Estimation
In practice, the population regression line has to be
estimated:
Yi 0 1 xi i
is estimated by y b0 b1 x
How good is the estimate?
14
Look at the distance between the points (xi,yi) and the line.
Residual
15
Vertical distance between observed point and fitted line is

called the residual.
That is ri=yi-(b0+b1xi)
ri estimates i, the error variable
Want to determine values of b0+b1 that best fit the data
choose the values which minimise the sum of the squared
differences between observed and estimated, i.e. choose
values of slope and intercept which minimise
n n
i
( y
i 1

y ) 2
i
( r ) 2
i 1
This is the least squares method
16
Choose our estimates of slope and intercept to give

the smallest residual sum of squares
Uses calculus to find estimates
Estimate intercept 0 by b0 , sometimes use 0 .

Estimate slope by b , sometimes use .
1 1 1
Estimation
17
HOW TO MINIMIZE the sum of the squared errors

(SSE)?
choose 0 , 1 to min imise

n 2
S ( 0 , 1 ) (Yi 0 1 X i )
i 1
Differentiating
S n
A) 2 (Yi 0 1 X i )

0 i 1
S n
B) 2 X i (Yi 0 1 X i )

1 i 1
Estimation
18
The regression line
19
0 Y 1 X
Y 0 1 X
( X , Y ) is on the fitted line
The Regression Line
20
Example
21
The investment in certain share portfolios (x) and the

value after a year (y)in $000 are given in the table
below.
x 1 2 4 5 8
y 3 3 7 6 12
Fit a regression line to these data by hand

Best first step look at a scatterplot!
22
Scatterplot of y vs x
12
10
8
y
2
0 1 2 3 4 5 6 7 8
x
Want to find b0, b1 for y b0 b1 x
23
Given that cov(x,y)=9.75, var(x)=7.5

x average=4, y average =6.2
cov( x, y) 9.75
1 b1 2
1.3
sx 7.5
b y b x 6.2 1.3*4 1
0 0 1
So, least squares regression line is
24
y 1 1.3 x
Fitted Line Plot
y = 1.000 + 1.300 x
S 1.16905
12
R-Sq 92.5%
R-Sq(adj) 90.0%
10
8
y
2
0 1 2 3 4 5 6 7 8
x
Can find predicted values and residuals,
25
yi 1 1.3xi ri yi yi yi yi
2
x y
1 3 2.3 0.7 0.49
2 3 3.6 -0.6 0.36
4 7 6.2 0.8 0.64
5 6 7.5 -1.5 2.25
8 12 11.4 0.6 0.36

03 Revisions L Regression

Uploaded by

Copyright:

Available Formats

03 Revisions L Regression

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

03 Revisions L Regression

Uploaded by

Copyright:

Available Formats

1

The dependent (or response) variable is the

The objective of regression analysis is to build a

Develops an equation which represents the relationship

To model the relationship between a response variable (Y) and an

where 0 is the intercept, 1 is the slope and is a random error

The errors are usually assumed to be

so is distributed as a Normal random variate

Bivariate Fit of Y By X. A scatterplot an X-Y plot graphically

Linear Fit. The fitted modeI is of the form

In reality predicted values wont be exact

0 and 1 are coefficients that determine the straight line

Called the population or true regression line

Is done using a process called Least Squares

Vertical distance between observed point and fitted line is

Choose our estimates of slope and intercept to give

Estimate intercept 0 by b0 , sometimes use 0 .

HOW TO MINIMIZE the sum of the squared errors

choose 0 , 1 to min imise

The investment in certain share portfolios (x) and the

Fit a regression line to these data by hand

Given that cov(x,y)=9.75, var(x)=7.5

1 3 2.3 0.7 0.49

2 3 3.6 -0.6 0.36

4 7 6.2 0.8 0.64

5 6 7.5 -1.5 2.25

8 12 11.4 0.6 0.36

You might also like