CHAPTER TWO

Simple Linear Regression

By : Habtamu Legese (Asst.Prof)

13/11/2023 By: Habtamu Legese (Assistant Professor) 1

 2.1. Concept of Regression Function
• Regression analysis is one of the most commonly used
tools in econometrics analysis. What is regression?
• Regression is a statistical method that attempts to
determine the strength and character of the relationship
between one dependent variable and a series of other
variables (known as independent variables).
• Regression analysis is concerned with describing and
explaining the relationship between the dependent and
one or more independent variables.
13/11/2023 By: Habtamu Legese (Assistant Professor) 2
13/11/2023 By: Habtamu Legese (Assistant Professor) 3
Cont.
• We denote the dependent variable by Y and the explanatory
variables by X1 , X 2 ,..., X k .
• If k = 1 , that is, there is only one explanatory variable, we
have what is known as simple regression. This is what we
discuss in this unit.
• On the other hand, if K > 1, that is, there are more than one
explanatory variable, we have what is known as multiple
regression.
• In a regression analysis our task is to estimate the population
regression function (PRF) on the basis of sample regression
function (SRF) as accurately as possible.
13/11/2023 By: Habtamu Legese (Assistant Professor) 4
Regression analysis has the following objectives & uses
To show the relationship among variables.
To estimate average value (mean) of the dependent variable
given the value of independent variable(s);
To test hypothesis about sign and magnitude of relationship
To forecast future value(s) of the dependent variable
It is to explain the variation in the dependent variable based on
the variation in one or more independent variables.

13/11/2023 By: Habtamu Legese (Assistant Professor) 5

2.1. Stochastic and Non-stochastic Relationships
• A relationship between X and Y, characterized as Y = f(X) is
said to be deterministic or non-stochastic if for each value of
the independent variable (X) there is one and only one
corresponding value of dependent variable (Y).
• On the other hand, a relationship between X and Y is said to
be stochastic if for a particular value of X there is a whole
probabilistic distribution of values of Y.
• In such a case, for any given value of X, the dependent
variable Y assumes some specific value only with some
probability.

13/11/2023 By: Habtamu Legese (Assistant Professor) 6

Example ………. Supply function
• Assuming that the supply for a certain commodity depends
on its price (other determinants taken to be constant) and the
function being linear, the relationship can be put as:

• The above relationship between P and Q is such that for a

particular value of P, there is only one corresponding value of
Q. This is, therefore, a deterministic (non-stochastic)
relationship since for each price there is always only one
corresponding quantity supplied.
13/11/2023 By: Habtamu Legese (Assistant Professor) 7
Cont.
• This implies that all the variation in Y is due solely to
changes in X, and that there are no other factors affecting the
dependent variable.
• If this were true all the points of price-quantity pairs, if
plotted on a two dimensional plane, would fall on a straight
line.
• However, if we gather observations on the quantity actually
supplied in the market at various prices and if we plot them
on a diagram we may see that they do not fall on a straight
line.
13/11/2023 By: Habtamu Legese (Assistant Professor) 8
Cont.
• The deviation of the observation from the line may be
attributed to several factors.
A. Omission of variables from the function
B. Random behaviour of human beings
C. Imperfect specification of the mathematical form of
the model
D. Error of aggregation
E. Error of measurement
13/11/2023 By: Habtamu Legese (Assistant Professor) 9
Cont.
• In order to take into account the above sources of
errors we introduce a random variable which is
usually denoted by the letter ‘μ’ or ‘ε’ and is called
error term or random disturbance or stochastic term
of the function.
• By introducing this random variable in the function the
model is rendered stochastic of the form:

13/11/2023 By: Habtamu Legese (Assistant Professor) 10

Cont.
• Thus a stochastic model is a model in which the
dependent variable is not only determined by the
explanatory variable(s) included in the model but also
by others which are not included in the model.

13/11/2023 By: Habtamu Legese (Assistant Professor) 11

Cont.
• If we have n observations on Y and X, we can write the
simple regression model (2.1.2) by adding subscripts as

• Furthermore, the above regression equation can be rearranged

for the error term as

13/11/2023 By: Habtamu Legese (Assistant Professor) 12

2.2. Simple Linear Regression model.
• A stochastic relationship with only one explanatory
variable is called simple linear regression model.

The term ‘simple’ refers to the fact that we use only two
variables (one dependent and one independent variable).
Linear refers to linear in parameters, it may or may not be
linear in the variable. The parameters appear with a power
of one & is not multiplied/divided by other parameters.
13/11/2023 By: Habtamu Legese (Assistant Professor) 13
Cont.
• The true relationship which connects the variables involved is
split into two parts: a part represented by a line and a part
represented by the random term ‘u’.

13/11/2023 By: Habtamu Legese (Assistant Professor) 14

Cont.
• The scatter of observations represents the true relationship
between Y and X.
• The line represents the exact part of the relationship and
the deviation of the observation from the line represents the
random component of the relationship.

13/11/2023 By: Habtamu Legese (Assistant Professor) 15

The Gaussian, standard, or classical linear regression model (CLRM)

• CLRM is the cornerstone of most econometric theory,

makes 10 assumptions.
• It is classical in the sense that it was developed first by
Gauss in 1821 and since then has served as a norm or
a standard against which may be compared the
regression models that do not satisfy the Gaussian
assumptions.

13/11/2023 By: Habtamu Legese (Assistant Professor) 16

 Assumption 1: Linear regression model.
1. The model is linear in parameters.
• The classicals assumed that the model should be linear in the
parameters regardless of whether the explanatory and the
dependent variables are linear or not.
• This is because if the parameters are non-linear it is difficult
for estimation.
• Example :

13/11/2023 By: Habtamu Legese (Assistant Professor) 17

Assumption 2: X values are fixed in repeated sampling.

• Values taken by the regressor X are considered fixed in

repeated samples.
• More technically, X is assumed to be non-stochastic. In other
words X is assumed to be known with certainty.
• What all this means is that our regression analysis is
conditional regression analysis, that is, conditional on the
given values of the regressor (s) X.

13/11/2023 By: Habtamu Legese (Assistant Professor) 18

Assumption 3: Zero mean value of disturbance ui.

• That is; given the value of X the mean or expected value of

the disturbance term is zero.
• Technically, the conditional mean value of is zero.

13/11/2023 By: Habtamu Legese (Assistant Professor) 19

Assumption 4: Homoscedasticity or equal variance of u . i

• Given the value of X, the variance of ui is the same for all

observations.
For all values of X, the u’s will show the same dispersion
around their mean.
• This constant variance is called homoscedasticity assumption
and the constant variance itself is called homoscedastic
variance.

13/11/2023 By: Habtamu Legese (Assistant Professor) 20

Cont.
The disturbance term (U) has a normal distribution
• This means the values of u (for each x) have a bell
shaped symmetrical distribution about their zero mean
and constant variance , i.e.

13/11/2023 By: Habtamu Legese (Assistant Professor) 21

Cont.
Randomness of disturbance term 𝒖𝒊 : the error term is
assumed to be a random variable.
• This means that the value which u may assume in any one
period depends on chance; it may be positive, negative or
zero.
• Every value has a certain probability of being assumed by u
in any particular instance.

13/11/2023 By: Habtamu Legese (Assistant Professor) 22

Assumption 5: No autocorrelation between the disturbances terms.

• This means the value which the random term assumed in one
period does not depend on the value which it assumed in any
other period.
• Algebraically,

13/11/2023 By: Habtamu Legese (Assistant Professor) 23

Assumption 6: Zero covariance between ui and Xi
This means there is no correlation between the random
variable and the explanatory variable.
• If two variables are unrelated their covariance is zero.

13/11/2023 By: Habtamu Legese (Assistant Professor) 24

Assumption 7: The number of observations n must be greater than
the number of parameters to be estimated
• Alternatively, the number of observations n must be
greater than the number of explanatory variables.
• From a single observation there is no way to estimate
the two unknowns, α and β.
• We need at least two pairs of observations to estimate
the two unknowns.

13/11/2023 By: Habtamu Legese (Assistant Professor) 25

Assumption 8: Variability in X values

The X values in a given sample must not all be the same.

Technically, var (X) must be a finite positive number.

ഥ , it is impossible to estimate the parameters.
if 𝑿𝒊 = 𝑿

13/11/2023 By: Habtamu Legese (Assistant Professor) 26

Assumption 9: The regression model is correctly specified.
• Alternatively, there is no specification bias or error in the
model used in empirical analysis.
• Some important questions that arise in the specification of
the model include the following:
(1) What variables should be included in the model?
(2) What is the functional form of the model? Is it linear in
the parameters, the variables, or both?
(3) What are the probabilistic assumptions made about the
Yi , the Xi, and the ui entering the model?

13/11/2023 By: Habtamu Legese (Assistant Professor) 27

Assumption 10: There is no perfect multicollinearity.

• That is, there are no perfect linear relationships among the

explanatory variables.

13/11/2023 By: Habtamu Legese (Assistant Professor) 28

Cont.
• We can now use the above assumptions to derive the following basic
concepts.
A. The dependent variable Y is normally distributed.
i

B. successive values of the dependent variable are

independent, i.e.

13/11/2023 By: Habtamu Legese (Assistant Professor) 29

 A. The dependent variable Y is normally distributed.

13/11/2023 By: Habtamu Legese (Assistant Professor) 30

B. successive values of the dependent variable are independent

13/11/2023 By: Habtamu Legese (Assistant Professor) 31

 2.2.2 Methods of estimation
•The parameters of the simple linear regression
model can be estimated by various methods.
Three of the most commonly used methods
are:
1. Ordinary least square method (OLS)
2. Method of moments (MM)

13/11/2023 By: Habtamu Legese (Assistant Professor) 32

 Ordinary least square method (OLS)
• The model Y = a + bX + U is called the true relationship
i i i

between Y and X because Y and X represent their respective

population value, a and b are called the true parameters
since they are estimated from the population value of Y and
X.
• But it is difficult to obtain the population value of Y and X
because of technical or economic reasons.
• So we are forced to take the sample value of Y and X. The
parameters estimated from the sample value are called the
estimators of the true parameters a and b and are symbolized
as and .
13/11/2023 By: Habtamu Legese (Assistant Professor) 33
Cont.

• Estimation of a and b by least square method (OLS)

involves finding values for the estimates and
which will minimize the sum of square of the residuals

13/11/2023 By: Habtamu Legese (Assistant Professor) 34

Method of Ordinary least square method (OLS)

13/11/2023 By: Habtamu Legese (Assistant Professor) 35

13/11/2023 By: Habtamu Legese (Assistant Professor) 36
Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 37

Cont.

• Substituting the values of α from (2.10) to (2.13), we get:

13/11/2023 By: Habtamu Legese (Assistant Professor) 38

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 39

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 40

Substituting (2.15) and (2.16) in (2.14), we
get

13/11/2023 By: Habtamu Legese (Assistant Professor) 41

Summary

= =

13/11/2023 By: Habtamu Legese (Assistant Professor) 42

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 43

 The formula in deviation form

13/11/2023 By: Habtamu Legese (Assistant Professor) 44

Summary

= =

13/11/2023 By: Habtamu Legese (Assistant Professor) 45

 2.3 The Method of Least Squares
Firm (i) Sales (Yi) Advertising Expense (Xi)
Numerical
1 11 10
Example:
2 10 7
Explaining sales
3 12 10
= f(advertising)
4 6 5
Sales are in
5 10 8
thousands
6 7 8
of Birr &
7 9 6
advertising 8 10 7
expenses are in 9 11 9
hundreds of Birr. 10 10 10
By: Habtamu Legese (Assistant Professor)
13/11/2023 46
 2.3 The Method of Least Squares
X i2 96
i Yi Xi X iYi Y = = 9.6
1 11 10 110 100 10
. 2 10 7 70 49 80
X= =8
3 12 10 120 100 10
4 6 5 30 25
ˆ
b=  X iYi  nXY
 i
5 10 8 80 64
X 2
 n X 2
6 7 8 56 64
7 9 6 54 36
ˆ 789  10(8)(9.6)
b= = 0.75
8 10 7 70 49 668  10(8) 2

9 11 9 99 81
10 10 10 100 100 aˆ = Y  bˆX
13/11/2023
Ʃ 96 80 789 668
By: Habtamu Legese (Assistant Professor)
aˆ = 9.6  0.75(8) = 3.6 47
 2.3 The Method of Least Squares

i Yi Xi X iYi X i2
1 11 10 110 100

. 2 10 7 70 49
3 12 10 120 100
4 6 5 30 25
5 10 8 80 64
6 7 8 56 64
7 9 6 54 36
8 10 7 70 49
9 11 9 99 81
10 10 10 100 100
Ʃ 96 80 789 668
13/11/2023 By: Habtamu Legese (Assistant Professor) 48
2.3 The Method of Least Squares
i Yi Xi 𝑋𝑖 − 𝑋ሜ Y−𝑌ሜ 𝑥y 𝑥𝑖2
1 11 10 2 1.4 2.8 4
2. 10 7 -1 0.4 -0.4 1
3 12 10 2 2.4 4.8 4
4 6 5 -3 -3.6 10.8 9
5 10 8 0 0.4 0 0
6 7 8 0 -2.6 0 0
7 9 6 -2 -0.6 1.2 4
8 10 7 -1 0.4 -0.4 1
9 11 9 1 1.4 1.4 1
10 10 10 2 0.4 0.8 4
Ʃ 96 80 0 0 21 28
13/11/2023 By: Habtamu Legese (Assistant Professor) 49
 2.3 The Method of Least Squares
i Yi Xi Yˆi = 3.6 + 0.75 X i ei = Yi  Yˆi e 2
i
Yˆi = 3.6 + 0.75 X i

e
1 11 10 11.1 -0.10 0.01
2 10 7 8.85 1.15 1.3225
2
i = 14.65
3 12 10 11.10 0.90 0.81
4 6 5 7.35
Yˆ = 9.6
-1.35 1.8225
5 10 8 9.60 0.40 0.16
6 7 8 9.60 -2.60 6.76
7
8
9
10
6
7
8.10
8.85
0.90
1.15
0.81
1.3225 e = 0 i
10.35 0.4225

e X
9 11 9 0.65
10 10 10 11.10 -1.10 1.21
i i =0
Ʃ13/11/202396 80 96 0 14.65
By: Habtamu Legese (Assistant Professor) 50
Example

13/11/2023 By: Habtamu Legese (Assistant Professor) 51

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 52

13/11/2023 By: Habtamu Legese (Assistant Professor) 53
13/11/2023 By: Habtamu Legese (Assistant Professor) 54
Estimation of a function with zero intercept

13/11/2023 By: Habtamu Legese (Assistant Professor) 55

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 56

Statistical Properties of Least Square
Estimators
• PROPERTIES OF OLS ESTIMATORS
• The ideal or optimum properties that the OLS estimates
possess may be summarized by well known theorem known as
the Gauss-Markov Theorem.
• Statement of the theorem: “Given the assumptions of the
classical linear regression model, the OLS estimators, in the
class of linear and unbiased estimators, have the minimum
variance, i.e. the OLS estimators are BLUE.

13/11/2023 By: Habtamu Legese (Assistant Professor) 57

Cont.
• According to the this theorem, under the basic
assumptions of the classical linear regression model, the
least squares estimators are linear, unbiased and have
minimum variance (i.e. are best of all linear unbiased
estimators).
• Some times the theorem referred as the BLUE theorem
i.e. Best, Linear, Unbiased Estimator. An estimator is
called BLUE if:
a. Linear: a linear function of the a random variable, such
as, the dependent variable Y.
b. Unbiased: its average or expected value is equal to
the true population parameter.
13/11/2023 By: Habtamu Legese (Assistant Professor) 58
Cont.
c. Minimum variance: It has a minimum variance
in the class of linear and unbiased estimators. An
unbiased estimator with the least variance is
known as an efficient estimator.

• According to the Gauss-Markov theorem, the

OLS estimators possess all the BLUE properties.

• Lets proof these properties one by one.

13/11/2023 By: Habtamu Legese (Assistant Professor) 59
13/11/2023 By: Habtamu Legese (Assistant Professor) 60
13/11/2023 By: Habtamu Legese (Assistant Professor) 61
13/11/2023 By: Habtamu Legese (Assistant Professor) 62
Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 63

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 64

2.2.2.4. Statistical test of Significance of the OLS Estimators
(First Order tests)
• After the estimation of the parameters and the
determination of the least square regression line, we
need to know how ‘good’ is the fit of this line to the
sample observation of Y and X.
• We need to measure the dispersion of observations
around the regression line.
• This knowledge is essential because the closer the
observation to the line, the better the goodness of fit,
i.e. the better is the explanation of the variations of Y
by the changes in the explanatory variables.
13/11/2023 By: Habtamu Legese (Assistant Professor) 65
Cont.
1. The coefficient of determination (the
𝟐
square of the correlation coefficient i.e. 𝑹 ).
This test is used for judging the explanatory
power of the independent variable(s).
2. The standard error tests of the
estimators. This test is used for judging the
statistical reliability of the estimates of the
regression coefficients.

13/11/2023 By: Habtamu Legese (Assistant Professor) 66

TESTS OF THE ‘GOODNESS OF FIT’ WITH

13/11/2023 By: Habtamu Legese (Assistant Professor) 67

13/11/2023 By: Habtamu Legese (Assistant Professor) 68
13/11/2023 By: Habtamu Legese (Assistant Professor) 69
13/11/2023 By: Habtamu Legese (Assistant Professor) 70
13/11/2023 By: Habtamu Legese (Assistant Professor) 71
13/11/2023 By: Habtamu Legese (Assistant Professor) 72
13/11/2023 By: Habtamu Legese (Assistant Professor) 73
13/11/2023 By: Habtamu Legese (Assistant Professor) 74
13/11/2023 By: Habtamu Legese (Assistant Professor) 75
2. TESTING THE SIGNIFICANCE OF OLS PARAMETERS

• To test the significance of the OLS parameter estimators we

need the following:

13/11/2023 By: Habtamu Legese (Assistant Professor) 76

13/11/2023 By: Habtamu Legese (Assistant Professor) 77
13/11/2023 By: Habtamu Legese (Assistant Professor) 78
13/11/2023 By: Habtamu Legese (Assistant Professor) 79
i) Standard error test

13/11/2023 By: Habtamu Legese (Assistant Professor) 80

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 81

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 82

13/11/2023 By: Habtamu Legese (Assistant Professor) 83
13/11/2023 By: Habtamu Legese (Assistant Professor) 84
Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 85

ii) Student’s t-test
• Like the standard error test, this test is also important to test
the significance of the parameters. From your statistics, any
variable X can be transformed into t using the general
formula:

13/11/2023 By: Habtamu Legese (Assistant Professor) 86

13/11/2023 By: Habtamu Legese (Assistant Professor) 87
Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 88

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 89

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 90

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 91

13/11/2023 By: Habtamu Legese (Assistant Professor) 92
13/11/2023 By: Habtamu Legese (Assistant Professor) 93
Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 94

13/11/2023 By: Habtamu Legese (Assistant Professor) 95
Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 96

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 97

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 98

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 99

13/11/2023 By: Habtamu Legese (Assistant Professor) 100
Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 101

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 102

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 103

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 104

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 105

Cont.

13/11/2023 By: Habtamu Legese (Assistant Professor) 106

2.2.3 Reporting the Results of Regression Analysis

13/11/2023 By: Habtamu Legese (Assistant Professor) 107

 Thank You
13/11/2023 By: Habtamu Legese (Assistant Professor) 108