Time Series Analysis and Forecasting

Lesson 03: Time Series Statistical Models

by
Dr. Rajat Kanti Samal
Assistant Professor

Department of Electrical Engineering

Veer Surendra Sai University of Technology
Burla, Odisha – 768018

Dr. R. K. Samal (EE) Lesson 03 Dec 2022 1 / 15

Stochastic Process
The primary objective of time series analysis is to develop mathematical
models that provide plausible descriptions for sample data.
A time series can be defined as a collection of random variables indexed
according to the order they are obtained in time.
This definition is useful to provide a statistical setting for describing
the character of data that seemingly fluctuate in a random fashion.
We may consider a time series as a sequence of random variables,
x1 , x2 , x3 , ..., where the random variable x1 denotes the value taken by
the series at the first time point and so on.
In general, a collection of random variables {xt }, indexed by t is referred
to as a stochastic process. t is typically discrete and vary over integers
t = 0, ±1, ±2, ... or some subset of integers.
The observed values of the stochastic process are generally referred to
as the realization of the stochastic process.
In this text, no notational difference is made for a stochastic process
or its realization, and the term time series is used to refer both.
Dr. R. K. Samal (EE) Lesson 03 Dec 2022 2 / 15
Continuous and discrete time series
Time series is displayed graphically by plotting the random variables on
the vertical axis, or ordinate, with the time scale as the abscissa.
It is usually convenient to connect the values at adjacent time periods
to reconstruct visually some original hypothetical continuous time series
that might have produced these values as a discrete sample.
The approximation of continuous time series by discrete time pa-
rameter series is because of the fact that sampled data will, mostly,
be discrete because of restrictions inherent in the method of collection.
Theoretical development also rest on the idea that a continuous pa-
rameter time series should be specified in terms of finite dimensional
distribution functions defined over a finite number of points in time.
The appearance of data can be changed completely by adopting an
insufficient sampling rate, known as aliasing, leads to distortion.
The fundamental visual characteristics distinguishing different time se-
ries is their degree of smoothness which is induced by supposition that
adjacent points in time are correlated.
Dr. R. K. Samal (EE) Lesson 03 Dec 2022 3 / 15
White Noise

1 A simple kind of generated series is a collection of uncorrelated random

variables wt , with mean 0 and variance σw2 .
2 It is generally used as a model for noise in engineering applications
where it is called white noise and is denoted by wt ∼ wn (0, σw2 )
3 The term white originates from the analogy with white light and signi-
fies that it contains all possible periodic oscillations with equal strength.
4 We will sometimes require the noise to be independent and identically
dis- tributed (iid) random variables denoted as wt ∼ iid(0, σw2 ).
5 A particularly useful white noise series is Gaussian White Noise, where
wt are independent normal variables. It is denoted by wt ∼ iid N(0, σ 2 ).
6 It stochastic behaviour of a time series can be modelled by white noise,
classical statistical methods will suffice.
7 The white noise can be replaced by a moving average (by its current
value and immediate past and future) which smooths the series.

Dr. R. K. Samal (EE) Lesson 03 Dec 2022 4 / 15

Moving Average and Filtering
1 For example, wt can be replaced by
1
vt = (wt−1 + wt + wt+1 ) (1)
3
where vt shows a smoother version of the original series.
2 It reflects the fact that the slower oscillations are more apparent and
some of the faster oscillations are taken out.
3 A linear combination of values in a time series is generally known as a
filtered series.
> library(astsa)
> w = rnorm(500,0,1) # 500 N(0,1) variates
> v = filter(w, sides=2, filter=rep(1/3,3)) # moving average
> par(mfrow=c(2,1))
> plot.ts(w, main="white noise")
> plot.ts(v, ylim=c(-3,3), main="moving average")
Dr. R. K. Samal (EE) Lesson 03 Dec 2022 5 / 15
Moving Average of Gaussian White Noise

Dr. R. K. Samal (EE) Lesson 03 Dec 2022 6 / 15

Autoregression

The white noise series wt can be used in a autoregression model in

the following equation.

xt = xt−1 − 0.9xt−2 + wt (2)

This is called autoregression as it represents a regression or prediction
of current value xt of the time series as a function of past two values
of the series.
A problem with startup values exist with the above equation as the
values of x0 and x−1 is needed for generating succeeding values.
The output series can be seen to have periodic behaviour similar to
speech series. Thus the autoregressive model and its generalizations
can be used as an underlying model for many observed series.

Dr. R. K. Samal (EE) Lesson 03 Dec 2022 7 / 15

Autoregression (cont.)

> w = rnorm(550,0,1) # 50 extra to avoid startup problems

> x = filter(w, filter=c(1,-.9), method="recursive")[-(1:50)]
> plot.ts(x, main="autoregression")

Dr. R. K. Samal (EE) Lesson 03 Dec 2022 8 / 15

Random Walk with Drift

A model for analysing trend is given below with initial condition x0 =0.

xt = δ + xt−1 + wt (3)
The constant δ is called drift. When δ=0, the equation is called a
random walk. This is so because when δ=0, the value of the time
series at time t is the value of series at time t − 1 plus a complete
random movement determined by wt .
The above equation can also be written as
t
X
xt = δt + wj (4)
j=1

The figure shows 200 observations generated from the model with
σw =1, δ= 0 and 0.2.

Dr. R. K. Samal (EE) Lesson 03 Dec 2022 9 / 15

Random Walk with Drift (cont.)

> set.seed(154) # so you can reproduce the results

> w = rnorm(200); x = cumsum(w) # two commands in one line
> wd = w +.2;
> xd = cumsum(wd)
> plot.ts(xd, ylim=c(-5,55), main="random walk", ylab=’’)
> lines(x, col=4); abline(h=0, col=4, lty=2);
> abline(a=0, b=.2, lty=2)
Dr. R. K. Samal (EE) Lesson 03 Dec 2022 10 / 15
Signal in Noise
Many realistic models for generating time series assume an underlying
signal with consistent periodic variations contaminated by adding
random noise.
Consider the model
 
t + 15
xt = 2 cos 2π + wt (5)
50
where t=1,2,...500
It can be noted that the sinusoidal waveform can be written as
Acos(2πωt + ϕ) where A is the amplitude, f is the frequency of
oscillation and ϕ is the phase shift.
In the above signal, A=2, ω = 1/50 (one cycle every 50 time points)
and ϕ = 2π 1550 = 0.6π.
An additive noise term was taken to be white noise with σw =1 and
σw = 5 drawn from normal distribution.
Addition of the noise obscures the signal. The degree to which signal
is obscured depends on the amplitude of the signal and size of σw .
Dr. R. K. Samal (EE) Lesson 03 Dec 2022 11 / 15
Signal in Noise (cont.)
The ratio of the amplitude of the signal to σw is called the signal-to-
noise ratio; the larger the SNR the easier it is to detect the signal.
Spectral Analysis is one of the possible techniques for detecting regu-
lar or periodic signals. In general, a simple additive model xt = st + vt
can be analysed where st denotes some unknown signal and vt denotes
a time series that may be white noise or correlated over time.
The problem of interest is extracting the signal st ; detection, estimation
of waveform and identifying underlying trend or seasonal component.
The above model where the signal has an autoregressive structure forms
the motivation for the state-space model.
cs = 2*cos(2*pi*1:500/50 + .6*pi); w = rnorm(500,0,1)
par(mfrow=c(3,1), mar=c(3,2,2,1), cex.main=1.5)
plot.ts(cs, main=expression(2*cos(2*pi*t/50+.6*pi)))
plot.ts(cs+w, main=expression(2*cos(2*pi*t/50+.6*pi) + N(0,1))
plot.ts(cs+5*w, main=expression(2*cos(2*pi*t/50+.6*pi)+ N(0,25
Dr. R. K. Samal (EE) Lesson 03 Dec 2022 12 / 15
Example of Signal in Noise

Dr. R. K. Samal (EE) Lesson 03 Dec 2022 13 / 15

Summary
In the previous examples, various combinations of random variables is
used for emulating real time series data.
Smoothness characteristics of the observed time series are introduced
by combining random variables in various ways.
Averaging independent random variables over adjacent time points or
looking at the output of difference equations that respond to white
noise inputs are common ways of generating correlated data.
The next aspect is knowing various theoretical measures used for de-
scribing how time series behave.
As in usual statistics, the complete description involves multivariate
distribution function of jointly sampled values.
More economical description can be done in terms of mean and au-
tocorrelation functions. Because correlation is an essential feature of
time series analysis, the most useful descriptive measures are those
expressed in terms of covariance and correlation functions.
Dr. R. K. Samal (EE) Lesson 03 Dec 2022 14 / 15
 Thank You

Dr. R. K. Samal (EE) Lesson 03 Dec 2022 15 / 15