Chapter 6
Chapter 6
Chapter 6
6.8 6.9
Statistical Averages . . . . . . . . . . . . . . . . . . . . . . . . . . . Joint and Conditional Density Functions . . . . . . . . . . . . . . . . . 6.10.1 Denition and Examples . . . . . . . . . . . . . . . . . . . . .
6.10.2 Ensemble Averages and Stationarity . . . . . . . . . . . . . . . 6.10.3 Time Averages and Ergodicity . . . . . . . . . . . . . . . . . . 6.11 Autocorrelation and Power Spectra . . . . . . . . . . . . . . . . . . . 6.11.1 Autocorrelation and Autocovariance . . . . . . . . . . . . . . . 6.11.2 Power Spectral Density . . . . . . . . . . . . . . . . . . . . . 6.11.3 Deterministic Power and Energy Signals . . . . . . . . . . . . . 6.12 Excitation of LTI Systems with Stationary Random Processes . . . . . . . 6.12.1 Expected Value of the Output Random Process . . . . . . . . . 6.12.2 Autocorrelation Function of the Output Random Process . . . . . 6.12.3 Power Spectral Density of the Output Random Process . . . . . . 6.12.4 Cross-Correlation between Input and Output Random Process . . .
42 43 49 49 53 55 66 66 67 68 70
6.1
Deterministic Signals:
can be expressed in form of a mathematical equation can be reproduced exactly with repeated measurements
Examples for Random Signals:
Number of car accidents in Lubbock during 1 year Quantization noise in an A/D converter Background noise in speech transmission Speckle noise in synthetic aperture radar images Tossing a coin
Life expectance of an engine Outcome of a roulette game Data transmission Bio-medical signals Rolling a dice
Random Experiment: Experiment with random outcome that is repeatable and for which the outcome of each experiment is not predictable.
Sample Space S : Set of all possible outcomes if a random experiment. Examples: 1. Tossing a coin 2. Rolling a die 3. Rolling two dice 4. No. of phone calls per day 5. Life expectance of an engine 6. Angular position of a pointer 7. Quantization error in an ADC
S1 S2 S3 S4 S5 S6 S7
= = = = = = =
{head, tail} {1, 2, 3, 4, 5, 6} S2 S2 ={(1,1),(1,2),. . . ,(6,5), (6,6)} 0 , 1, 2, 3, . . . [0, ) [0, 2 ) [0, Q) or S7 = [Q/2, Q/2)
Tossing a coin: S = {head, tail}, A1 = {head}, A2 = {tail}, A3 = {head, tail} Rolling a die: S = {1, 2, 3, 4, 5, 6}, A1 = {3}, A2 = {2, 4, 6}, A3 = {# 5} Quantization error in an ADC: S = [Q/2, Q/2), A1 = [0, Q/2), A2 = Q/4, A3 = (Q/4, Q/4)
6.2
Probability
Denition: In order for an operator P that assigns a real number, called the probability, to each event A in the sample space S , to be a valid probability assignment, it has to satisfy the following 3 axioms:
0 P (A) 1 P (S ) = 1 if A1 and A2 are mutually exclusive (or disjoint) events, i.e. the occurrence of one outcome precludes the occurrence of the other, then: P (A1 or A2) = P (A1 A2) = P (A1 + A2) = P (A1) + P (A2)
Venn Diagram:
S
A2 A1
P (head) = P (A1) = 0.5, P (tail) = P (A2) = 0.5, P (head or tail) = P (A3) = P (A1) + P (A2) = P (S ) = 1
6.2
Probability
Properties of Probabilities:
) = 1 P (A), P (A P () = 0
P (A1 + A2) = P (A1) + P (A2) P (A1A2) with P (A1A2): probability of the joint occurrence of A1 and A2
S
S A1 A2 A1 A2
P (S ) = P (1 + 2 + 3 + 4 + 5 + 6) =
P (k ) = 1
1 6
3 6
P (Ai) = 1
6.2
Probability
Estimation of Probability P (A): (A) for the probability P (A) that the event A occurs is given by the An estimate P relative frequency of occurrence of A: (A) = NA P N with N : number of times the experiment is repeated NA: number of times the event A occurred The estimate is the more accurate the larger N is. For N the deviation from the probability P (A) generally tends towards zero. Example: Tossing a Coin A fair coin is tossed 100 times, head occurs 51 times, tail occurs 49 times. (head) = 51 , (tail) = 49 P P 100 100
6.2
Probability
Example: Rolling a Fair Die The following plots show the relative frequency of occurrence of the event that the number is 1 or 2 when rolling the die 100 times and 800 times.
result is 1 or 2 1 relative frequence of occurrence relative frequence of occurrence 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0 20 40 60 no. of experiments 80 100 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 0 200 400 600 no. of experiments 800 result is 1 or 2
Compare to:
P (1 + 2) = P (1) + P (2) =
2 = 0.33 6
6.2
Probability
6.3
Conditional Probability P (B |A): Probability of the event B knowing that event A has occurred.
P (BA) = P (B ) P (A|B )
With P (AB ) = P (BA) we obtain Bayes Theorem:
P (B )P (A|B ) P (A)
6.3
10
Example: Binary Data Transmission A binary data source sends a 0 with probability 0.4 and a 1 with probability 0.6. The data is transmitted through an erroneous channel. The probability for a 0 to be received correctly is 0.9, the probability for a 1 to be received correctly is 0.95. In case of an error, a 1 is received instead of a 0 and vice versa.
transmit signal x P(x=0)=0.4 0 erroneous transmission channel P(y = 0|x = 0) = 0.9 receive signal y 0 P(y=0)
P(y = 1|x = 0) = 0.1 P(y = 0|x = 1) = 0.05 P(x=1)=0.6 1 P(y = 1|x = 1) = 0.95 1 P(y=1)
Probability to receive a 0 if a 0 was sent: P (y = 0|x = 0) = 0.9 Probability to receive a 1 if a 0 was sent: P (y = 1|x = 0) = 0.1 Probability to receive a 1 if a 1 was sent: P (y = 1|x = 1) = 0.95 Probability to receive a 0 if a 1 was sent: P (y = 0|x = 1) = 0.05
Dr. Tanja Karp 6.3 Conditional Probability and Statistical Independence 11
Joint Probabilities:
Probability to send a 0 and receive a 0: P (x = 0 y = 0) = P (x = 0) P (y = 0|x = 0) = 0.4 0.9 = 0.36 Probability to send a 0 and receive a 1: P (x = 0 y = 1) = P (x = 0) P (y = 1|x = 0) = 0.4 0.1 = 0.04 Probability to send a 1 and receive a 1: P (x = 1 y = 1) = P (x = 1) P (y = 1|x = 1) = 0.6 0.95 = 0.57 Probability to send a 1 and receive a 0: P (x = 1 y = 0) = P (x = 1) P (y = 0|x = 1) = 0.6 0.05 = 0.03
Probabilities to receive a 0 or 1:
Probability to receive a 0: P (y = 0) = P (x = 0 y = 0) + P (x = 1 y = 0) = 0.36 + 0.03 = 0.39 Probability to receive a 1: P (y = 1) = P (x = 0 y = 1) + P (x = 1 y = 1) = 0.04 + 0.57 = 0.61
6.3
12
Conditional Probabilities (knowing what happens at the receiver): Probability that a 0 was sent if a 0 is received: P (x = 0 y = 0) P (x = 0)P (y = 0|x P (x = 0|y = 0) = = P (y = 0) P (y = 0) Probability that a 1 was sent if a 0 is received: P (x = 1 y = 0) P (x = 1)P (y = 0|x P (x = 1|y = 0) = = P (y = 0) P (y = 0) Probability that a 1 was sent if a 1 is received: P (x = 1 y = 1) P (x = 1)P (y = 1|x P (x = 1|y = 1) = = P (y = 1) P (y = 1) Probability that a 0 was sent if a 1 is received: P (x = 0 y = 1) P (x = 0)P (y = 1|x P (x = 0|y = 1) = = P (y = 1) P (y = 1)
transmit signal x 0 erroneous transmission channel P(x = 0|y = 0) = 0.923
= 0)
= 0.923
= 1)
= 0.077
= 1)
= 0.934
= 0)
= 0.066
P(y=1)=0.61
13
Statistical Independence: Two events A and B are called statistically independent, if P (B |A) = P (B ) and thus:
P (x = i y = j ) = P (x = i) P (y = j ),
i, j = 0, 1
P (x = 0) P (y = 0) = 0.4 0.39 = 0.156 = P (x = 0 y = 0) = 0.36 P (x = 0) P (y = 1) = 0.4 0.61 = 0.244 = P (x = 0 y = 1) = 0.04 P (x = 1) P (y = 1) = 0.6 0.61 = 0.366 = P (x = 1 y = 1) = 0.57 P (x = 1) P (y = 0) = 0.6 0.39 = 0.234 = P (x = 1 y = 1) = 0.03
Dr. Tanja Karp 6.3 Conditional Probability and Statistical Independence 14
6.4
Motivation:
Random Variables
So far, the outcomes of a random experiment were described verbally, e.g. head and tail for tossing a coin. The description of an event A can become long and cumbersome. Random Variable: A random variable X maps each of the possible outcomes of a random experiment to a real number. X : SI R
random experiment real axis X (1 ) X (2 ) 1 * 3 * sample space S * 2 X (3 )
6.4
Random Variables
15
Examples:
Tossing a coin: S = {head, tail} X (head) = 0, Rolling a die: S = {1, 2, 3, 4, 5, 6} X (1) = 1, X (2) = 2, X (3) = 3, X (4) = 4, X (5) = 5, X (6) = 6
Another valid but less intuitive mapping is:
X (tail) = 1
X (1) = 5.6, X (2) = , X (3) = 0.5, X (4) = 34.6, X (5) = 99, X (6) = 56.7 Rolling two dice: S = {(1, 2) : 1, 2 {1, 2, 3, 4, 5, 6}}
Event A: sum of numbers is larger than 8
A = {(1, 2) : 1 + 2 > 8}
Random variable:
X ((1, 2)) = 1 + 2
Probability:
P (A) = P (X > 8)
6.4
Random Variables
16
A = { : [/4, /2)}
Random variable:
X () = /rad
Probability:
6.5
The cumulative distribution function FX (x) associated with a random variable X is dened as: FX (x) = P (X () x) The probability density function (pdf), denoted by p(x) or pX (x), of a random variable X with cumulative distribution function FX (x) is given by:
d p X (x ) = FX (x) or FX (x) = dx
Dr. Tanja Karp
pX (t) dt
17
0 FX (x) 1 for all x I R FX (x1) FX (x2) for all x1, x2 I R with x1 < x2 lim FX (x) = 1,
x x
lim FX (x) = 0
xx0 +0
For a discrete random variable X (i) with associated probabilities P (i), the cumulative distribution function can be expressed as: FX (x) =
:
i
P (i)u(x X (i))
p(x) dx = 1
For a discrete random variable X (i) with associated probabilities P (i), the probability density function can be expressed as: p(x) =
Dr. Tanja Karp
: i=0
P (i) (x X (i))
18
S = {head, tail},
Random Variable:
X (head) = 0,
Cumulative distribution function:
X (tail) = 1
&
FX (x) =
Probability density function:
p(x)
(0.5)
(0.5) x
2
19
Example: Angular Position of a Pointer on a Rotating Wheel Sample Space: S = [0, 2 ) All angles in the range from 0 to 2 are equally likely
&
p(x) =
p(x)dx =
0
p(x)dx = 2K = 1
&
p(x) =
&
FX (x) =
p(x)
FX (x) 1 0.5 x
1/2
20
3 3 P( < X ) = FX ( ) FX ( ) = 2 2 2 2
Probability that the angle is exactly :
3/2
p(t) dt =
/2
1 2
P (X = ) =
p(x) dx = 0
6.6
Denition:
The Histogram
Tabulation of the frequency of occurrence of a random variable within preset ranges. The total number of events is equal to the number of sets (also called cells). The normalized histogram is obtained by dividing the height of each cell by the total number of events. The normalized histogram provides an estimate for the probability density function p(x) of a random variable.
6.6
The Histogram
21
S = {head, tail},
250
X (head) = 0,
X (tail) = 1,
0.7 0.6
0 x
6.6
The Histogram
22
&
S = [0, 2 ),
X (angle) = angle/rad,
p(x) =
Normalization of the histogram for a continuous random variable: N (xi) = xi xi P N where xi denotes a cell of width xi.
Histogram: 400 experiments, 10 cells 50 15 Histogram: 400 experiments, 50 cells
40 10 30
x
20 5 10
Nx 0 1 2 3 x 4 5 6 0 0
3 x
6.6
The Histogram
23
200
60
40
20
0.05
0.05
3 x
3 x
6.6
The Histogram
24
6.7
6.7.1
&
p(x) =
&
1 for a x b ba 0 otherwise FX (x) =
for x < a
p(x) 1 1 (b a) x 0 0.5
FX (x)
25
6.7.2
Lets assume that a random experiment has two mutual exclusive events A and B with probabilities P (A) = p and P (B ) = 1 p = q . The experiment is repeated n times. The random variable X is assigned the value i if event A occurs exactly i times out of the n repetitions. The probability for X = i, i = 0, . . . , n is given by:
P (X = i) = n i
pq
i n i
,
binomial coecient:
n i
n! i!(n i)!
Number of combinations to place i occurrences of event A on n possible positions: n! n(n 1)(n 2) (n i + 1) = (n i)! Out of these there are i! combinations that we cannot distinguish between. Probability density function and cumulative distribution function:
p(x) =
n : i=0
P (X = i) (x i) =
n : n i=0
pq
i n i
(x i)
FX (x) =
n : i=0
P (X = i)u(x i) =
n : n i=0
pq
i n i
u(x i)
26
Example: Rolling a die n times A die is rolled 4 times. Sketch p(x) and FX (x) for the event that a 3 occurs on top of die i = 0, . . . , 4 times. Event A: number on top of die is 3, P (A) = p = 1/6. Event B : number on top of die is not 3, P (B ) = q = 1 p = 5/6. Probability density function:
4 : i=0 4
: 4 i=0
p(x) =
P (X = i) (x i) =
(1/6) (5/6)
4i
(x i)
= = = = =
4! 4 4 (5/6) = (5/6) = 0.482 4! 0! 4! 1 3 1 3 (1/6) (5/6) = 4 (1/6) (5/6) = 0.386 3! 1! 4! 2 2 2 2 (1/6) (5/6) = 6 (1/6) (5/6) = 0.116 2! 2! 4! 3 1 3 1 (1/6) (5/6) = 4 (1/6) (5/6) = 0.015 1! 3! 4! 4 4 3 (1/6) = (1/6) = 7.71 10 4! 0!
27
FX (x)
6.7.3
The exponential distribution depends on a parameter t. It approximates the binomial distribution for large n. ti P (X = i) = exp(t) i! Probability density function and cumulative distribution function:
p(x) =
:
i
ti exp(t) (x i), i!
F X (x ) =
:
i
ti exp(t) u(x i) i!
Examples: number of particles emitted over a time duration in radioactive decay, number of telephone calls arriving at a switching venter during a certain time period, binary data transmission with low error rates
Dr. Tanja Karp 6.7 Some Important Probability Distributions 28
6.7.4
&
for x < 0
&
for x < 0
FX (x) =
1 exp(xt) for x 0
cumulative distribution function
0.8
0 1
Examples: characterization of waiting times for processes without aging, e.g. time period between two calls at a calling center, time to repair a defect component of a system
Dr. Tanja Karp 6.7 Some Important Probability Distributions 29
6.7.5
The Gaussian distribution depends on parameters m I R and > 0. It is also denoted as N (m, 2) distribution.
p(x) =
x
1 2 1
exp
(x m) 2 2
exp
FX (x) =
probability density function 0.8 0.7 0.6 0.5 0.4 0.3 0.2 m=1, =3 m=1, =1 m=3, =0.5
(y m) 2 2
dy
0.8
p(x)
0.1 0 5 4 3 2 1 0 x 1 2 3 4 5
30
The integral for the cumulative distribution function FX (x) cannot be evaluated in closed form and requires numerical evaluation. Normalized Gaussian Distribution:
m+k
P (X (m + k )) =
1 2
exp
(y m) 2 2
dy
Substitute z = (y m)/ :
P (X (m + k )) =
1 exp 2
z 2
dz
Q(x) =
x
1 exp 2
z 2
dz
we obtain
P (X (m + k )) = 1 Q(k)
A random variable has a Gaussian distribution if it depends on many independent random variables with identical distribution. Examples: Electrical energy consumption in a metropolitan area, noise in a transmission channel, arbitrary measurement errors
Dr. Tanja Karp 6.7 Some Important Probability Distributions 31
6.8
Statistical Averages
A random variable is completely determined by its probability density function p(x) or its cumulative distribution function FX (x), respectively. In many cases p(x) and FX (x) are dicult to estimate and knowledge about the average outcome of a random experiment is sucient (e.g. average life expectance of an engine, average time spent in a waiting queue, average gain/loss of a slot machine). Expected Value E {X }: For a random variable X with probability density function p(x), the expected value E {X } = mX is given by:
E {X } =
x p(x)dx = mX
If X is a discrete random variable that takes the value xi, i = 1, 2 . . . N < , with probability P (X = xi), then
p(x) =
N : i=1
P (X = xi) (x xi),
E {X } =
N : i=1
xi P (X = xi)
The expected value is also called mean value, average value, or rst moment.
6.8
Statistical Averages
32
x p(x)dx
A knowledge of all the moments of a random variable, provided they are nite, species a random variable as completely as a knowledge of the probability density function. The second moment is called the mean-square value of X and its square root is called the root mean square (rms) value of X .
(x mX ) p(x)dx
The second order central moment E {(X mX )2} is called the variance of a random variable X . It measures the average quadratic deviation of X from its average value mX .
2 X
= E {|(X mX )| } =
(x mX ) p(x)dx
X is called the standard deviation of X and is an indicator for the eective width of the probability density function (the larger X the more spread out is the pdf of X ).
Chebyshevs Inequality:
33
S = {head, tail},
X (head) = 0,
X (tail) = 1,
p(x) (0.5) (0.5)
Average value mX :
2 Variance X :
E {X } =
x p(x)dx
E {(X mX ) } =
(x 0.5) p(x) dx
= 0.5
= 0.5
(x 0.5) (x) + (x 1) dx
2 2
= 0.25
6.8
Statistical Averages
34
S = [0, 2 ),
Probability density function:
X (angle) = angle/rad
p(x) 1/2
&
p(x) =
0
2 Variance X :
Average value mX :
E {X } =
x p(x)dx
2
E {(X mX ) } =
0
(x ) p(x) dx 1 2 3
1 = 2
x dx =
0
1 2
(x ) dx =
0
Standard deviation: X =
1 3
6.8
Statistical Averages
35
Y = g (X ),
with X , Y : random variables
E {Y } = E {g (X )} =
g (x)p(x)dx
Repeatedly taking the mathematical expectance: E {E {X }} = E {X } Properties of the variance: X = E {(X mX ) } = E {X } mX Y = aX + b, Y = E {(Y mY ) } = a X
2 2 2 2 2 2 2 2
6.9
In many cases a random experiment depends on several random variables. Example: Determine how the likelihood to be overweight ed depends on the age of a person.
random experiment: choosing one person
height X1 (1 ) X1 (2 )
weight
age
X2 (1 ) X2 (3 ) X2 (2 )
X3 (2 )
1 * 3 *
* 2
X1 (3 )
X3 (3 ) X3 (1 )
sample space S
FX,Y (x, y ) = P (X x, Y y )
where X and Y are two random variables dened on the same sample space S .
37
FX,Y (x, y ) = P (X x, Y y ) =
pX,Y (t, s) ds dt
where X and Y are two random variables dened on the same sample space S . Marginal Probability Function:
pX (x) =
p Y (y ) =
pX,Y (x, y ) dx
FX (x) = P (X x, Y ) =
y
pX,Y (t, s) ds dt =
y
pX (t) dt
F Y (y ) = P (X , Y y ) =
pX,Y (t, s) ds dt =
pY (s) ds
38
39
6.10
6.10.1 Denition and Examples
Random Processes
random variable
random experiment X (t , 1 )
1 1
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
* 1 * i * 3 * 2
* X (t , ) 2
0.5 0 0.5 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2
* X (t , 3 )
1 0.5 0 0.5 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2
sample space S
X (t , i )
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
t1
Dr. Tanja Karp 6.10 Random Processes
t2
40
Random Process: A random process X (t) describes the mapping of a random experiment with sample space S onto an ensemble of sample functions X (t, i). For each point in time t1, X (t1) describes a random variable. Example: Rolling a Die Random variable: X = i if number i is on top of the die Random process: Y (t) = X cos(0t). Example: Tossing a Coin N Times Random variable: Xn = 0 if the nth result is head, Xn = 1 if the nth result is tail Random process: Y (t) =
Y (t , 1 ) t Y (t , 1 ) t Y (t , 1 ) t Y (t , 4 ) t
Dr. Tanja Karp 6.10 Random Processes
2N
n=1
Xn rect(t n + 0.5).
Y (t , 2N ) t
41
h( )X (t )d
For each time instance of a random process, the average value, variance etc. can be calculated from all sample functions X (t, i). Expected Value E {X (t)}: For a random process X (t) with probability density function pX (t)(x), the expected value E {X (t)} = mX (t) is given by:
E {X (t)} =
x pX (t)(x)dx = mX (t)
Variance X (t):
|x mX (t)| pX (t)(x)dx
Random Processes
42
For a stationary random process the probability density function is independent of time t, thus the expected value and the variance are also a constant over time.
t, t0
X (t) = X (t + t0) = X
So far, the average value and the variance of a random process X (t) were calculated based on the probability density function pX (t). However, in practical experiments the probability density function of a random process is often unknown. Also, in many cases, there is only one sample function X (t, i) available. Therefore, it is favorable to average over time instead of taking the ensemble average. Average Value mX (i):
mX (i)
2 Variance X ( ) : i 2 X ( ) i
1 = lim T T
T /2
X (t, i)dt
T / 2
1 = lim T T
6.10
T /2 T / 2 i
(X (t, i) mX ) dt
Random Processes
43
ensemble average
random experiment X (t , 1 )
1 1
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
* 1 * i * 3 * 2
* X (t , ) 2
0.5 0 0.5 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2
* X (t , 3 )
1 0.5 0 0.5 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2
sample space S
X (t , i )
time average
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2
t1
t2
6.10
Random Processes
44
Ergodicity: A stationary random process X (t) is called ergodic, if the time averages of each sample function X (t, i) converge towards the corresponding ensemble average with probability one. In practical applications ergodicity is often assumed since just one sample function is available and therefore the ensemble averages cannot be taken. Example 1: Random process: X (t) = A cos(0t) A: discrete random variable with P (A = 1) = P (A = 2) = 0.5 0: constant Ensemble average:
6.10
Random Processes
45
Example 2: Random process: X (t) = A A: discrete random variable with P (A = 1) = P (A = 2) = 0.5 Ensemble average:
mX (1)
1 = lim T T 1 = lim T T
T /2
T /2
1 dt = 1
T / 2 T /2
mX (2)
2 dt = 2
T / 2
time averages taken for dierent sample functions are not identical to the ensemble average, the random process is thus not ergodic.
6.10
Random Processes
46
Example 3: Tossing a Coin N Times Random variable: Xn = 0 if the nth result is head, Xn = 1 if the nth result is tail. Random process: Y (t) =
Y (t , 1 ) t Y (t , 1 ) t Y (t , 1 ) t Y (t , 4 ) t Y (t , 2N ) t
N 2 n=1
Xn rect(t n + 0.5)
Ensemble average:
m Y ( i )
1 = N
Y (t, i)dt =
0
1 n2 (n1 0 + n2 1) = N N
for N , n1 and n2 converge towards N/2 and mY (t,i) = mY (t) = mY . The random process is thus ergodic.
Dr. Tanja Karp 6.10 Random Processes 47
6.10
Random Processes
48
6.11
6.11.1
We are interested in how the value of a random process X (t) evaluated at t2 depends on its value at time t1. At t1 and t2 the random process is characterized by random variables X1 and X2, respectively. The relationship between X1 and X2 is given by the joint probability density function
X (t , 1 )
1 1 0.5
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
X (t , 2 )
0 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2
0.5 1
X (t , 3 )
0.5 0 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2
0.5
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
t1
t2
= E {X (t1) X (t2)} =
Autocovariance Function:
(x1 mX (t1))(x2 mX (t2)) pX (t1)X (t2)(x1, x2)dx1dx2 = RXX (t1, t2) mX (t1)mX (t2)
Autocorrelation and Autocovariance Function of a Stationary Random Process: The joint probability density function of a stationary process does not change if a constant value t is added to both t1 and t2.
=
Dr. Tanja Karp
6.11
50
Since the average value is a constant, the autocovariance function is given by:
Symmetry: RXX ( ) = RXX ( ) Mean Square Average: RXX (0) = E {X (t)2} 0 Maximum: RXX (0) |RXX ( )| Periodicity: if RXX (0) = RXX (t0), then RXX ( ) is periodic with period t0.
6.11
51
Wide Sense Stationary (WSS) Random Process: A random process X (t) is called WSS if the following three properties are satised:
The average value of the random process is a constant: mX (t) = mX The autocorrelation and autocovariance function only depend on the time dierence = t1 t2: RXX (t1, t2) = RXX (t2 t1) = RXX ( ) CXX (t1, t2) = CXX (t2 t1) = CXX ( )
2 The variance is a constant and nite: X = CXX (0) = RXX (0) m2 X <
1 RXX ( ) = lim T T 1 T T
T /2
T /2
CXX ( ) = lim
XT (t, i): sample function of random process X (t) windowed to be of length T (starting at T /2 ending at T /2).
Dr. Tanja Karp 6.11 Autocorrelation and Power Spectra 52
6.11.2
Motivation:
Description of random processes in the frequency domain Calculation of the Fourier Transform of a sample function is not useful We assume in the following that the random process considered is at least WSS if not stationary.
The power spectral density (psd) of a WSS random process X (t) is dened as the Fourier Transform of the autocorrelation function RXX ( ):
SXX ( ) = F{RXX ( )} =
RXX ( ) exp(j )d
Inverse transform:
1 RXX ( ) = 2
SXX ( ) exp(j ) d
SXX ( ) = SXX ( ),
Dr. Tanja Karp 6.11
SXX ( ) 0,
Autocorrelation and Power Spectra
Im{SXX ( )} = 0
53
Rxx( ) = lim
1 T T
1 T T
xT (t)xT (t + )dt
T / 2
Sxx( ) =
Rxx( ) exp(j )d
T /2
1 = lim T T 1 = lim T T
T /2
xT (t)
T / 2 T /2
xT (t + ) exp(j )d dt
XT ( ) exp(jt)
1 = XT ( ) lim T T
xT (t) exp(jt) dt
T / 2
6.11
54
6.11.3
Power Signal: The autocorrelation function and the power spectral density can also be calculated for a deterministic power signal f (t). In this case, the signal simply replaces the random process. With fT (t) = f (t) rect(t/T ) we obtain for the autocorrelation function:
1 Rf f ( ) = lim T T
and for the power spectral density:
T /2
fT (t)fT (t + )dt
T / 2
Sf f ( ) =
1 P = Rf f (0) = lim T T
T /2
1 2 fT (t)dt = 2
Sf f ( )d
T / 2
6.11
55
Energy Signals: For an energy signal f (t) an energy autocorrelation function can be dened as
Rf f ( ) =
f (t)f (t + )dt = f ( ) f ( )
Applying the Fourier Transform to the energy autocorrelation function, we obtain the energy spectral density as:
Sf f ( ) =
Rf f ( ) exp(j )d = F ( )F ( ) = |F ( )|
E = Rf f (0) =
f (t) dt =
1 2
Sf f ( )d =
1 2
|F ( )|
6.11
56
6.11.4
X (t) = A sin(0t + ) A, 0: constant values, : random variable with probability density function p(x): p(x) =
Average value:
The average value is a constant and independent of t. Since mX = 0 the autocorrelation and autocovariance functions are identical.
Dr. Tanja Karp 6.11 Autocorrelation and Power Spectra 57
Autocorrelation function:
RXX (t1, t2) = E {0.5A cos(0(t2 t1)) 0.5A cos(0(t2 + t1) + 2)} = 0.5A E {cos(0(t2 t1))} 0.5A E {cos(0(t2 + t1) + 2)} = 0.5A cos(0(t2 t1)) 0 = RXX ( )
The autocorrelation function only depends on = t2 t1 but not on the absolute values of t1 and t2. Power spectral density:
2 2 2
X (t, i) = A sin(0t + i)
Average value (time average):
mX (i)
1 = lim T T 1 = lim T T
T /2
X (t, i)dt
T / 2 T /2
A sin(0t + i)dt = 0 = mX
T / 2
RX (i)X (i)( ) = = = =
T /2 1 lim X (t, i)X (t + , i)dt T T T / 2 T /2 1 2 A sin(0t + i) sin(0(t + ) + i)dt lim T T T / 2 T /2 A2 1 lim cos(0 ) + cos(0(2t + ) + 2i)dt 2 T T T /2 A2 cos(0 ) = RXX ( ) 2
Time averages of one sample function and ensemble averages are identical the random process is ergodic.
Dr. Tanja Karp 6.11 Autocorrelation and Power Spectra 59
A = 1, 0 = 2
Sample Functions 1 X(t,1) 0 1 1 X(t,2) 0 1 1 X(t,3) 0 1 1 X(t,4) 1.5 0 1 1 X(t,5) 0 1 0.5 1 2 0.5 Autocorrelation Function R
XX
()
5 0
5 0.5 3 2 1 0 1 2 3
2 t in sec
0 10
10
6.11
60
Example 2: Binary Data Transmission A binary sequence is transmitted by rectangular pulses of width Tb. The amplitude of the pulse is determined by each bit, i.e. it is one if the bit is one and zero if the bit is zero. We assume that ones and zeros are equally likely and that each bit is statistically independent of all others. Using ergodicity, we obtain the following results from a sample function X (t, 1): Average value (sample function of length N bits with n1 ones):
mX
T 0
N Tb 0
X (t, 1)dt
6.11
61
Autocorrelation Function R
XX
()
0.5 0 5 10 t/Tb 15 20
0.25
Shifted Copies of Sample Function 1.5 1 0.5 0 0.5 1.5 1 0.5 0 0.5 1.5 1 0.5 0 0.5 X(t+0.3T ,1)
b
0 5
0 /Tb
X(t+T ,1)
0.5 0 5 10 15 20 0.25
X(t+1.5T ,1)
10 t/Tb
15
20
0 15
10
0 Tb
10
15
6.11
62
f (t)f (t + )dt = ( )
Sf f ( ) = F{Rf f ( )} = Sa (/2)
Autocorrelation Function Power Spectral Density
10
10
15
6.11
63
Example 4: White Noise A random process n(t) is called white noise, if it has a constant power spectral density of /2 watts per Hz measured over positive frequencies. If in addition the random process has zero mean (mn = 0), the power spectral density is given by:
Snn( ) = /2
Autocorrelation function:
1
for all
Rnn( ) = F
{Snn( )} =
(t) 2
Since only the rst and second moment of the process are known, the probability density function cannot be uniquely determined. In the case of a Gaussian probability density function, the process is called white Gaussian noise. If the white Gaussian noise is added to the signal, we denote it as additive white Gaussian noise (AWGN).
6.11
64
1 2
5 t
10
Rnn () (/2)
Snn () /2
6.11
65
6.12
Excitation of an LTI system with sample function x(t, i) of a stationary random process x(t).
x(t , i )
y(t , i )
&
my = E {y (t)} = E {x(t) h(t)} = E
'
x(t )h()d
E {x(t )} h( )d =
6.12
66
6.12.2
&
'
=E
h( )x(t )d
h()x(t + )d
=
=
h( )h()Rxx( + )dd
h( )h( + )d Rxx( )d
R E ( ) hh
6.12.3
Rhh( ) = h( ) h( ) h( ) H ( ),
E
h( ) H ( )
2
Rhh( ) H ( )H ( ) = |H ( )|
and thus
2
Syy ( ) = Sxx( ) |H ( )|
Rxx ()
x(t , i )
y(t , i )
Ryy () = Rxx () RE hh ()
Sxx ()
6.12
68
Example: Ideal Lowpass Filtering of White Noise Input random process ni(t):
HLP ( ) =
Output random process no(t):
P no =
1 2
Snono ( )d =
1 2
0.5d =
c
c = fc 2
Sno no ()
/2 c c
Dr. Tanja Karp 6.12
HLP () 1 Sni ni ()
/2 c
c
69
6.12.4
The autocorrelation function describes the statistical properties of two random variables X1 and X2 taken from the same random process at times t1 and t2, respectively, X1 = X (t1) and X2 = X (t2). The cross-correlation function describes the statistical properties of two random variables X1 and Y2 taken from two dierent random processes X (t) and Y (t) (here input and output of an LTI system) at times t1 and t 2, respectively, such that X1 = X (t1) and Y2 = Y (t2). It is dened as:
RXY ( ) = E {X (t) Y (t + )}
Two random processes X (t) and Y (t) are called uncorrelated if
6.12
70
Here:
&
'
=E
x(t)
h()x(t + )d
h()E {x(t)x(t + )} d
h()Rxx( )d = h( ) Rxx( )
Example: System Identication An LTI system with unknown impulse response is excited with a white noise random process ni(t) with power spectral density Snini = /2. The output noise process is no(t). The cross-correlation between input and output noise process is given by: Rnino ( ) = h( ) Rni,ni ( ) = h( ) ( ) = h( ) 2 2
6.12
71