ECE 302: Lecture 4.

1 Probability Density Functions

Prof Stanley Chan

School of Electrical and Computer Engineering

Purdue University

c Stanley Chan 2020. All Rights Reserved.

1 / 20
How to define probability for continuous events?

c Stanley Chan 2020. All Rights Reserved.

2 / 20
Outline

Today’s lecture: Understand probability density functions (PDFs).

Intuition
Definition and properties
Connecting with PMF

c Stanley Chan 2020. All Rights Reserved.

3 / 20
Intuition

How would you define P[{x ∈ A}]?

Measure the size of a set:

“size” of A
P[{x ∈ A}] = . (1)
“size” of Ω

c Stanley Chan 2020. All Rights Reserved.

4 / 20
Example
Suppose that the sample space is the interval Ω = [0, 5] and the event is
A = [2, 3]. To measure the “size” of A, we can integrate A to determine
the length. That is,
R R3
“size” of A A dx dx 1
P[{x ∈ [2, 3]}] = =R = R25 = . (2)
“size” of Ω dx 5
Ω 0 dx

More formally,
R R
A dx dx
P[{x ∈ A}] = R = A
Ω dx |Ω|
Z
1
= dx. (3)
A |Ω|
| {z }
equally important over Ω

c Stanley Chan 2020. All Rights Reserved.

5 / 20
Relax equiprobable

What happens if we want to relax the “equiprobable” assumption?

Replace the constant function 1/|Ω| with fX (x). This will give us
Z
P[{x ∈ A}] = fX (x) dx. (4)
A | {z }
replace 1/|Ω|

If you compare it with a PMF, we note that when X is discrete, then

X
P[{x ∈ A}] = pX (x).
x∈A

c Stanley Chan 2020. All Rights Reserved.

6 / 20
Probability density function

Figure: [Left] A probability mass function (PMF) tells us the relative frequency of
a state when computing the probability. In this example, the “size” of A is
pX (x2 ) + pX (x3 ). [Right] A probability density function (PDF) is the infinitesimal
version of the PMF. Thus, the “size” of A is the integration over the PDF.

c Stanley Chan 2020. All Rights Reserved.

7 / 20
Outline

Today’s lecture: Understand probability density functions (PDFs).

Intuition
Definition and properties
Connecting with PMF

c Stanley Chan 2020. All Rights Reserved.

8 / 20
Definition
Definition
A probability density function fX of a random variable X is a mapping
fX : Ω → R, with the property that
Non-negativity: fX (x) ≥ 0 for all x ∈ Ω
R
Unity: Ω fX (x)dx = 1
R
Measure of a set: P[{x ∈ A}] = A fX (x)dx

c Stanley Chan 2020. All Rights Reserved.

9 / 20
Definition
Definition
Let X be a continuous random variable. The probability density function
(PDF) of X is a function fX : Ω → R, when integrated over an interval
[a, b], yields the probability of obtaining a ≤ X ≤ b:
Z b
P[a ≤ X ≤ b] = fX (x)dx. (5)
a

c Stanley Chan 2020. All Rights Reserved.

10 / 20
Examples

Example 1. Let fX (x) = 3x 2 with Ω = [0, 1]. Let A = [0, 0.5]. Then, the
probability P[{X ∈ A}] is
Z 0.5
1
P[0 ≤ X ≤ 0.5] = 3x 2 dx = . 
0 8

Example 2. Let fX (x) = 1/|Ω| with Ω = [0, 5]. Let A = [3, 5]. Then, the
probability P[{X ∈ A}] is
Z 5 Z 5
1 1 2
P[3 ≤ X ≤ 5] = dx = dx = . 
3 |Ω| 3 5 5

c Stanley Chan 2020. All Rights Reserved.

11 / 20
Can fX (x) ≥ 1?

Yes.
fX (x) is not the probability of having X = x
fX (x) is the probability per unit length
Z x+δ
P[x ≤ X ≤ x + δ] = fX (x)dx ≈ fX (x) · δ. (6)
x

c Stanley Chan 2020. All Rights Reserved.

12 / 20
Examples

Example. Consider a random variable X with PDF fX (x) = 2√1 x for any
0 < x ≤ 1, and is 0 otherwise. We can show that fX (x) → ∞ as x → 0.
However, fX (x) remains a valid PDF because
∞ 1 1
√
Z Z
1
fX (x)dx = √ dx = x = 1. 
−∞ 0 2 x 0

Remark. Since isolated points have zero measure in the continuous space,
the probability of an open interval (a, b) is exactly the same as the
probability of a closed interval:

P[[a, b]] = P[(a, b)] = P[(a, b]] = P[[a, b)].

c Stanley Chan 2020. All Rights Reserved.

13 / 20
Outline

Today’s lecture: Understand probability density functions (PDFs).

Intuition
Definition and properties
Connecting with PMF

c Stanley Chan 2020. All Rights Reserved.

14 / 20
How to write a PMF in terms of a PDF?
Use delta function:
X
fX (x) = pX (xk )δ(x − xk ). (7)
xk ∈Ω

c Stanley Chan 2020. All Rights Reserved.

15 / 20
Examples

Example 1. If X is a Bernoulli random variable with PMF pX (1) = p and

pX (0) = 1 − p, then the corresponding PDF can be written as

fX (x) = pδ(x − 1) + (1 − p)δ(x − 0). 

Example 2.
If X is a binomial random variable with PMF

pX (k) = kn p k (1 − p)n−k , then the corresponding PDF can be written as
n n  
X X n k
fX (x) = pX (k)δ(x − k) = p (1 − p)n−k δ(x − k). 
k
k=0 k=0

c Stanley Chan 2020. All Rights Reserved.

16 / 20
Example

Example 3. Let X be a discrete random variable with PMF

1
pX (k) = , k = 1, 2, . . . ,
2k
The continuous representation of the PMF can be written as
∞ ∞  
X X 1
fX (x) = pX (k)δ(x − k) = δ(x − k).
2k
k=1 k=1

Find the probability P[1 ≤ X ≤ 2].

c Stanley Chan 2020. All Rights Reserved.

17 / 20
Example

P[1 ≤ X ≤ 2]
Z 2 Z ∞ 
2X 
1
= fX (x)dx = δ(x − k)dx
1 1 k=1 2k
Z 2 
1 1
= δ(x − 1) + δ(x − 2) + . . . dx
1 2 4
Z 2
1 2 1 2
Z Z
1
= δ(x − 1)dx + δ(x − 2)dx + δ(x − 3)dx + |{z}
...
2 1 4 1 8 1
| {z } | {z } | {z } =0
=1 =1 =0
1 1 3
= + = .
2 4 4

c Stanley Chan 2020. All Rights Reserved.

18 / 20
Summary

Intuition
Probability is a measure of the size of a set
Use length/area/volume to measure the size of a continuous set
fX (x) is the weight when calculating the size
Definition and properties
Probability per unit length
fX (x) ≥ 1 is okay
Connecting with PMF
PMF is a train of delta function c Stanley Chan 2020. All Rights Reserved.
19 / 20
Questions?

c Stanley Chan 2020. All Rights Reserved.

20 / 20