CS 4104

APPLIED MACHINE LEARNING

Dr. Hashim Yasin

National University of Computer
and Emerging Sciences,
Faisalabad, Pakistan.
 Course Details
2

 Credit Hours: 3
 Focus on general fundamental concepts, techniques
and methods for Machine Learning that have been
employed in different types of applications.
 The objective is
 To familiarize the participants with machine learning
algorithms,
 To familiarize with some of the most common questions
regarding analysis of the available data.

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Tentative Marks Distribution
3

Marks (%) Marks (%)

Item Name
(Undergrad.) (Grad.)
Quizzes 10-15 10-15
Assignments /
10-20 10
Project
Research Paper
- 10
Presentation
Mid Exam1 15 15
Mid Exam 2 15 15
Final Exam 40-50 40-50
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Recomended Books
4

Machine Learning
Tom Mitchell

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Recomended Books
5

Pattern Classification
Richard O. Duda,
Peter E. Hart,
David G. Stork

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Contents
6

 Evaluating Hypothesis, Classification, Regression

 Linear & Logistic Regression
 Decision Trees and its variants
 ID3, SLIQ
 Random Forest
 Concept Learning
 Find-S, Candidate Elimination algorithm
 Artificial Neural Network
 Perceptron
 Multilayer networks
 Backpropagation
 Deep learning

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Contents
7

 Radial Basis Function (RBF) Network

 Bayesian Learning
 Bayes theorem
 Naïve Bayes Classifier
 Bayesian Belief Network

 Optimization
 Genetic Algorithm
 Support Vector Machine
 Instance-based Learning
 K-nearest Neighbors

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Contents
8

 Kernel Machines
 Unsupervised Learning
 Clustering
 Reinforcement Learning
 Q-Learning
 Hidden Markov Model
 Dimensionality Reduction
 PCA, KPCA
 Density Estimation
 …
Dr. Hashim Yasin Applied Machine Learning (CS4104)
MACHINE LEARNING
 Machine Learning
10

 Machine Learning is the study of algorithms that

 improve their performance P

 at some task T

 with experience E

 well-defined learning task: <P,T,E>

 Optimize a performance criterion using example data
or past experience

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Machine Learning Application
11

 Text Analysis and NLP

 A computer program automatically translates from Arabic to
English and vice versa.
 A program analyses the text and execute the summary of the
text.

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Machine Learning Application
12

 Document Classification

 Company home page

 Personal home page

 University home page

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Machine Learning Application
13

 Document Retrieval & Recognition

 Spam Detection

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Machine Learning Application
14

 Signal Processing & Speech recognition:

 Speech classification

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Machine Learning Application
15

 Stock Market Prediction

 Weather Prediction

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Machine Learning Application
16

 Face Recognition
Training examples of a person

Test images

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Machine Learning Application
17

 Person Identification

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Machine Learning Application
18

 Action, Gesture or Activity Recognition

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Machine Learning Application
19

 Machine learning is preferred approach to

 Speech recognition
 Natural language processing

 Computer vision

 Medical outcomes analysis

 Robot control

 Computational biology

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Machine Learning Application
20

 This trend is accelerating

 New sensors / IO devices
 Improved data capturing techniques, networking,

 Faster computers with high processing speed

 Software becomes too complex to write by hand

 Improved machine learning algorithms

 Demand for self-customization to user, environment

Dr. Hashim Yasin Applied Machine Learning (CS4104)

FUNCTION APPROXIMATION
 Function Approximation
22

 Given a bunch of examples of

 input

 output

 Find a function which does a good job of expressing

the relationship between them.

 The problem of learning a function from examples,

is complicated.
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Function Approximation
23

 Simplest form: learn a function from examples

f is the target function
An example is a pair (x, f(x))
“Polynomial Game”: Learn polynomial from examples
x f(x) f(x) = ?
1 1
2 4 f(x) = x2
3 9
4 16
5 25

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Function Approximation
24

 Simplest form: learn a function from examples

f is the target function
An example is a pair (x, f(x))
“Polynomial Game”: Learn polynomial from examples
x f(x) f(x) = ?
1 1
2 1
f(x) = (x-4)(x-3)(x-2)(x-1)x+1
3 1
4 1
5 121

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Function Approximation
25

Problem Setting:
 Set of possible instances (instance space with fixed
distribution 𝐷𝑋 ) 𝑋
 A corresponding target space 𝑌

 Unknown target function

𝑓: 𝑋 → 𝑌
 Set of function hypotheses
𝐻 = ℎ ℎ: 𝑋 → 𝑌
Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Function Approximation
26

Input:
 Training examples {𝑥𝑖 , 𝑦𝑖 } of unknown target function

Output:
 Hypothesis ℎ ∈ 𝐻 that best approximates the target
function f

𝑒𝑟𝑟𝑜𝑟𝐷𝑋 ℎ = 𝐸𝐷𝑋 (𝑒𝑟𝑟𝑜𝑟(𝑓 𝑥 , ℎ(𝑥)))

the expected (average) classification error on instances
drawn according to 𝐷𝑋 should be minimal

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Function Approximation
27

 There are following different categories for the

function approximation:

 Memory

 Averaging

 Generalization

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Example
28

When to drive the car? It depends on,

 Temperature

 Expected precipitation

 Day of the week

 Whether need to shop on the way back home

 What are you wearing

Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Memory
29

Temp Precip Day Shop Cloths

80 None Sat No Casual Walk
19 Snow Mon Yes Casual Drive
65 none Tue No Casual Walk

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Memory
30

Temp Precip Day Shop Cloths

80 None Sat No Casual Walk
19 Snow Mon Yes Casual Drive
65 none Tue No Casual Walk
19 Snow Mon Yes Casual ??

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Averaging
31

Temp Precip Day Shop Cloths

80 None Sat No Casual Walk
80 None Sat No Casual Walk
80 None Sat No Casual Drive
80 None Sat No Casual Drive
80 None Sat No Casual Walk
80 None Sat No Casual Walk
80 None Sat No Casual Walk
80 None Sat No Casual ?

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Averaging
32

Temp Precip Day Shop Cloths

80 None Sat No Casual Walk
80 None Sat No Casual Walk
80 None Sat No Casual Drive
80 None Sat No Casual Drive
80 None Sat No Casual Walk
80 None Sat No Casual Walk
80 None Sat No Casual Walk
80 None Sat No Casual Walk

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Generalization
33

Temp Precip Day Shop Cloths

71 None Fri Yes Casual Drive
36 None Sun Yes Casual Walk
62 Rain Weds No Casual Walk
93 None Mon No Casual Drive
55 None Sat No Formal Drive
80 None Sat No Casual Walk
19 Snow Mon Yes Casual Drive
65 None Tues no Casual Walk

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Generalization
34

Temp Precip Day Shop Cloths

71 None Fri Yes Casual Drive
36 None Sun Yes Casual Walk
62 Rain Weds No Casual Walk
93 None Mon No Casual Drive
55 None Sat No Formal Drive
80 None Sat No Casual Walk
19 Snow Mon Yes Casual Drive
65 None Tues No Casual Walk
58 Rain Mon No Casual ??

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Generalization
35

❑ He's going to walk because it's raining today and

the only other time it rained, he walked.
❑ He's going to drive because he has always driven
on Mondays.
❑ He's going to walk because he only drives if he is
wearing formal clothes, or if the temperature is
above 90 or below 20.

The question of which one to choose is hard.

Dr. Hashim Yasin Applied Machine Learning (CS4104)
 Learning Associations
36

 Basket analysis:
P (Y | X ) probability that somebody who buys X also
buys Y where X and Y are products/services.
Example: P ( bread | milk ) = 0.7
TID Items
Market-Basket 1 Bread, Milk
transactions 2 Bread, Cake, Eggs
3 Milk, Cake, Coke
4 Bread, Milk, Cake,
5 Bread, Milk, Cake, Coke

Dr. Hashim Yasin Applied Machine Learning (CS4104)

LEARNING
 Learning
38

 An agent is learning if it improves its performance on

future tasks after making observations about the world.
 Learning is the ability to improve its behavior based on
experience.
 This could mean the following:
 The range of behaviors is expanded;
◼ the intelligent agent can do more.
 The accuracy level to perform tasks is improved;
◼ the intelligent agent can do things in a better way.
 The efficiency in terms of speed is improved;
◼ the intelligent agent can do things faster.

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Learning
39

 Any component of an agent can be improved by

learning from data.

 The improvements may depend on four major factors:

 Which component is to be improved.

 What prior knowledge the agent already has.

 What representation is used for the data and the

component.
 What feedback is available to learn from.

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Inductive Learning
40

 Simplest form: learn a function from examples

𝑓 is the target function

An example is a pair (𝑥, 𝑓(𝑥))

 Problem: find a hypothesis ℎ

such that ℎ ≈ 𝑓
given a training set of examples

 This is a highly simplified model of real learning:

 Assumes examples are given

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Inductive Learning Method
41

 Construct/adjust h to agree with f on training set

 (h is consistent if it agrees with f on all examples)
 E.g., curve fitting:

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Inductive Learning Method
42

 Construct/adjust h to agree with f on training set

 (h is consistent if it agrees with f on all examples)
 E.g., curve fitting:

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Inductive Learning Method
43

 Construct/adjust h to agree with f on training set

 (h is consistent if it agrees with f on all examples)
 E.g., curve fitting:

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Inductive Learning Method
44

 Construct/adjust h to agree with f on training set

 (h is consistent if it agrees with f on all examples)
 E.g., curve fitting:

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Inductive Learning Method
45

 Construct/adjust h to agree with f on training set

 (h is consistent if it agrees with f on all examples)
 E.g., curve fitting:

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Inductive Learning Method
46

 Construct/adjust h to agree with f on training set

 (h is consistent if it agrees with f on all examples)
 E.g., curve fitting:
Ockham’s razor: prefer
the simplest hypothesis
consistent with data

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Inductive Learning Method
47

Ockham’s razor: prefer the simplest

hypothesis consistent with data

Dr. Hashim Yasin Applied Machine Learning (CS4104)

 Acknowledgement
48

Tom Mitchel, Russel & Norvig, Andrew Ng, Alpydin &

Ch. Eick.

Dr. Hashim Yasin Applied Machine Learning (CS4104)