Pattern Recognition: Statistical and Neural: Lonnie C. Ludeman

Nanjing University of Science & Technology
Pattern Recognition:
Statistical and Neural
Lonnie C. Ludeman
Lecture 24
Nov 2, 2005
1
Lecture 24 Topics
1. Review and Motivation for Link Structure

2. Present the Functional Link Artificial Neural
Network.
3. Simple Example- design using ANN and FLANN
4. Performance for Neural Network Designs
5. Radial Basis Function Neural Networks
6. Problems, Advantages, Disadvantages, and
promise of Artificial Neural Network Design
2
Generalized Linear Discriminant Functions
w1
g1(x) x
w2
g2(x) x
… wj
x
gj(x) + g(x)
x
…
wM
gM(x) x
Review 1 3
Patterns are linearly separated in the 3-dim space
Separating plane
4
Review 2
Example: Decision rule using one nonlinear
discriminant function g(x)
Given the following g(x) and decision rule
Illustrate the decision regions R1 and R2

where we respectively classify as C1 and C2
for the decision rule above
Review 3 5
Solution:
R1 decide C1
R2 decide C2
every where else
every where else
Review 4 6
Find a generalized linear discriminant
function that separates the classes
Solution:
d(x) = w1f1(x)+ w2f2(x)+ w3f3(x)
+ w4f4(x) +w5f5(x) + w6f6(x)
T
= w f (x)
in the f space (linear)
Review 5 7
where
in the original pattern space: (nonlinear)
Review 6 8
Decision Boundary in original pattern space
x2
from C1
2
from C2
1 2 3 4 x1
-1
Boundary
-2 d(x) = 0
Review 7 9
Potential Function Approach – Motivated
by electromagnetic theory + from C1
- from C2
Sample space
Review 8 10
Plot of Samples from the two classes
Review 9 11
Given Samples x from two classes C1 and C2
S1 C1
S2 C2
Define Total Potential Function

K(x) = ∑ K(x, xk) - ∑ K(x, xk)
x k C S1 xk C S2
Potential
Function
Decision Boundary
K(x) = 0
Review 10 12
Algorithm converged in 1.75 passes through
the data to give final discriminant function as
Review 11 13
Functional Link Neural Network
14
Quadratic Functional Link
15
Fourier Series Functional Link
16
Principal Component Functional Link
fk(x), k=1 to N are chosen as

the eigen vectors of the
sample covariance matrix
17
Example: Comparison of Neural Net and
functional link neural net
Given two pattern classes C1 and C2 with the
following four patterns and their desired outputs
18
(a)Design an Artificial Neural Network to
classify the two patterns given
(b) Design a Functional Link Artificial
Neural Network to classify the
patterns given.
(c) Compare the Neural Net and
Functional Link Neural Net designs
19
(a) Solution: Artificial Neural Net Design
Select the following structure
20
After training using the training set and the
backpropagation algorithm the design
becomes
Values determined
by neural net
21
(b) Solution: Functional Link
Artificial Neural Net Design
22
A neural net was trained using the functional
link output patterns as new pattern samples
The resulting weights and structure are
23
(c) Comparison Artificial Neural Net (ANN)
and Functional Link Artificial Neural Net
(FLANN} Designs
FLANN has simpler structure than the

ANN with only one neural element and
Link.
Fewer iterations and computations in the

training algorithm for FLANN.
FLANN design may be more sensitive to

errors in patterns. 24
Determining Performance of Neural Net Design
on Training Set
25
Determine Performance for
Design using Training Set
Classify each member of the training

set using the neural network design.
Test Design
on Testing Set
Classify each member of the testing

set using the neural network design.
26
Could use
(a) Performance Measure ETOT
(b) The Confusion Matrix
(c) Probability of Error
(d) Bayes Risk
27
(a) Local and global errors- Used in
Neural Net Design procedure
Local Measure
Global Measure
28
(b) Confusion Matrix- Example
Correct Incorrect
Classification Classification
29
(c) Probability of Error- Example
Estimates of Probabilities of being Correct
Estimate of Total Probability of Error
30
(d) Bayes Risk Estimate
31
Radial Basis Function (RBF)
Artificial Neural Network
Functional
Link
32
Functional Form of RBF ANN
where
Examples of Nonlinearities
33
Design Using RBF ANN
Let F(x1, x2, … , xn) represent the function we
wish to approximate.
For pattern classification F(x) represents the

class assignment or desired output (target value)
for each pattern vector x a member of the training set
Define the performance measure E by
E
We wish to Minimize E by selecting
M, a ,b1, b2, ... , bM and z1, z2, ... zM
34
Finding the Best Approximation using RBF ANN
Usually broken into two parts

(1st ) Find the number M of prototypes and
the prototypes { zj : j=1, 2, ... , M} by using a
clustering algorithm(Presented in Chapter 6)
on the training samples
(2nd ) With these fixed M and { zj: j=1,2, ... , M}
find the a ,b1, b2, ... , bM that minimize E.
Notes:
You can use any minimization procedure you wish.
Training does not use the Backpropagation Algorithm
35
Problems Using Neural Network Designs
Failure to converge
Selection of insufficient structure
Max iterations too small
Lockup occurs
Limit cycles
Good performance on training set –

poor performance on testing set
Training set not representative of variation
Too strict of a tolerance - “grandmothering”
36
Advantages of Neural Network Designs
Can obtain a design for very complicated

problems.
Parallel structure using identical elements

allows hardware or software implementation
Structure of Neural Network Design

similar for all problems.
37
Other problems that can be solved using
Neural Network Designs
System Identification
Functional Approximation
Control Systems
Any problem that can be placed in the

format of a clearly defined desired output
for different given input vectors.
38
Famous Quotation
“Neural network designs are the second
best way to solve all problems”
39
Famous Quotation
??????????
??????????
40
Famous Quotation
??????????
??????????
The promise is that a Neural Network can
be used to solve all problems; however,
with the caveat that there is always a better
way to solve a specific problem.
41
So what is the best way to
solve a given problem ???
42
43
?
A design that uses and
understands the structure
of the data !!!
44
Summary Lecture 24
1. Reviewed and Motivated Link Structure

2. Presented the Functional Link Artificial Neural
Network.
3. Presented Simple Example with designs using
ANN and FLANN
4. Described Performance Measures for Neural
Network Designs
5. Presented Radial Basis Function Neural
Networks
45
6. Discussed Problems, Advantages,
Disadvantages, and the Promise of
Artificial Neural Network Design
46
End of Lecture 24
47

Pattern Recognition: Statistical and Neural: Lonnie C. Ludeman

Uploaded by

Copyright:

Available Formats

Pattern Recognition: Statistical and Neural: Lonnie C. Ludeman

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Pattern Recognition: Statistical and Neural: Lonnie C. Ludeman

Uploaded by

Copyright:

Available Formats

Nanjing University of Science & Technology

1. Review and Motivation for Link Structure

Given the following g(x) and decision rule

Illustrate the decision regions R1 and R2

every where else

every where else

in the f space (linear)

in the original pattern space: (nonlinear)

Define Total Potential Function

fk(x), k=1 to N are chosen as

Select the following structure

The resulting weights and structure are

FLANN has simpler structure than the

Fewer iterations and computations in the

FLANN design may be more sensitive to

Classify each member of the training

Classify each member of the testing

Estimate of Total Probability of Error

For pattern classification F(x) represents the

Usually broken into two parts

Good performance on training set –

Can obtain a design for very complicated

Parallel structure using identical elements

Structure of Neural Network Design

Any problem that can be placed in the

1. Reviewed and Motivated Link Structure

You might also like