LOGO

Chapter 7 Artificial Neural Network

Artificial
Intelligence (AI)
Kecerdasan Buatan

Wahyu Setyo Pambudi, ST., MT

Electrical Engineering
International University of Batam
Electrical Engineering Computerized Control and Industrial Instrumentations

Objective
Understanding of artificial neural
networks (ANNs) and fuzzy systems,
and their integration in realizing
information system.
Programming skill to realize
intelligent modern system controls.
Programming language are allowed
Visual Basic, C++, C# and Pascal.

Electrical Engineering Computerized Control and Industrial Instrumentations

Contents
1. Basics of neuroscience and neuron
models.
2. Feedforward and Recurrent neural
networks (NN).
3. ANN applications

References
Principles of neurocomputing for science &
engineering, Fredric M Hum, Ivica Kostanic,
McGraw-Hill Inc., 2001
Neuro-fuzzy and soft computing, JSR Jang,
CT Tsun, E. Mizutani, Prentice Hall Inc.,
1997.
Artificial Intelligence A Guide to Intelligent
Systems Second Edition, Michael
negnevitsky, Addison Wesley., 2005.
Introduction to Neural Networks fo C#, 2nd
Edition, Jeff Heaton, Heaton Research Inc,
2008
Other related book and selected journal
papers

Preview
A neural network is a massively parallel distributed
processor made up of simple processing units, which
has a natural propensity for storing experiential
knowledge and making it available for use. It
resembles the brain in two respects :
1. Knowledge is acquired by the network from its
environment through a learning process.
2. Interneuron
connection
strengths,
known
as
synaptic weight, are used to store the acquired
knowledge

Block Diagram Representation of Nervous System

Electrical Engineering Computerized Control and Industrial Instrumentations

Human Learning Method

I hear, and I forget.

I see, and I remember.

I do, and I understand.

Electrical Engineering Computerized Control and Industrial Instrumentations

Basics of Neuroscience 1
Sistem saraf manusia terdiri
dari jaringan luas elemen
komputasi (neuron) dengan
otak sebagai unit pusat.
Otak terhubung ke reseptor
yang membawa informasi
sensorik, dan memberikan
perintah untuk efektor.
Sebuah neuron biologis terdiri
tiga komponen utama:
dendrite, cell body, and axon.
Fungsi dari sistem saraf
diwakili oleh aktivitas neuron,
topologi koneksi, dan
kekuatan sistem saraf .
Electrical Engineering Computerized Control and Industrial Instrumentations

Basics of Neuroscience 2

Dendrite = input pada sel saraf, input ini mendapatkan sinyal

informasi dari sel saraf yang lain.
Nucleus = tempat untuk memproses sinyal informasi
(memberikan penguatan atau pelemahan terhadap action
potential yang diterima).
Axon = saluran informasi.
Axon Terminal = terminal output dan terhubung dengan sel saraf
yang lain.
Electrical Engineering Computerized Control and Industrial Instrumentations

Basics of Neuroscience 3

Biological neural network

Architecture of a typical
artificial neural network

Analogy between biological and artificial neural networks

Electrical Engineering Computerized Control and Industrial Instrumentations

Basics of Neuroscience 4

Information is coded in action potential.

Action potential (spike) results from a change in ion
concentrations, Na+ and K+ inside and outside membrane.
An action potential is propagated without decay along the
axon
(axon can be viewed as a transmission line).
Speed of propagation (1m/s unmylinated axon, 100m/s
mylinated )
Electrical Engineering Computerized Control and Industrial Instrumentations

Basics of Neuroscience 5

Synapses
The excitation of a neuron
affects other neurons through
contacts of synapses.
Small space between
presynaptic membrane and
postsynaptic membrane is
called synaptic cleft.
Communication between
neurons is facilitated by the
release of small packet
chemicals (neurotransmitter)
into the synaptic cleft.

Electrical Engineering Computerized Control and Industrial Instrumentations

Basics of Neuroscience 6
Classification on NN

Electrical Engineering Computerized Control and Industrial Instrumentations

Basics of Neuroscience 7
Synapse Characteristics
Potential changes cross
postsynaptic membranes
cause signals to propagate
along dendrites and cell
bodies of all neighboring
neurons.
Signal flow is unidirectional from presynaptic neuron to postsynaptic neuron.
Delay (0.5 to 2 ms) caused by mechanism of transmission across synaptic cleft
involved release, diffusion, and reception of neurotrasmitters.
Functioning of the synapse is very sensitive to overall physiological conditions of
the body and to drugs.
Synapse plasticity: facilitates or inhibits chemical synapses.
Synapses differ in shape, size, form, and effectiveness.
Electrical Engineering Computerized Control and Industrial Instrumentations

Basics of Neuroscience 8
Excitation and Inhibition
Excitatory synapse:
excitation of presynaptic
neuron contributes to
depolarization of postsynaptic
neuron.
Inhibitory synapse:
excitation of presynaptic
neuron contributes to
hyperpolarization of
postsynaptic neuron.

Electrical Engineering Computerized Control and Industrial Instrumentations

Basics of Neuroscience 9
Neuron Connection

Electrical Engineering Computerized Control and Industrial Instrumentations

Basics of Neuroscience 10
A neuron with 4 dendrites, each
receiving a signal via a single synapse
from a visual receptor.
Four synapse A,B,C, and D in increasing
distances from axon hillock.
Stimulation in the order ABCD does not
excite the neuron (b).
Stimulation in the order DCBA results
the four peaks reach axon hillock
simultaneously, and excites the
neuron. The neuron can detect the
direction and speed of motion (c.
Geometry / connection pattern has
impact on the function of neuron.

Electrical Engineering Computerized Control and Industrial Instrumentations

Basics of Neuroscience 11
Neural Network vs Computer

Electrical Engineering Computerized Control and Industrial Instrumentations

Historical of Neural Network

Historical notes
McCulloh and Pit, 1943. They presenting 5 assumptions
governing the operations of neurons. No learning, but their
neuron model can act a certain logic function. Their model laid
the foundation for future development in neural networks.
Donald Hebb, 1949. He describes learning process that was
postulated from neurological viewpoint. He stated that
information is stored in the connection of neuron and
postulated a learning strategy for adjustment of connection
weight.
Von Neumann, 1958. He mentioned about importance of
memory in biological system as in the electronic one.
Frank Rossenbelt, 1958. Perceptron, first precisely defined
computationally oriented neural network.
Widrown and Hoff, 1960. LMS learning rule, ADLINE, MADLINE.
Electrical Engineering Computerized Control and Industrial Instrumentations

Historical of Neural Network

Minsky and Papert, 1969. They doubt about limitation and
improvement of perceptron.
Kohonen and Aderson, 1972. Addressable associative
memories.
Werbos, 1974. First description of backpropagation
algorithm.
Lee and Lee, 1975. Fuzzy McCulloh-Pit neuron model.
Grosberg, 1976. Development of physiology of cortical
organization.
Amari, 1976. Pattern Associator.
Hopfiled, 1982, Modern age of neural network, recurrent
neural network
Kohonen, 1982. Self Organizing Map (SOM).
.
Broomhead and Lowe, 1988. Radial basis function in
designing neural net.
Electrical Engineering Computerized Control and Industrial Instrumentations

McCulloh-Pit Neuron
This model was proposed by Warren S
McCulloh and Walter Pitts, 1943.
This neuron model is a very simple twostate device.
Five physical assumptions regarding the
neuron model:
1. Activity of neuron is in binary state.
2. A certain number of fixed neuron
synapse must be excited within a
inhibition state
discrete time step to excite neuron
nw c
3. Delay is the time takes to broadcast the
1 if u
information on the synapse.
yq (k 1)
0 if u
4. Activity of any nonzero inhibitory
nm
synapse
will
absolutely
prevent
u wi xi (k )
excitation of neuron at that discrete
i 1
time step.
set = h w, h:integer
5. Structure of the network is fixed over
y will fire if h w
Electrical Engineering time.
Computerized Control and Industrial Instrumentations

McCulloh-Pits Neuron for logic

function

Electrical Engineering Computerized Control and Industrial Instrumentations

Neuron Models with Continuous

Transfer Characteristics
Outputs yj (j-1..n) of n neuron are inputted to input
n
ith neuron (xi).

xi wij y j i
j 1

Output yi can be determined by unipolar or bipolar

sigmoid function

1
yi

1 exp( xi )

1
n

1 exp[ xi ( wij y j i )]
j 1

2
yi
1
1 exp( xi )

2
n

1 exp[ xi ( wij y j i )]

j 1

Electrical Engineering Computerized Control and Industrial Instrumentations

Model that including time delay and

leakage current

dxi
xi
wij y j I i
C
dt
Ri

Electrical Engineering Computerized Control and Industrial Instrumentations

Electrical circuit model of

neuron

Electrical Engineering Computerized Control and Industrial Instrumentations

Feedforward and Recurrent Neural

Networks
uq is a linear combiner of
input (xj) and synaptic
weight (wqj)

f() activation function

uq wqj x j wqT x xT wq
j 1

Activation potential
Nonlinear model of an ANN

vq wqj x j
j 0

vq uq q
Output of activation
function
n

yq f (vq )

yq f (vq ) f ( wqj x j q )
j 1

Electrical Engineering Computerized Control and Industrial Instrumentations

Activation Functions
A. Linear (identity) function

Activation function dapat

digunakan untuk fungsi
linear maupun nonlinear.
Pemilihan dari activation
function
berdasarkan
permasalahan
khusus
yang akan diselesaikan.

yq f lin (vq ) vq
Electrical Engineering Computerized Control and Industrial Instrumentations

Activation Functions
B1. Hard limiter function

0 if vq 0
yq f hl (vq )
1 if vq 0

B2. Symetrical hard limiter function

1 if vq 0

yq f shl (vq ) 0 if vq 0
1 if v 0
q

Electrical Engineering Computerized Control and Industrial Instrumentations

Activation Functions
C1. Piecewise linear function
(saturating function)

0
if
v

1
1
1

yq f sl (vq ) vq if - vq
2
2
2

1
if
v

C2. Symetric piecewise linear

function

1 if vq 1

yq f ssl (vq ) vq if -1 vq 1
1 if v 1
q

Electrical Engineering Computerized Control and Industrial Instrumentations

Activation Functions
D1. Binary sigmoid function

1
yq fbs (vq )
vq
1 e

Derivative of binary sigmoid function

gbs (vq )

dfbs(vq )
dvq

vq

(1 e

vq 2

fbs (vq )[1 fbs (vq )]

Electrical Engineering Computerized Control and Industrial Instrumentations

Activation Functions
Hyperbolic tangent sigmoid (bipolar
sigmoid) function
v

e q e q
yq f hts (vq ) tanh( vq ) vq
v
e e q
2 v

1 e q

2 v
1 e q

g hts (vq )

df hts
[1 f hts (vq )][1 f hts (vq )]
d vq

Electrical Engineering Computerized Control and Industrial Instrumentations

Assignment
1. Design random data classification
and 4 inputs AND, OR, XOR, XNOR
gate using ANN Error
Backpropagation.
Please submit assignment until March 3th
2014.

Electrical Engineering Computerized Control and Industrial Instrumentations

Format Report
1. Landasan Teori.
2. Metode Penyelesaian.
3. Analisa Program.
4. Kesimpulan.
Format project pada program vs C#
NPM_SKM_(No. Tugas)

Electrical Engineering Computerized Control and Industrial Instrumentations

LOGO

I hear, and I forget.

I see, and I remember.
I do, and I understand.

Electrical Engineering Computerized Control and Industrial Instrumentations