Associative Memory

Network
• An associative memory network can
store a set of patterns as memories
• When the associative memory is being
presented with a key pattern, it
responds by producing one of the
stored patterns, which closely
resembles or relates to the key pattern.
• The recall is through association of the
key pattern, with the help of
information memorized. If the given memory is content
addressable, the incorrect string
• These types of memories are also "Albert Einstein" as a key is
sufficient to recover the correct
called as content- addressable name "Albert Einstein."
memories (CAM)
• Conventional memory in computers
– Address based memory
• The aim of an associative memory is, to produce
the associated output pattern whenever one of
the input pattern is applied to the neural network.
• Associative memory
– Content based memory
– Keyword based memory
– Pattern based memory
• Input data is correlated with that of the stored data in the CAM.
• The stored patterns must be unique, i.e., different patterns in each
location.
• Associative memory makes a parallel search within a stored data file.

CAM architecture
Two types of associative memories
• Auto-associative memory network
• Hetero-associative memory network

• Both these nets are single layer nets in which the weights
are determined in a manner that the net stores a set of
pattern associations.
• Each of this association is an input-output vector pair, s:t.
• If each of the output
vectors is same as the
input vectors with which
it is associated, then the
net is a said to be auto
associative memory net.

• If the output vectors are

different from the input
vectors then the net is
said to be hetero
associative memory net.
Training Algorithms for Pattern Association

• Hebb Rule
• Outer Products Rule
Hebb Rule
• It is widely used for finding the weights of an associative
memory neural net.
• The weights are updated until there is no weight change.
• It can be used with patterns that are being represented as
either binary or bipolar vectors.
Outer Products Rule
• Outer products rule is an alternative method for finding
weights of an associative net.
Outer product.
Let s and t be row vectors.
Then for a particular training pair s:t

s1  s1t1......s1tm  w11......w1m 

   
s2t1......s2tm  
W( p) = s ( p)t( p) =   t1,......tm = 
T
= 
     
sn snt1......sntm  wn1......wnm 
Auto-associative memory network
• For an auto-associative net, the training input and target
output vectors are identical.
• The process of training is often called storing the vectors,
which may be binary or bipolar.
• The performance of the net is judged by its ability to
reproduce a stored pattern from noisy input; performance is,
in general, better for bipolar vectors than for binary vectors.
• This type of memory network is commonly used in
applications such as speech and image recognition, where
the input data may be incomplete or noisy.
Architecture

• The input layer consists of n input units and the output layer also
consists of n output units
• The training input and target output vectors are the same.
Training Algorithm
Testing Algorithm
Applications
• Speech processing
• Image processing
• Pattern classification, etc.
❑ An autoassociative net to store one vector:
recognizing the stored vector.
❑ Step 0. The vector s = (1, 1, 1, - 1) is stored with the
weight matrix:

❑ Step 1. For the testing input vector:

– Step 2. x = (1, 1, 1, - 1).
– Step 3. y-in = (4, 4, 4, - 4).
– Step4. y = f ( 4 , 4 , 4 , - 4 ) = ( 1 , 1 , 1 , - 1 ) .
❑ The preceding process of using the net can be
written more succinctly as:

❑ As before, the differences take one of two forms:

"mistakes" in the data or "missing" data.
❑ The only "mistakes" we consider are changes from +
1 to - 1 or vice versa.
❑ We use the term "missing" data to refer to a
component that has the value 0, rather than either +
1 or -1
❑ Testing an autoassociative net: one mistake in the
input vector.
❑ The reader can verify that the net also recognizes
the vectors formed when one component is
"missing."
❑ Those vectors are (0, 1, 1, -1), (1, 0, 1, - 1), (1, 1, 0,
- 1), and (1, 1, 1, 0).
❑ In general, a net is more tolerant of "missing" data
than it is of "mistakes“ in the data, as the examples
that follow demonstrate.
❑ Testing an autoassociative net: two "missing" entries
in the input vector.
❑ The vectors formed from (1, 1, 1, - 1) with two
"missing" data are (0, 0, 1, - 1),(0, 1, 0, -1), (0, 1, 1,
0), (1, 0, 0, -1), (1, 0, 1, 0), and (1, 1, 0, 0).
❑ Testing an autoassociative net: two mistakes in the
input vector
❑ The vector (- 1, - 1, 1, - 1) can be viewed as being
formed from the stored vector (1, 1, 1, - 1) with two
mistakes (in the first and second components).
❑ We have: (-1, -1, 1, -1).W = (0,0,0,0).
❑ The net does not recognize this input vector.
Hetero-associative memory network
• In case of a hetero associative neural net, the training input and the target output
vectors are different.
• The weights are determined in a way that the net can store a set of pattern
associations.
• The association here is a pair of training input target output vector pairs (s(p),
t(p)), with each vector s(p) has n components and each vector t(p) has m
components.
• The determination of weights is done either by using Hebb rule or delta rule.
• The net finds an appropriate output vector, which corresponds to an input vector x,
that may be one of the stored patterns or a new pattern.
• This type of memory network is commonly used in applications such as data
compression and data retrieval.
• It can be noticed that for a hetero associative net, the training input and
target output vectors are different.
• The input layer consists of n number of input units and the output layer
consists of m number of output units.
• There exist weighted interconnections between the input and output
layers. The input and output layer units are not correlated with each other.
Training Algorithm
 TestingAlgorithm

If the responses of the net are binary, then the activation

function to be used is
 Example
❑ A Heteroassociative net trained using the Hebb rule.
❑ Suppose a net is to be trained to store the following mapping from
input row vectors s = ( s1, s2, s3, s4) to output row vectors t = (t1,
t2).
❑ The training is accomplished by the Hebb rule, which is defined
as:
❑ Training:
❑ Outer product:
❑ Second sample:
❑ Third sample:
❑ Fourth sample:
❑ Final W:
❑ Testing a heteroassociative net using the training
input.
❑ Activation function:

❑ The weights are as found in Examples 3.1 and 3.2.

❑ First input: (the same for other inputs)
❑ Using Matrix (First input):
❑ Testing a heteroassociative net with input similar to
the training input.
❑ The test vector x = (0, 1, 0, 0) differs from the
training vector s = (1, 1, 0,0) only in the first
component. We have:

❑ Thus, the net also associates a known output

pattern with this input.
❑ Testing a heteroassociative net with input that is not
similar to the training input.
❑ The test pattern (0 1, 1, 0) differs from each of the training
input patterns in at leas two components.
❑ The output is not one of the outputs with which the
net was trained; in other words, the net does not
recognize the pattern.
❑ In this case, we can view x = (0, 1, 1, 0) as differing
from the training vector s = (1, 1, 0, 0) in the first and
third components, so that the two "mistakes" in the
input pattern make it impossible for the net to
recognize it.
❑ This is not surprising, since the vector could equally
well be viewed as formed from s = (0, 0, 1, 1), with
"mistakes" in the second and fourth components
 Heteroassociative Recurrent Neural Net:
Bidrectional Associative Memory
• Recurrent neural nets can be used as hetero associators.
• Bidirectional Associative Memory (BAM) is an example (Kosko 1988,
1992).
• Hopfield net is an auto associative recurrent neural net.
• BAM network consists of two layers of neurons, connected by
bidirectional weighted paths.
• The net iterates sending signals back and forth between the two layers
until each neuron’s activation remains constant for several steps.
• Bidirectional associative memory nets can respond to input to either layer.
• A BAM net alternates between updating activations for each layer.
• Calls the two layers X-layer and Y-layer instead of input and output layers.
 Bidirectional Associative Memory
• A BAM has n units in the X-layer and m units in the Y-layer.
• The connections between the layers are bidirectional, i.e., if the
weight matrix for signals sent from X-layer to Y-layer is W, the
weight matrix for signals sent from Y-layer to X-layer is WT.
• Three types of BAMS: binary, bipolar and continuous. Binary
and bipolar are similar (Discrete), although bipolar works better
for many problems
Bidirectional Associative Memory
• Like most nets, we have a choice of activation function, use of 0 or
other thresholds.
• BAMs use step function, with the possibility of non-0
threshold.
• Hebb Rule or Delta Rule can be used to train a BAM.
• For a BAM, the weights are computed using a pre-defined
formula and do not change.
• Compute weights for each pair of input si(p) and tj(p), and add all
the individual weight matrices up. For example,
Architecture
 Algorithm
❑ The two bivalent (binary or bipolar) forms of BAM
are closely related.
❑ In each, the weights are found from the sum of the outer products of the
bipolar form of the training vector pairs.
❑ Also, the activation function is a step function, with the possibility of a
nonzero threshold.
 Algorithm
❑ For binary patterns:

❑ For bipolar patterns:

 Algorithm
❑ For binary input vectors, the activation function for
the X-layer and Y-layer are:

❑ For bipolar input vectors, the activation function for

the X-layer and Y-layer are:
 Algorithm
❑ The algorithm is written for the first signal to be sent
from the X-layer to the Y-layer.
❑ Signals are sent only from one layer to the other at any
step of the process, not simultaneously in both
directions.
Algorithm
Algorithm
 Continuous BAM
❑ A continuous bidirectional associative memory
transforms input smoothly and continuously into
output in the range [0, 1] using the logistic sigmoid
function as the activation function for all units.
❑ For binary input vectors (s(p), t ( p ) ) , p = 1, 2, . . . , P,
the weights are determined by the aforementioned
formula:

❑ The activation function is the logistic sigmoid:

Example
Construct and test a BAM network too associate letters E
and F with simple bipolar input-output vectors. The target
output for E is ( -1, 1) and for F is (1, 1). The display
matrix size is 5 X 3.

The input patterns are

The inputs are
X vectors as input: The weight matrix is obtained by
The total weight matrix is
 Testing the network with test vectors "E" and "F."

For test pattern E, computing net input we get

Applying activations, we get y = [ -1 1], hence correct response is obtained.

• For test pattern F Computing net input, we get

Applying activations over the net input, to calculate output, we get y = [1 1], hence correct
response is obtained.
Y vectors as input:
 Testing the network

For test pattern E, now the input is [-1 1]. Computing net input, we have

which is the correct response.

For test pattern F, now the input is [1 1]. Computing net input, we
have

Applying the activation functions, we get

which is the correct response.

• Thus, a BAM network has been constructed and tested in both the
directions from X to Y and Y to X.