REVIEW ARTICLES

Investigation of sequence processing:

A cognitive and computational neuroscience
perspective
Raju S. Bapi1,2,*, V. S. Chandrasekhar Pammi1, K. P. Miyapuram1 and Ahmed1
1
Department of Computer and Information Sciences and 2Centre for Cognitive Science, University of Hyderabad, Hyderabad 500 046, India

ing a lever in a specified sequence, etc. that require sequence

Serial order processing or sequence processing underlies
many human activities such as speech, language, skill processing. Such experimental investigation strategies
learning, planning, problem-solving, etc. Investigating form the core of cognitive neuroscience. Secondly, a model-
the neural bases of sequence processing enables us to ling approach can be adopted that enables mathematical
understand serial order in cognition and also helps in and computational formalization of the dynamics of brain
building intelligent devices. In this article, we review activation observed through experiments. Modelling and
various cognitive issues related to sequence processing simulation methods form the core of computational neu-
with examples. Experimental results that give evidence roscience. It is expected that computational neuroscience
for the involvement of various brain areas will be de- efforts would eventually pave the way for building intel-
scribed. Finally, a theoretical approach based on statis- ligent devices.
tical models and reinforcement learning paradigm is Before enumerating the issues related to sequence process-
presented. These theoretical ideas are useful for study-
ing, serial and parallel aspects of cognition are clarified by
ing sequence learning in a principled way. This article
also suggests a two-way process diagram integrating taking face perception as an example. Recognizing a face ap-
experimentation (cognitive neuroscience) and theory/ pears to involve parallel processing, whereby various fea-
computational modelling (computational neuroscience). tures of the face are apprehended simultaneously and
This integrated framework is useful not only in the pre- recognition ensues from the parallel operations. However,
sent study of serial order, but also for understanding experimental evidence suggests that face perception involves
many cognitive processes. serial processing. Yarbus2, in a classic experiment, monitored
subjects’ eye movements as they viewed portraits. Subjects
Keywords: Cognitive science, computational modelling, reported apprehending the portrait as a whole, but their
reinforcement learning, serial order, sequence learning. eye movements revealed a different phenomenon. During
the process of perceiving the face, observer’s attention moved
‘ . . . the coordination of leg movements in insects, the song from one point of fixation to another. By analysing the
of birds, the control of trotting and pacing in a gaited distribution of points of fixation, the duration of fixation
horse, the rat running the maze, the architect designing a and the distinctive cyclic pattern of examination, Yarbus con-
house, the carpenter sawing a board present a problem of cluded that the subjects made saccadic eye movements,
sequences of action . . .’. fixating successively at the most informative parts of the
— Karl Lashley1 image. Thus these observations point out that underlying
an apparently parallel process of face perception, there is
SERIAL order in behaviour has been studied for a long time. a serial oculomotor process. This example is presented to
Serial order processing or sequence processing is a key clarify the meaning of serial and parallel processes and
issue in many areas of cognitive science such as auditory also to point out that many cognitive phenomena have both
perception, visual perception (three-dimensional object rec- the aspects.
ognition), speech perception, language, skilled behaviour, In this article, we take serial order in cognition as an
goal-directed planning, and problem-solving. The role of example to illustrate how a cognitive phenomenon can be
serial order in cognition can be investigated using a two- investigated in a multidisciplinary fashion. Experimental
pronged approach. First, an experimental approach can be investigation of various aspects of serial order such as
adopted that reveals brain areas (neural bases) involved and perception, learning, representation, organization, neural
their interactions while human or animal subjects are engaged bases comprises the cognitive neuroscience perspective.
in tasks such as pressing buttons in a particular order, operat- We propose a theoretical modelling approach, wherein
Markov models and reinforcement learning are combined in
order to understand sequence processing from the compu-
tational neuroscience perspective. Finally, we advocate
*For correspondence. (e-mail: bapics@uohyd.ernet.in) an integrated framework that combines experimentation
1690 CURRENT SCIENCE, VOL. 89, NO. 10, 25 NOVEMBER 2005
 REVIEW ARTICLES

and computational modelling to enable holistic investiga- proposed that maturation of the prefrontal cortex might
tion of a cognitive phenomenon. make possible the age-related developmental progression
of human infants on working memory tasks. The working
memory studies described above give clues to the adaptive
Issues related to serial order
origin of some behavioural sequences.
Certain types of adaptive behaviour such as imitation
In the following sections, a detailed discussion of several
also involve serial order. It has also been proposed that
issues related to serial order is presented. The issues discussed
propensity to engage in complex sequential activities
are the origin of sequential behaviour in humans, possible
such as imitation is a precursor to high-level behaviours
representation schemes available, the organization of se-
such as empathy and mental simulation5. This kind of
quences, the order and timing of individual elements of a
imitation-based learning is a useful adaptive behaviour
sequence, learning modes, stages and strategies applica-
and is being increasingly used for training robots. Schaal6
ble for sequence learning and the issues in sequence percep-
demonstrated that imitation learning offers a promising
tion. It is to be noted that no single paradigm covers all
route to gain new insights into mechanisms of perceptual
the issues in a unified fashion. Therefore, we discuss each
motor control that could ultimately lead to the creation of
of these issues with the help of a representative example.
autonomous humanoid robots.
Thus, some behavioural sequences have pre-wired origin,
Origins of sequential behaviour while many more are learned.

There are broadly two classes of origin for sequential behav-

iour – one is evolutionary and the other is through learn- Representation of sequences
ing. A variety of natural sequential patterns or ritualistic
behaviours can be seen in many species, including humans. In the following we present evidence for both distributed
Some examples are the grooming movements in rats, locomo- and local representation of human movements. Geor-
tion, mastication, rhythmic respiratory movements, etc. gopoulos et al.7 have shown that reaching movements of the
Many species apart from humans are also capable of ac- arm are represented not by single neurons, but by the com-
quiring sequential behaviours via instrumental condition- bined activity of a large population of cells in the motor
ing. In the following, we give one example of pre-wired cortex of the brain. These researchers recorded 568 cells
behaviour (grooming) and two examples for learned behav- in monkey cortex, while it performed reaching arm move-
iour (working memory, imitation). ments. They found that each individual cell responded
Berridge and Whishaw3 have studied ritualistic grooming best for arm movements in a given direction, and its re-
movements in rats. They demonstrated a crucial role for sponse gradually tailed-off for arm movements in adja-
the neostriatum in exhibiting species-specific (genetic or cent directions. They argued that the direction of movement
pre-wired) sequencing behaviours in rats. On the other hand, is represented in the motor cortex by a population vector,
examples of learned (adaptive) behaviour include speech, which rotated when the monkey intended to move a lever
language, skills, etc. We now present examples of sequential in a rotated direction, prior to movement. This experiment
nature in adaptive behaviours, such as working memory illustrates the distributed nature of representation for ele-
and imitation learning. mental movements in the motor cortex. Recent studies by
Working memory tasks involve linking events across time Lu and Ashe8 suggest that apart from elemental move-
and are also essential for goal-directed behaviour. An impor- ments, sequential movements also have distributed repre-
tant component in serial order in behaviour is the ability sentation.
to hold information on-line, and being able to sequence On the other hand, there is experimental evidence sup-
behaviours in order to attain the desired goal. Diamond4 porting local representation of sequence of movements.
studied performance of infants and monkeys on delayed Tanji and Shima9 have found a group of cells in the cere-
response tasks that assess the working memory capacity. bral cortex of monkeys whose activity is exclusively related
These tasks require subjects to hold information for a to a sequence of movements performed in a particular order.
length of time in the mind, even in the absence of external Two monkeys were trained to perform three movements
cue. The subject initially watches the experimenter hide a (push, pull or turn a manipulandum) in four different or-
desired object in one of two identical wells. After a brief ders. Before each movement, monkeys waited for a tone
delay, during which the wells are hidden from view, the that served as a movement trigger. During learning phase,
subject is allowed to reach out and retrieve the hidden ob- the correct movement was indicated by a green (push),
ject. Infants of 7.5–9 months fail delayed response under yellow (pull) or red (turn) light. After five learning trials,
the same conditions and in the same way, as do monkeys the monkeys performed the sequence from internal memory
with lesions of the dorsolateral prefrontal cortex. It is in the absence of external visual cues. It was observed
therefore suggested that improved performance on this task that the cells in the supplementary motor area (SMA) were
is an index of maturation of frontal cortex function. Diamond4 active while the animal was waiting to perform a motor
CURRENT SCIENCE, VOL. 89, NO. 10, 25 NOVEMBER 2005 1691
REVIEW ARTICLES

sequence of turn–pull–push. But such activity was not In human behaviour, hierarchical structuring has been
observed while the monkey was waiting to perform a dif- argued to be essential for many acquired skills, such as
ferent sequence of turn–push–pull for which a different language, problem-solving and everyday planning10–13.
set of cells was activated. Further, studies show that representation at the higher level
In summary, while studies by Georgopoulos et al.7, and supports grouping of low-level units to form what are popu-
Lu and Ashe8 indicated a distributed representation for larly known as chunks14–19. Chunking also enables over-
individual arm movements, studies by Tanji and Shima9 coming the limitations imposed by limited-capacity working
suggested a possible local representation for sequences of memory, whose limit is proposed to be 7 ± 2 constituents20.
movements. In summary, strength of the evidence from experimental
and theoretical studies so far points to a hierarchical rep-
resentation.
Organization of sequences

Internal organization of behavioural sequences can be either Encoding of serial order

linear (flat) or nonlinear (hierarchical) (Figure 1). Lashley1
argued that the sequential responses that appear to be organ- The order of encoding of sequence of entities into long-term
ized in linear and flat fashion concealed an underlying hi- memory is another issue to be considered in serial order.
erarchical structure. Hierarchical representations of sequences The items could be stored in the order they are encountered,
have an edge over linear representations. They allow easier i.e. the first items are stored with more emphasis than the
access to common subroutines of sequences, easier to last items or the other way round. If the first items are stored
self-repair in the event of failure, and combine efficient more strongly and recalled more easily, then the process
local action at low hierarchical levels while maintaining the is said to have primacy effect. On the other hand, if the
guidance of an overall structure. A linear (flat) organiza- last items are emphasized more and retrieved with ease,
tion of a sequence will be in the form of one long linear then the process is said to possess recency effect. Figure 2
string of actions, as shown in Figure 1. While the repre- shows a serial learning curve that is typically seen in list
sentation is simple from storage point of view, there can learning tasks. This curve reveals that subjects recall words
be potential problems during retrieval. For instance, if the presented at the beginning and the end of a list, better than
nth element has to be retrieved, all the n – 1 preceding words presented in the middle. Items at the beginning may
elements have to be processed. Further, if there is a break take advantage of the long-term memory laid down by more
in the chain, subsequent elements will become inaccessi- frequent rehearsals. Items at the end are easy to recall because
ble. On the other hand, a hierarchical representation they are active in working/short-term memory21. Any
would have multiple levels of representation. Figure 1 shows study of serial order has to address the primacy and re-
a two-level hierarchical representation. At the lower cency issues.
level, representation of the elements of the sequence is flat.
At the higher level, control nodes (chunk nodes) are con-
nected among themselves in a linear fashion and also Timing-related issues
connect to their respective sequence elements forming a
hierarchy. A break in the link between lower level nodes Apart from issues related to representation, organization
does not render any part of the sequence inaccessible, and order of sequences, the timing of individual units in a
since the control nodes (chunk nodes) would still be able sequence is also an important factor. For example, in poetry,
to facilitate access to the lower level nodes. if the relative timing is not maintained, the rhythm is lost.

a1234567 a1 a2 a3 a4 a5 a6 a7

a1234567 a123 a4567

a1 a2 a3 a4 a5 a6 a7

Figure 1. Flat versus hierarchical representation. Schematic diagram depicts a sequence of seven elements represented in flat
(linear) and hierarchical (nonlinear) arrangement. In hierarchical organization, nodes a123 and a4567 represent chunks of elements.

1692 CURRENT SCIENCE, VOL. 89, NO. 10, 25 NOVEMBER 2005

 REVIEW ARTICLES

do not become aware of the repeating sequence, they ex-

hibit improved response times during the execution of the
recurring sequence compared to the random ones. The
improved response times are attributed to implicit learning.
It has been found that distinct brain areas are involved in
implicit and explicit learning modes24.

Learning stages: It is a common observation that when

a skill, involving sequence of entities, is being acquired,
List position we need to be more attentive in the initial phase; however,
during the later, more automatic phase, attention can be
Figure 2. Serial position curve showing the relationship between the engaged in other tasks. Fitts25 proposed a framework for skill
list position of an item and associated probability of recall. The first
half of the curve where the first few elements have higher recall prob-
acquisition that included two major stages in the develop-
ability depicts the primacy effect. In the second half, the last few items ment of a cognitive skill: a declarative stage in which
have higher probability of recall, pointing to the recency effect in serial facts about the skill domain are interpreted and a proce-
list learning.
dural stage in which the domain knowledge is directly
embodied in procedures for performing the skill. Brain
activation differences were also observed when subjects
On the other hand, it has minimal effect in prose-like text.
learned a sequential skill and progressed from the early,
There are three attributes for timing, namely execution
more deliberate and attentive stage to the late and more
time of each individual unit in a sequence, time delay be-
automatic stage26–28.
tween two consecutive units, and overall time taken for
sequence execution. The first two attributes can be either
Learning strategies: We can distinguish learning strate-
fixed throughout the entire sequence or variable. The varia-
gies broadly into two main categories: supervised and unsu-
tions could possibly depend on the context of occurrence
pervised, based on whether evaluative feedback was provided
of each individual unit in the sequence. Thus timing and
or not. Feedback signal provides an assessment of the
temporal modulation issues are important for deeper investi-
performance of the system during the learning process. In
gation of sequence processing. In a recent review, Janata
supervised learning, we assume that the teacher provides
and Grafton22 suggested three important subcomponents
the desired response at each instant of time that can be used
related to understanding neural basis of serial order in music.
to calculate the errors and make appropriate corrections in
They are timing, attention and sequence learning. Janata
order to eventually achieve the desired target. In a variation of
and Grafton22 argued that studying aspects of music would
supervised learning called reinforcement learning, a
lead to better understanding of complex human behaviours.
coarse feedback indicating the quality of the output is
provided without specifying the desired response itself. In
Learning of sequences unsupervised learning, the desired response is not known.
Thus explicit error information cannot be used to improve
In this section, we will discuss different aspects related to behaviour in unsupervised learning. The system needs to
learning, such as implicit and explicit learning modes, discover the inherent regularities present in the inputs and
stages in learning, and learning strategies. self-organize the information. It has been proposed that a
different set of brain areas is associated with different
Learning modes: Learning of serial order may be operat- learning strategies29. Thus adaptive sequential behaviours
ing in one of the two modes – explicit learning mode or could be acquired using any one of the strategies, i.e. super-
implicit learning mode. Explicit learning includes con- vised, reinforcement, or unsupervised learning.
scious attempts to construct a representation of the task;
directed search of memory for similar or analogous task-
relevant information, and conscious attempts to derive and Perception issues in sequence processing
test hypotheses related to the structure of the task23. This
type of learning has been distinguished from alternative A potential theory of serial order needs to deal with various
modes of learning, termed implicit learning, in which task- sequence perception problems organized broadly into two
relevant information is acquired automatically and without categories: sequence recognition and sequence recall/
conscious awareness of what is being learnt. An example generation. These problems can be stated formally as shown
of implicit learning task is SRT (serial reaction time) task, below30,31.
where the subjects execute sequential finger movements in
response to visual cues that appear one after another. Un- (i) Recognition: Given a sequence Si, Si+1, . . . , Sk, the
known to the subjects, a repeating sequence is embedded recognition problem involves determining if the given se-
among the random sequence of cues. Although subjects quence is legitimate or not.
CURRENT SCIENCE, VOL. 89, NO. 10, 25 NOVEMBER 2005 1693
REVIEW ARTICLES

Si, Si+1 . . . , Sk → Yes or No, where 1 ≤ i ≤ k ≤ ∞. ning from left to right, serves as a standard landmark
called the central sulcus. The cortex anterior to the central
(ii) Recall/generation: Given a sequence Si, Si+1 . . . , Sk, sulcus is called the anterior (frontal) lobe and that which is
recall or generation problem involves generating or re- posterior is called the posterior lobe. Anterior lobe com-
calling the next item Sk+1. prises the prefrontal cortex, supplementary motor area,
pre-motor area and the primary motor area. Posterior lobe
Si, Si+1 . . . , Sk → Sk+1, where 1 ≤ i ≤ k ≤ ∞. comprises the sensory cortex, parietal cortex, occipital
(visual) cortex and the temporal areas. Some of the prominent
In both recognition and recall, an internal representation nuclei that lie below the cerebrum, i.e. in the sub-cortical
of sequence may be needed for making perceptual judgments region are the basal ganglia, the hippocampus and the amyg-
and for prediction (or generation) of subsequent elements, dala.
respectively. Future experimental investigations would need The cerebellum is located below the cerebral cortex.
to tease out differences in the cognitive processes involved The spinal cord connects the brain at the base near the
and the concomitant differences in brain areas sub-serving medulla oblongata. In the remainder of the section, we
recognition and recall/generation of sequences. describe results from our own and other experimental efforts
investigating the brain areas associated with various aspects
of serial order.
Brain areas sub-serving aspects of serial order

The current working assumption in neuroscience is that Brain areas involved in serial order
various functions are organized in a modular fashion in
the brain. In this section, we shall first present a brief over-
In this subsection we summarize some representative experi-
view of various brain areas and their function.
mental findings corresponding to the issues related to se-
The central nervous system is composed of the brain
rial order elucidated earlier.
and the spinal cord. Stimuli from sensory/peripheral organs
are carried via the spinal cord and processed in the brain.
Origins: The area specifically proposed for procedural
The corresponding motor signals (outputs) get transmitted
memory (such as skills and habits) is the striatum32. Berridge
to the appropriate peripheral system via the spinal cord. The
and Whishaw3 showed that lesions of the precentral corti-
brain is composed of three major components, namely the
cal areas or of other neo-cortical areas did not affect the
cerebrum, the cerebellum, and the sub-cortical areas. The
performance of ritualistic behavioural sequences in rats.
brain is divided into two hemispheres called the left hemi-
On the other hand, lesions of the neostriatum, which receives
sphere (left lobe; Figure 3) and the right hemisphere (right
inputs from the cerebral cortex, impaired the performance
lobe). These hemispheres have rich interconnections
thereby implicating a role for the striatum in pre-wired
through the corpus callosum. A prominent groove on the
behavioural sequences. Curran24 and Clegg et al.33 also
surface of the cerebrum when viewed from the top and run-
summarized evidence for striatal involvement in sequence
learning based on experiments involving implicit learning
tasks such as the SRT.

Representation: Studies by Georgopoulos et al.7 and Lu

and Ashe8 suggested a distributed representation for individ-
ual arm movements in the motor cortex of rhesus mon-
keys. On the other hand, Tanji and Shima9 demonstrated
experimental evidence supporting local representation of
sequence of movements in the SMA of monkeys.

Organization and timing: Apart from the above findings,

there are other studies that investigated neural bases re-
sponsible for the timing and chunking aspects in serial
order. Timing aspects in sequence learning have been ex-
plored and the cerebellum is found to be critically in-
volved34,35. Disruption of chunk representation of movement
sequences in pre-SMA was demonstrated using transcranial
magnetic stimulation study by Kennereley et al.36. The
review of Janata and Grafton22 suggested that the cerebel-
Figure 3. Generic brain areas. The four lobes of the brain (Image
from Purves et al., Life: The Science of Biology, 4th edn; used with lum, SMA, premotor cortex, basal ganglia and parietal
permission from Sinauer Associates Inc., USA, 1999). cortices are involved in timing aspects of both perceptual
1694 CURRENT SCIENCE, VOL. 89, NO. 10, 25 NOVEMBER 2005
 REVIEW ARTICLES

and motor tasks. They pointed out that these areas might Theoretical framework and computational modelling
be playing an important role in perception–action cycle.
So far, we had highlighted the cognitive neuroscience
perspective of serial order in cognition. In this section a
Learning: We pointed out earlier, the various aspects of
discussion is presented on how computational neuroscience
sequence learning, namely learning modes, learning stages
efforts can be directed towards theoretical investigation of
and learning strategies. Curran24 described distinct brain
serial order. In the literature, several researchers described
areas involved in the implicit and explicit modes of se-
computational models for serial learning. These models
quence learning. He suggested involvement of striatum in
can be broadly classified into three categories – biologically
the implicit learning mode and the medial, temporal and
inspired models, connectionist models, and hybrid models.
diencephalic brain regions during the explicit learning
In biologically inspired computational models, some aspects
mode. Squire and Zola32 also reported similar findings
of anatomical organization and function are mimicked44,45.
during implicit and explicit memory-related tasks.
In connectionist models, the aim is to mimic the overall
Studies by Jupetner et al.26,27 have shown that as learning
behaviour of the biological system rather than replicating
progressed from controlled (new learning) stage to automatic
the internal organizational details46. In hybrid models,
stage in a finger movement learning task, brain activation
engineering principles enable the construction of models
shifted from the anterior to the posterior parts both in the
that illuminate biological function. These models usually do
neocortex and the sub-cortical structures. Sakai et al.28
not attempt an explicit replication of anatomical organiza-
demonstrated transition of brain activity from the frontal
tion47,48.
regions (pre-SMA and dorsolateral prefrontal cortex) to the
In order to illustrate the typical activity taken up under
parietal regions (precuneus and lateral parietal cortex) as
the enterprise of computational modelling, we present here
learning advanced from early to non-early (intermediate
a theoretical framework for modelling sequence processing
and late) stages respectively. Our behavioural37 and functional
focusing on the organizational aspects. The framework
magnetic resonance imaging experiments investigated
we propose here falls under the third category, i.e. hybrid
different aspects of procedural memory such as represen-
computational model. We propose a two-level model where
tation38, complexity39 and learning mode40. We proposed
Markov models and reinforcement learning are combined to
possible cortical localization of various modules and
specifically address how biological systems learn to organize
mappings that subjects use while practising a set of finger
sequential information in a hierarchical fashion.
movements in response to visual stimuli. In early stages
Curran24 summarized results from the implicit learning
of learning this task, subjects may follow a long route in
studies and concluded that in sequence learning tasks, basal
which the response is mediated by a visuo-spatial mapping
ganglia systems may enable extraction of first-order sequen-
followed by a spatial-motor mapping. In the late stages of
tial dependencies and that cortical–basal ganglia loops
learning this task, subjects may follow a shorter route,
may be responsible for learning higher order structures.
where they utilize a direct visuo-motor mapping. Further,
Based on this empirical evidence we propose a two-level
we hypothesized that there are two sequence representations,
model where at the lower level, first-order sequential depend-
effector-independent in visual/spatial coordinates and ef-
encies are extracted and at the higher level, hierarchical
fector-dependent representation in motor coordinates.
structure corresponding to the entire sequence is captured
Possible neural bases for the effector-independent se-
using reinforcement learning.
quence representation may be in the parietal–prefrontal
network and effector-dependent representation may be in
the SMA–primary motor cortical network. Basal ganglia First-order Markov model
structures may also be differentially involved in supporting
different representations and the premotor cortex may Markov model is a well-formulated mathematical framework
mediate various mappings37,38,41. for capturing first and higher order sequential dependen-
Doya29,42 suggested that the cerebellum, basal ganglia, and cies among random variables describing the behaviour of
cerebral cortex are specialized for different strategies of a system. The main assumption (also called Markov
learning, namely supervised learning, reinforcement assumption) is that prediction of the next state depends
learning and unsupervised learning respectively. only on a portion of the previous history of state transi-
tions. In the case of first-order Markov models, the prob-
ability that qt (state at time t) is equal to i is completely
Perception: In a recent investigation, Pasupathy and
predictable by knowing qt–1 and ignoring the rest of the
Miller43 demonstrated that the learning-related activity in
previous state history (a second-order Markov model
prefrontal cortex and striatum showed different time courses
would require qt–1 and qt–2 to predict qt). A formal defini-
during associative learning. Their results on monkeys sug-
tion of the first-order Markov model is given below.
gested that the striatum generates quick predictions about
the behavioural choice and the prefrontal cortex reveals
the slower accumulation of the correct answer. P(qi = i|qt–1 = j, qt–2 = k, . . .) = P(qt = i|qt–1 = j).

CURRENT SCIENCE, VOL. 89, NO. 10, 25 NOVEMBER 2005 1695

REVIEW ARTICLES

Reinforcement learning and RL is expected to reveal whether models based on this

framework can really solve the hierarchical sequence decision
Reinforcement learning (RL) has been proposed as a bio- problems. Such models would mimic the hierarchical nature
logically realistic framework for learning sequential deci- of organization of sequences as observed in biological
sions in animals and humans49. In this paradigm, the systems, without an explicit replication of the anatomical
sequential decision problem involves assuming a policy (a organization within.
mapping from the states to possible actions) and learning As already pointed out earlier, a unified framework for
a value function over the state space, so that the sequence addressing various aspects of serial order has not been at-
of actions maximizes the expected future reward. The most tempted either for designing cognitive neuroscience experi-
popular method for learning the value function is the ments or for constructing computational models. This still
method of temporal difference (TD). A formal definition is remains an open problem.
given below.
Conclusion and future work
V(t) = E[r(t + 1) + r(t + 2) + L],
δ(t) = r(t) + V(t) – V(t – 1), Sequencing is an essential aspect of animal and human
behaviour. An introduction to serial order, an essential
where V(t) represents the value of a state, r(t) is the reward aspect of human behaviour, is given. A brief summary of
and δ(t) is the temporal difference signal all at time t. E[.] various aspects of serial order is presented here. The issues
represents the expectation or averaging operator. Value of discussed are the origin, representation schemes, organization,
a state, V(t), is set to be the average future reward that is likely order, timing, learning and perception. Brain areas related
to be obtained in the current state. Temporal difference signal, to sequencing are summarized and our own empirical efforts
δ(t), tracks the difference between the expected reward and in this direction are also described. A theoretical framework
actual reward and serves as the reinforcement learning combining the mathematical ideas of Markov models and
(internal feedback) signal. RL is proposed as an example for computational modelling
First-order Markov models alone would capture a flat of hierarchical sequences. There is an urgent need to formu-
organization of the sequence and do not incorporate late unified framework for investigation of serial order
learning. Although RL models incorporate learning, the from cognitive neuroscience and computational neurosci-
policies that are learned would still have a flat organization. ence perspectives.
We propose that by combining these two models and util- There is also an immediate need to facilitate a bridge
izing the TD error signal, hierarchical policies could be between the cognitive and computational neuroscience
learned. Computer modelling (simulation) of the two- streams. Figure 5 depicts an integration of the efforts in-
level architecture (Figure 4) combining Markov models volved in understanding serial order or sequencing. There
are two main efforts in this direction – one is experimen-
tation (cognitive neuroscience) and the other is building
ACTOR theoretical framework and undertaking computational
modelling work (computational neuroscience). The former
Hierarchical
Policy

Markov Dependencies
Understanding
serial order/sequencing
ACTION

CRITIC Example: Music, planning, language,

STATE

problem solving
Value
Function TD Error
Behavioural and Functional
REWARD neural correlates understanding

Theoretical framework/
ENVIRONMENT Behavioural and computational
imaging experiments modelling

Figure 4. Block diagram of the proposed hybrid model. Actor–critic

based model incorporating Markov model and hierarchical policy as Figure 5. Block diagram depicting scientific endeavour. Relationship
sub-modules in the actor module. The two levels in the actor module, between experimentation (cognitive neuroscience) and modelling (com-
namely the Markov model and the hierarchical policy module would putational neuroscience) is emphasized in promoting holistic under-
enable learning hierarchical sequence decision problems. standing of serial order.

1696 CURRENT SCIENCE, VOL. 89, NO. 10, 25 NOVEMBER 2005

 REVIEW ARTICLES

effort enables understanding of the behavioural and neural 19. Pammi, V. S. C., Miyapuram, K. P., Bapi, R. S. and Doya, K.,
correlates of sequencing. The latter effort engenders func- Chunking phenomenon in complex sequential skill learning in
humans. In LNCS, Proceedings of International Conference on
tional understanding. It is difficult, if not impossible, to Neural Information Processing (eds Pal, N. R. et al.), Springer-
gain the complete functional understanding of a cognitive Verlag, Heidelberg, 2004, vol. 3316, pp. 294–299.
phenomenon by a pure empirical approach. Similarly, a 20. Miller, G. A., The magical number seven, plus or minus two:
pure theoretical or modelling approach will run the risk Some limits on our capacity for processing information. Psychol.
of lacking the biological realism and relevance. Hence, the Rev., 1956, 63, 81–97.
21. Grossberg, S., Some networks that can learn, remember, and repro-
empirical and modelling efforts are of utmost importance duce any number of complicated space time patterns I. J. Math.
and they reinforce each other. The other point emphasized Mech., 1969, 19, 53–91.
here is that these efforts must go hand-in-hand. The mod- 22. Janata, P. and Grafton, S. T., Swinging in the brain: Shared neural
elling exercise may also spawn further predictions to be substrates for behaviors related to sequencing and music. Nature
experimentally verified in future. Although the current ar- Neurosci., 2003, 6, 682–687.
23. Anderson, J. R., Acquisition of cognitive skill. Psychol. Rev., 1982,
ticle has focused on sequence learning, the proposed inte- 89, 369–406.
gration would serve as an indispensable model for organizing 24. Curran, T., On the neural mechanisms of sequence learning.
scientific endeavours toward understanding any cognitive PSYCHE, 1995, 2.
phenomenon. 25. Fitts, P. M., Perceptual motor skill learning. In Categories of Human
Learning (ed. Melton, A. W.), Academic Press, New York, 1964,
pp. 243–285.
1. Lashley, K. S., The problem of serial order in behavior. In Cerebral 26. Jueptner, M., Stephan, K. M., Frith, C. D., Brooks, D. J., Fracko-
Mechanisms in Behavior (ed. Jeffress, L. A.), Wiley, New York, wiak, R. S. J. and Passingham, R. E., Anatomy of motor learning.
1951, pp. 112–136. I. Frontal cortex and attention to action. J. Neurophysiol., 1997,
2. Yarbus, A. L., Eye movements during perception of complex ob- 77, 1313–1324.
jects. In Eye Movements and Vision (ed. Riggs, L. A.), Plenum 27. Jueptner, M., Frith, C. D., Brooks, D. J., Frackowiak, R. S. J. and
Press, New York, 1967, pp. 171–196. Passingham, R. E., Anatomy of motor learning. II. Subcortical
3. Berridge, K. C. and Whishaw, I. Q., Cortex, striatum, and cerebel- structures and learning by trial and error. J. Neurophysiol., 1997,
lum: control of syntactic grooming sequences. Exp. Brain Res., 77, 1325–1337.
1992, 90, 275–290. 28. Sakai, K., Hikosaka, O., Miyauchi, S., Takino, R., Sasaki, Y. and
4. Diamond, A., Evidence for the importance of dopamine for prefrontal Pütz, B., Transition of brain activation from frontal to parietal areas
cortex functions early in life. Philos. Trans. R. Soc. London, Ser. in visuomotor sequence learning. J. Neurosci., 1998, 18, 1827–
B, 1996, 351, 1483–1494. 1840.
5. Arbib, M. and Rizzolatti, G., Neural expectations: a possible evolu- 29. Doya, K., What are the computations in the cerebellum, the basal
tionary path from manual skills to language. Commun. Cognit., ganglia, and the cerebral cortex. Neural Networks, 1999, 12, 961–
1996, 29, 393–424. 974.
6. Schaal, S., Is imitation learning the route to humanoid robots? 30. Sun, R., Introduction to sequence learning. In Sequence Learning:
Trends Cognit. Sci., 1999, 3, 233–242. Paradigms, Applications and Algorithms (eds Sun, R. and Giles,
7. Georgopoulos, A. P., Lurito, J. T., Petrides, M. and Schwartz, A. C. L.), Springer-Verlag, LNAI, 2000, vol. 1828, pp. 1–10.
B., Mental rotation of the neuronal population vector. Science, 31. Sun, R. and Giles, L., Sequence learning: from prediction and rec-
1989, 243, 234–236. ognition to sequential decision making. IEEE Intelligent Syst.,
8. Lu, X. and Ashe, J., Anticipatory activity in primary motor cortex 2001, 16, 67–70.
codes memorized movement sequences. Neuron, 2005, 45, 967–973. 32. Squire, L. R. and Zola, S. M., Structure and function of declara-
9. Tanji, J. and Shima, K., Role for supplementary motor area cells tive and nondeclarative memory systems. Proc. Natl. Acad. Sci.
in planning several movements ahead. Nature, 1994, 371, 413–416. USA, 1996, 93, 13515–13522.
10. Chomsky, N., In Syntactic Structures, Mouton & Co, The Hague, 33. Clegg, B. A., DiGirolamo, G. J. and Keele, S. W., Sequence learn-
1957. ing. Trends Cognit. Sci., 1998, 2, 275–281.
11. Newell, A., Shaw, J. C. and Simon, H. A., Elements of a theory of 34. Shin, J. C. and Ivry, R. B., Spatial and temporal sequence learning
human problem-solving. Psychol. Rev., 1958, 65, 151–166. in patients with Parkinson’s disease or cerebellar lesions. J. Cognit.
12. Miller, G. A., Galanter, E. and Pribram, K. H., In Plans and the Neurosci., 2003, 15, 1232–1243.
Structure of Behaviour, Holt, Rinehart & Winston, New York, 1960. 35. Sakai, K., Hikosaka, O., Takino, R., Miyauchi, S., Nielsen, M. and
13. Newell, A. and Simon, H., In Human Problem Solving, Prentice Tamada, T., What and when: Parallel and convergent processing
Hall, NJ, 1972. in motor control. J. Neurosci., 2000, 20, 2691–2700.
14. Wickelgren, W. A., Context sensitive coding, associative memory, 36. Kennereley, S. W., Sakai, K. and Rushworth, M. F. S., Organiza-
and serial order in (speech) behavior. Psychol. Rev., 1969, 76, 1– tion of action sequences and role of the Pre-SMA. J. Neuro-
15. physiol., 2004, 91, 978–993.
15. Wickelgren, W. A., Webs, cell assemblies, and chunking in neural 37. Bapi, R. S., Doya, K. and Harner, A. M., Evidence for effector in-
nets: Introduction. Can. J. Exp. Psychol., 1999, 53, 118–131. dependent and dependent representations and their differential
16. MacKay, D. G., The problem of flexibility, fluency, and speed- time course of acquisition during motor sequence learning. Exp.
accuracy trade-off in skilled behavior. Psychol. Rev., 1982, 89, Brain Res., 2000, 132, 149–162.
483–506. 38. Bapi, R. S., Graydon, F. X. and Doya, K., Time course of learning
17. Rosenbaum, D. A., Kenny, S. B. and Derr, M. A., Hierarchical of visual and motor sequence representations. Society for Neuro-
control of rapid movement sequences. J. Exp. Psychol.: Hum. Per- science Annual Meeting, USA, 2000.
cept. Perform., 1983, 9, 86–102. 39. Pammi, V. S. C., Miyapuram, K. P., Bapi, R. S., Samejima, K.
18. Sakai, K., Kitaguchi, K. and Hikosaka, O., Chunking during human and Doya, K., Acquisition of complex sequential skills: Behav-
visuomotor sequence learning. Exp. Brain Res., 2003, 152, 229– ioral and fMRI Investigation. Building the Brain, Manesar, India,
242. 2003.

CURRENT SCIENCE, VOL. 89, NO. 10, 25 NOVEMBER 2005 1697

REVIEW ARTICLES
40. Pammi, V. S. C., Miyapuram, K. P., Bapi, R. S., Samejima, K. and Networks: Steps Toward Principled Integration (eds Honavar, V.
Doya, K., The activation of orbitofrontal cortex reflects trial and and Uhr, L.), Academic Press, San Diego, 1994, pp. 241–268.
error processes in a visuomotor sequence learning task. Networks 47. Suri, R. E. and Schultz, W., Dopamine-like reinforcement signal
and Behavior, Bangalore, 2003. improves learning of sequential movements by neural network.
41. Grafton, S. T., Hazeltine, E. and Ivry, R. B., Abstract and effector- Exp. Brain Res., 1998, 121, 350–354.
specific representations of motor sequences identified with PET. J. 48. Bapi, R. S. and Doya, K., Multiple forward model architecture for
Neurosci., 1998, 18, 9420–9428. sequence processing. In Sequence Learning: Paradigms, Algo-
42. Doya, K., Complementary roles of basal ganglia and cerebellum in rithms, and Applications (eds Sun, R. and Giles, L.), Springer Ver-
learning and motor control. Curr. Opin. Neurobiol., 2000, 10, lag, Germany, 2001, pp. 309–320.
732–739. 49. Sutton, R. S. and Barto, A. G., Reinforcement Learning: An Introduc-
43. Pasupathy, A. and Miller, E. K., Different time courses of learning- tion, MIT Press, Cambridge, MA, 1998.
related activity in the prefrontal cortex and striatum, Nature, 2005,
433, 873–876.
44. Berns, G. and Sejnowski, T. J., A computational model of how the ACKNOWLEDGEMENTS. We thank the anonymous reviewers for
basal ganglia produce sequences. J. Cognit. Neurosci., 1998, 10, their suggestions for refining our manuscript. We also thank Drs Kiran-
108–121. mayi S. Bapi and S. Durga Bhavani for their helpful discussions.
45. Dominey, P. F., Arbib, M. A. and Joseph, J. P., A model of cortico- Pammi Chandrasekhar and Ahmed acknowledge CSIR, New Delhi for
striatal plasticity for learning oculomotor associations and se- senior research fellowship.
quences. J. Cognit. Neurosci., 1995, 7, 311–336.
46. Servan-Schreiber, D., Cleeremans, A. and McClelland, J. L., Graded
state machines: The representation of temporal contingencies in
simple recurrent networks. In Artificial Intelligence and Neural Received 15 December 2004; revised accepted 16 June 2005

1698 CURRENT SCIENCE, VOL. 89, NO. 10, 25 NOVEMBER 2005