Mental Imagery

MENTAL IMAGERY
Topic Editors
Joel Pearson and Stephen M. Kosslyn
PSYCHOLOGY HUMAN NEUROSCIENCE

FRONTIERS COPYRIGHT ABOUT FRONTIERS
STATEMENT
© Copyright 2007-2013 Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering
Frontiers Media SA.
All rights reserved.
approach to the world of academia, radically improving the way scholarly research is managed.
All content included on this site,
The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share
such as text, graphics, logos, button and generate knowledge. Frontiers provides immediate and permanent online open access to all
icons, images, video/audio clips,
downloads, data compilations and
its publications, but this alone is not enough to realize our grand goals.
software, is the property of or is
licensed to Frontiers Media SA
(“Frontiers”) or its licensees and/or
subcontractors. The copyright in the FRONTIERS JOURNAL SERIES
text of individual articles is the
property of their respective authors, The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online
subject to a license granted to
Frontiers.
journals, promising a paradigm shift from the current review, selection and dissemination
The compilation of articles processes in academic publishing.
constituting this e-book, as well as All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service
all content on this site is the
exclusive property of Frontiers. to the scholarly community. At the same time, the Frontiers Journal Series operates on a revo-
Images and graphics not forming lutionary invention, the tiered publishing system, initially addressing specific communities of
part of user-contributed materials
may not be downloaded or copied scholars, and gradually climbing up to broader public understanding, thus serving the interests
without permission. of the lay society, too.
Articles and other user-contributed
materials may be downloaded and
reproduced subject to any copyright
or other notices. No financial DEDICATION TO QUALITY
payment or reward may be given for
any such reproduction except to the Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interac-
author(s) of the article concerned.
tions between authors and review editors, who include some of the world’s best academicians.
As author or other contributor you
grant permission to others to Research must be certified by peers before entering a stream of knowledge that may eventually
reproduce your articles, including reach the public - and shape society; therefore, Frontiers only applies the most rigorous and
any graphics and third-party
materials supplied by you, in unbiased reviews.
accordance with the Conditions for Frontiers revolutionizes research publishing by freely delivering the most outstanding research,
Website Use and subject to any
copyright notices which you include evaluated with no bias from both the academic and social point of view.
in connection with your articles and By applying the most advanced information technologies, Frontiers is catapulting scholarly
materials.
publishing into a new generation.
All copyright, and all rights therein,
are protected by national and
international copyright laws.
The above represents a summary
only. For the full conditions see the
WHAT ARE FRONTIERS RESEARCH TOPICS?
Conditions for Authors and the Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are
Conditions for Website Use.
collections of at least ten articles, all centered on a particular subject. With their unique mix
Cover image provided by Ibbl sarl,
Lausanne CH of varied contributions from Original Research to Review Articles, Frontiers Research Topics
unify the most influential researchers, the latest key findings and historical advances in a hot
ISSN 1664-8714 research area!
ISBN 978-2-88919-149-9 Find out more on how to host your own Frontiers Research Topic or contribute to one as an
DOI 10.3389/978-2-88919-149-9 author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org
July 2013 | Mental Imagery | 1

MENTAL IMAGERY
Topic Editors:
Joel Pearson, The University of New South Wales, Australia
Stephen M. Kosslyn, Stanford University, USA
Our ability to be conscious of the world

around us is often discussed as one of the most
amazing yet enigmatic processes under scientific
investigation today. However, our ability to
imagine the world around us in the absence of
stimulation from that world is perhaps even
more amazing. This capacity to experience
objects or scenarios through imagination, that
do not necessarily exist in the world, is perhaps
one of the fundamental abilities that allows us
successfully to think about, plan, run a dress rehearsal of future events, re-analyze past
events and even simulate or fantasize abstract events that may never happen.
Empirical research into mental imagery has seen a recent surge, due partly to the
development of new neuroscientifc methods and their clever application, but also due
to the increasing discovery and application of more objective methods to investigate this
inherently internal and private process.
As this topic is being cross-hosted in both Frontiers in Human Neuroscience and Frontiers
in Perception Science, we invite researchers from different fields to submit opinionated but
balanced reviews, new empirical, theoretical, philosophical or technical papers covering any
aspect of mental imagery. In particular, we encourage submissions focusing on different
sensory modalities, such as olfaction, audition somatosensory etc. Similarly, we support
submissions focusing on the relationship between mental imagery and other neural and
cognitive functions or disorders such as visual working memory, visual search or disorders
of anxiety.
Together, we hope that collecting a group of papers on this research topic will help to unify
theory while providing an overview of the state of the field, where it is heading, and how
mental imagery relates to other cognitive and sensory functions.

Table of Contents
05 Mental Imagery
Joel Pearson and Stephen M. Kosslyn
06 Hemispheric Differences with in the Fronto-Parietal Network Dynamics
Underlying Spatial Imagery
Alexander T. Sack and Teresa Schuhmann
16 Unmasking the Perky Effect: Spatial Extent of Image Interference on Visual
Acuity
Adam Reeves and Catherine Craver-Lemley
23 Training Visual Imagery: Improvements of Metacognition, But not Imagery
Strength
Rosanne L. Rademaker and Joel Pearson
34 New Percepts via Mental Imagery?
Fred W. Mast, Elisa M. Tartaglia and Michael H. Herzog
39 An Emerging Paradigm: A Strength-Based Approach to Exploring Mental
Imagery
Tadhg E. MacIntyre, Aidan P. Moran, Christian Collet and Aymeric Guillot
51 Vividness of Visual Imagery and Incidental Recall of Verbal Cues, When
Phenomenological Availability Reflects Long-Term Memory Accessibility
Amedeo D’Angiulli, Matthew Runge, Andrew Faulkner, Jila Zakizadeh, Aldrich Chan
and Selvana Morcos
69 The Effects of Visual Imagery on Face Identification: An Erp Study
Jianhui Wu, Hongxia Duan, Xing Tian, Peipei Wang and Kan Zhang
77 Electrophysiological Potentials Reveal Cortical Mechanisms for Mental Imagery,
Mental Simulation, and Grounded (Embodied) Cognition
Haline E. Schendan and Giorgio Ganis
99 A Cross-Modal Perspective on the Relationships Between Imagery and Working
Memory
Lora T. Likova
113 Verbal to Visual Code Switching Improves Working Memory in Older Adults: An
fMRI Study
Mariko Osaka, Yuki Otsuka and Naoyuki Osaka
121 Mental Imagery for Musical Changes in Loudness
Freya Bailes, Laura Bishop, Catherine J. Stevens and Roger T. Dean
130 Imagining is Not Doing but Involves Specific Motor Commands: A Review of
Experimental Data Related to Motor Inhibition
Aymeric Guillot, Franck Di Rienzo, Tadhg MacIntyre, Aidan Moran and
Christian Collet

152 Mental Imagery of Speech: Linking Motor and Perceptual Systems Through
Internal Simulation and Estimation
Xing Tian and David Poeppel
163 Effect of Biomechanical Constraints in the Hand Laterality Judgment Task:
Where Does It Come from?
Gilles Vannuscorps, Agnesa Pillon and Michael Andres
172 Understanding Immersivity: Image Generation and Transformation Processes in
3D Immersive Environments
Maria Kozhevnikov and Rupali P. Dhond
182 Abacus in the Brain: A Longitudinal Functional MRI Study of a Skilled Abacus
User with a Right Hemispheric Lesion
Satoshi Tanaka, Keiko Seki, Takashi Hanakawa, Madoka Harada, Sho K. Sugawara,
Norihiro Sadato, Katsumi Watanabe and Manabu Honda

EDITORIAL
published: 23 April 2013
doi: 10.3389/fpsyg.2013.00198
Mental imagery
Joel Pearson 1* and Stephen M. Kosslyn 2
1
School of Psychology, The University of New South Wales, Sydney, NSW, Australia
2
Minerva University, San Francisco, CA, USA
*Correspondence: jpearson@unsw.edu.au
Edited by:
Philippe G. Schyns, University of Glasgow, UK
Reviewed by:
Philippe G. Schyns, University of Glasgow, UK
Our ability to be conscious of the world around us is often dis- it could be argued that this ability is one of the main factors
cussed as one of the most amazing yet enigmatic processes under that have allowed us as a species to dominate our planet so
scientific investigation today. However, our ability to imagine the profoundly.
world around us in the absence of stimulation from that world is Empirical research into mental imagery has seen a recent
perhaps even more amazing. surge, which is partly a result of new neuroscientific methods
Our capacity to re-experience objects or scenarios that we’ve and their clever application—but is also due to the discovery and
encountered before, and to notice new things about those expe- application of additional sorts of objective methods to investigate
riences, is itself remarkable. But perhaps more remarkable still this inherently internal and private process.
is our ability to experience objects or events that do not exist Here we introduce an inspiringly broad range of work that
in the world, through our imagination. This is perhaps one of focuses on mental imagery. This ebook contains the work from
the fundamental abilities that allow us successfully to plan, run a broad range of researchers in different fields, both empirical
dress rehearsals of future events, re-analyze the past—and even work and reviews. Chapters range from the role of imagery in
simulate or fantasize events that may never happen. In short, music, biomechanics, and mathematics to the functions of the
cerebral hemispheres in imagery and imagery’s effects on sensory
perception.
This collection provides a cohesive and broad-spectrum
addition to the rapidly growing field of mental imagery. This set
of articles provides theoretical insights and an overview of the
state of empirical understanding, where it is heading, and how
mental imagery relates to other cognitive and sensory functions.
Received: 24 March 2013; accepted: 02 April 2013; published online: 23 April 2013.
Citation: Pearson J and Kosslyn SM (2013) Mental imagery. Front. Psychol. 4:198. doi:
10.3389/fpsyg.2013.00198
This article was submitted to Frontiers in Perception Science, a specialty of Frontiers
in Psychology.
Copyright © 2013 Pearson and Kosslyn. This is an open-access article distributed
under the terms of the Creative Commons Attribution License, which permits use,
distribution and reproduction in other forums, provided the original authors and
source are credited and subject to any copyright notices concerning any third-party
graphics etc.
www.frontiersin.org April 2013 | Volume 4 | Article 198 | 5

REVIEW ARTICLE
published: 28 June 2012
doi: 10.3389/fpsyg.2012.00214
Hemispheric differences within the fronto-parietal network

dynamics underlying spatial imagery
Alexander T. Sack * and Teresa Schuhmann
Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, Netherlands
Edited by: Spatial imagery refers to the inspection and evaluation of spatial features (e.g., distance,
Joel Pearson, The University of New
relative position, configuration) and/or the spatial manipulation (e.g., rotation, shifting, reori-
South Wales, Australia
enting) of mentally generated visual images. In the past few decades, psychophysical as
Reviewed by:
Raymond Van Ee, University Utrecht, well as functional brain imaging studies have indicated that any such processing of spa-
Netherlands tially coded information and/or manipulation based on mental images (i) is subject to similar
Angelika Lingnau, University of behavioral demands and limitations as in the case of spatial processing based on real visual
Trento, Italy
images, and (ii) consistently activates several nodes of widely distributed cortical networks
*Correspondence:
in the brain.These nodes include areas within both, the dorsal fronto-parietal as well as ven-
Alexander T. Sack , Faculty of
Psychology and Neuroscience, tral occipito-temporal visual processing pathway, representing the “what” versus “where”
Maastricht University, aspects of spatial imagery. We here describe evidence from functional brain imaging and
Universiteitssingel 40, 6200 MD brain interference studies indicating systematic hemispheric differences within the dorsal
Maastricht, Netherlands.
fronto-parietal networks during the execution of spatial imagery. Importantly, such hemi-
e-mail: a.sack@maastrichtuniversity.nl
spheric differences and functional lateralization principles are also found in the effective
brain network connectivity within and across these networks, with a direction of informa-
tion flow from anterior frontal/premotor regions to posterior parietal cortices. In an attempt
to integrate these findings of hemispheric lateralization and fronto-to-parietal interactions,
we argue that spatial imagery constitutes a multifaceted cognitive construct that can be
segregated in several distinct mental sub processes, each associated with activity within
specific lateralized fronto-parietal (sub) networks, forming the basis of the here proposed
dynamic network model of spatial imagery.
Keywords: spatial imagery, object imagery, brain imaging, imagery and parietal cortex, imagery and premotor
cortex, imagery and frontal cortex, spatial attention, spatial working memory
SPATIAL IMAGERY – A MULTIFACETED Objects in visual imagery can be manipulated much like actual
COGNITIVE-PSYCHOLOGICAL CONSTRUCT objects. Hence, a mentally generated inner image can easily also be
Humans are capable of performing a variety of higher order cog- mentally transformed, distorted, or rotated in our mind. This can
nitive abilities such as problem solving, reasoning, contemplating, help to reason about the consequences of a potential correspond-
but also language comprehension, object recognition, spatial ori- ing physical manipulation (Kosslyn et al., 1998). (Visuo)Spatial
entation, or the vivid re-experience of previously perceived or imagery particularly refers to the inspection and evaluation of spa-
processed information stored in memory. All of these cognitive tial features (e.g., distance, relative position, configuration) and/or
functions require, and are to a large extent based on, our ability the spatial manipulation (e.g., rotation, shifting, reorienting) of
to generate, inspect, and manipulate inner mental representations mentally generated visual images. When we speak about spatial
of objects, events, and scenes that are not physically present. This imagery in the remainder of this article, we thus refer to the mental
ability of mental imagery thus describes a multi-facetted set of representation of visual objects, events, or scenes which are either
cognitive processes that are at the heart of most forms of abstract mainly defined by spatial characteristics (e.g., the visual imagina-
reasoning or contemplating (Kosslyn et al., 1995; Cohen et al., tion of a spatial configuration) and/or which require in addition to
1997; Kanwisher and Wojciulik, 2000; Riesenhuber and Poggio, the mere generation of the mental representation, a spatial analy-
2000). sis or manipulation to be mentally performed upon this mental
While mental imagery by itself is a multifaceted psychologi- visual image.
cal construct that shows conceptual and neurobiological overlap Spatial imagery, just like all forms of imagery, is by definition
with related cognitive processes such as attention and memory, a subjective, private experience that cannot be measured directly,
it is useful to also subdivide mental imagery according to the but has to be empirically inferred by indirect measures. These mea-
sensory modality based on which the mental representation is sures vary from subjective self reports on the vividness or size of
generated. In this sense, the processes that are involved in generat- the mental image, to more objective tasks such as mentally rotat-
ing, inspecting, and manipulating visual images in the absence of ing a visually presented object to assess whether it matches, or is
visual input are referred to as visual mental imagery (Finke, 1989). mirrored to, a second visual object (Shepard and Metzler, 1971).
www.frontiersin.org June 2012 | Volume 3 | Article 214 | 6

Sack and Schuhmann Network model of spatial imagery
It is believed that the completion of such mental rotations rely, at and in some tasks early visual cortex (EVC; Stokes et al., 2011)
least partly, on spatial mental imagery. In line with this rational, and/or even primary visual cortex (Kosslyn et al., 1999; Slotnick
some studies have shown that mental rotation tasks are indeed per- et al., 2005; de Borst et al., 2012). Likewise, brain regions within
formed by mentally rotating an object as if it were moving through the dorsal visual processing pathway are recruited during the spa-
the intermediate positions along a trajectory, as would occur if tial processing or manipulation of these mental representations
the object was physically rotated (Kosslyn et al., 1998; Carpen- (Kawashima et al., 1995;Mellet et al., 1995, 1996; Cohen et al.,
ter et al., 1999; Richter et al., 2000). Since accuracy and response 1996; Tagaris et al., 1997; Kosslyn et al., 1998;Sack et al., 2002, 2005,
latency of these mental spatial rotations can be objectively mea- 2008). These cortical regions within the dorsal pathway that in this
sured and compared with other experimental conditions of, e.g., sense are maybe more strictly related to the spatial aspect of spa-
real manual rotation (Sack et al., 2007), such psychophysical exper- tial imagery are the bilateral inferior and superior parietal lobule
iments offer a means for assessing spatial imagery performance in (SPL; Richter et al., 1997; Knauff et al., 2000;Trojano et al., 2000,
a behaviorally more controlled manner. 2002;Sack et al., 2002, 2005, 2008), bilateral intraparietal sulcus
Although it appeared that the question of which exact brain (IPS); precuneus; (Mellet et al., 1996; Trojano et al., 2000;Sack et al.,
areas are activated during spatial imagery largely depends on the 2002, 2005, 2008), middle forntal gyrus (MFG), supplementary
specific features of the imagery task being investigated, e.g., which motor area (SMA), frontal eye fields (FEF), and premotor cortex
spatial operation has to be performed based on which mental (PMC; Kawashima et al., 1995;Mellet et al., 1995, 1996; Cohen
object, the emerging picture of brain imaging studies is that our et al., 1996; Tagaris et al., 1997; Kosslyn et al., 1998; Richter et al.,
capability to mentally visualize, inspect, and manipulate objects 2000; Trojano et al., 2000; Lamm et al., 2001;Sack et al., 2002, 2005,
is subserved by distributed cortical networks that include regions 2008; Sack, 2009; de Borst et al., 2012). Regarding this spatial aspect
that are similarly activated when performing comparable percep- of spatial imagery, Thompson et al. (2009) suggested differentiat-
tual operations (Thompson et al., 2009; Cichy et al., 2011; but ing between visualizing spatial locations versus mentally trans-
see also Lee et al., 2011, nicely showing that although imagery forming locations, both relying on distinct neural sub networks
and perception have similar neural substrates, they may involve within the dorsal pathway. Concretely, whereas the visualization
different network dynamics; Seurinck et al., 2011). Another impor- of spatial locations recruited mainly areas within occipito-parietal
tant and converging finding of these previous imaging studies is sulcus, medial posterior cingulate, and precuneus, mental spatial
that both conceptually and in terms of underlying neural mecha- transformations were correlated more with activation in supe-
nisms it seems important to distinguish cortical regions and neural rior portions of the parietal lobe and in the postcentral gyrus.
mechanisms involved in tasks that require participants to mentally Still, since any spatial inspection or manipulation during spatial
represent specific object categories (Ishai et al., 2000) or specific imagery requires some sort of mental (object) representation upon
features of objects (e.g., color, size, shape), from those cortical which the spatial operation can be based and performed on, the
regions and neural mechanisms involved in tasks that explic- core neural network of spatial imagery typically includes brain
itly require processing of spatially coded information or spatial areas of both the dorsal fronto-parietal as well as ventral occipito-
manipulation (Trojano et al., 2000). This distinction of object ver- temporal visual processing pathway. Figure 1 depicts this core
sus spatial imagery can be regarded as analogs to the dichotomy network of spatial imagery in the brain, segregated and color-
between ventral (what) versus dorsal (where) information pro- coded in order to distinguish the spatial dorsal (red-colored) from
cessing during visual perception (Mishkin and Ungerleider, 1982; the content ventral (rose-colored) network in the brain activated
Mishkin et al., 1983;Haxby et al., 1991, 1994). during spatial imagery.
THE NEUROBIOLOGICAL SEGREGATION OF WHAT AND HEMISPHERIC LATERALIZATION DURING SPATIAL IMAGERY
WHERE DURING SPATIAL IMAGERY As can be seen in Figure 1, most functional imaging studies show
The neurobiological mechanisms underlying spatial imagery are bilateral fronto-parietal networks to be activated during the execu-
characterized by widely distributed cortical networks with a mul- tion of spatial imagery. The fact that both, left and right posterior
titude of nodes and interaction-patterns in the brain. Numerous parietal cortex (PPC; mostly SPL and IPS) is recruited during
neuropsychological (Levine et al., 1985; Farah et al., 1988) and spatial imagery, is, at first glance, in contrast to most neuropsycho-
neuroimaging studies (Cohen et al., 1996;Mellet et al., 1996, 1998; logical studies on patients with focal brain lesions which generally
D’Esposito et al., 1997; Richter et al., 1997; Knauff et al., 2000; propose a dominant role of the left hemisphere in visual imagery
Trojano et al., 2000) have aimed at unraveling the neural foun- (Farah et al., 1985; D’Esposito et al., 1997). In a critical clinical
dations of mental imagery using a wide variety of imagery tasks review on visual mental imagery, Trojano and Grossi (1994) pre-
(for a review see Kosslyn et al., 2001). These imaging studies have sented a number of single cases as well as group studies which
consistently revealed that the pure imagination and mental rep- demonstrate a dominant role of left posterior parietal areas for
resentation of a specific mental object results in neural activity mental imagery. Nonetheless, the authors also reported evidence
within category-specific occipital-temporal regions of the ventral of the role of the right hemisphere in visuospatial imagery as
visual processing pathway (Ishai et al., 2000, 2002; O’Craven and well as in perceptual visuospatial processing. Right brain dam-
Kanwisher, 2000), including superior occipital areas (Mellet et al., aged patients with neglect also show neglect symptoms in imagery
1995, 1996; D’Esposito et al., 1997; de Borst et al., 2011), inferior tasks, and non-neglect right hemisphere patients show visuospatial
temporal regions (Carpenter et al., 1999; Mechelli et al., 2004; de deficits in perceptual visuospatial processing and during imagery
Borst et al., 2012), parahippocampal cortex (de Borst et al., 2012), tasks. While the left hemisphere seems to have a specific role for
Frontiers in Psychology | Perception Science June 2012 | Volume 3 | Article 214 | 7

FIGURE 1 | The core neural network of spatial imagery. This figure lobe; IPS, Intraparietal sulcus; MFG, middle forntal gyrus; DLPFC,
depicts the neural network of spatial imagery including brain areas of both dorsolateral prefrontal cortex; FEF, frontal eye fields; PMC, premotor
the dorsal fronto-parietal (red-colored) as well as ventral occipito-temporal cortex; Precuneus; and SMA, supplementary motor area. Most prominent
(rose-colored) visual processing pathway. It summarizes in one figure the regions within the ventral occipito-temporal network include bilateral EVC,
different regions identified in various imagery studies as described in the early visual cortex; IT, inferior temporal cortex; IO, inferior occipital cortex;
Section “The neurobiological segregation of what and where during PHG, parahippocampal gyrus. The mesial superior frontal gyrus (mSFG)
spatial imagery” of the current manuscript. Most prominent regions within plays a special integrative role in the context of spatial imagery and is
the dorsal fronto-parietal network include bilateral SPL, superior parietal therefore color-coded separately.
mental imagery, the right hemisphere seems to be of a more general correlate with reaction time during spatial imagery performance.
relevance for visuospatial functions (Trojano and Grossi, 1994). These results support the involvement of both parietal lobes in
Our group has contributed to the question of hemispheric mental imagery, but suggest that each parietal lobe might have a
lateralization within bilateral parietal cortex (PC) during spa- distinct functional role at different moments in time. The sequen-
tial imagery by using conventional functional magnetic reso- tial activation from left to right suggests that the early left and late
nance imaging (Trojano et al., 2000), fMRI mental chronometry right parietal activation during spatial imagery support different
(Formisano et al., 2002), repetitive (Sack et al., 2002), and time- components of the cognitive process, for example the generation
resolved (Sack et al., 2005) transcranial magnetic stimulation and subsequent analysis of the visual image. We therefore con-
(TMS) experiments. By using a spatial imagery task that involves cluded that within the bilateral PPC activity during spatial imagery,
the generation as well as spatial comparison of mental images, we the left PPC underlies the generation of mental images, while the
demonstrated, using event-related fMRI, that the bilateral parietal right PPC subserves the spatial processing upon these images.
activity associated with this task includes a temporal activation Such modular models of spatial imagery that propose a division
sequence from left to right PC. When relating and modeling dif- of labor between hemispheres in which the generation of men-
ferent features of the fMRI responses to the behavioral measures, tal representation from memory rely primarily on structures in
we found that the duration of activation of the early left pari- the posterior left hemisphere, while spatial operations upon these
etal activation and the onset of the (late) right parietal activation mental representation rely primarily on structures in the posterior

right hemisphere, also provide a solution to the aforementioned Carpenter et al., 1999; Knauff et al., 2000;Trojano et al., 2000,
apparent discrepancy between lesion and imaging studies with 2002; Lamm et al., 2001;Sack et al., 2002, 2005, 2008; de Borst
regard to the hemispheric lateralization of visual imagery, and are et al., 2012). With regard to the spatial processing component of
in this sense in agreement with both neuropsychological lesion as spatial imagery, a strong focus has been put on the bilateral PPC
well as brain imaging findings of spatial imagery (for review see activation due to the prominent role of PPC within the dorsal
Sack, 2009). However, from fMRI measurements alone one cannot spatial processing stream. Carpenter et al. (1999) correlated the
assess the exact functional necessity or behavioral contribution of increment of reaction time during mental rotation of cubes with
a given brain area for a specific mental sub process, such as mental changes in regional cerebral activation. With higher angular dis-
image generation versus spatial analysis. We therefore used non- parity, activation increased in the parietal lobes bilaterally, but
invasive functional brain stimulation to focally and transiently not in the temporal lobe. This discrepancy indicates that the PC
disrupt neural processing in either left or right PC during spa- plays a central role in the visuospatial transformations of mental
tial imagery, and assessed the respective behavioral effect of this rotation (Goebel et al., 1998; Formisano et al., 2002; Sack et al.,
unilateral functional lesion within PPC on spatial imagery perfor- 2002) whereas the ventral (temporal) pathway, which is essential
mance (Sack et al., 2002). This brain interference study revealed for identifying a figure, does not specifically support this oper-
a hemisphere-specific effect of parietal stimulation with only the ation. This again seems to strengthen and justify the prominent
right parietal disruption leading to spatial imagery impairments. role of PPC in spatial imagery research. However, importantly, this
These results contribute new constraints to the modular model study also revealed that the activity in the motor areas of the frontal
of bilateral activation in spatial imagery and are at first glance lobe was significantly higher during the mental rotation paradigm
not in accordance with the aforementioned hemispheric lateral- as compared to a motor control condition, suggesting that “the so-
ization and division of labor between hemispheres during spatial called motor areas are not simply involved in motor planning and
imagery. Indeed, if left PPC underlies the generation of men- execution” (Carpenter et al., 1999), but play a crucial role in the
tal representations and right PPC reflects the spatial operations computation of imagined motion of objects as well. In a similar
upon these mental representations, a suppression of either of these vein, Richter et al. (2000) used fMRI to investigate the participa-
brain regions should result in impaired spatial imagery perfor- tion of the neocortical motor areas in the Shepard and Metzler’s
mance. Fortunately, based on a combined fMRI and transcranial (1971) mental rotation task. Seven regions of interest (ROIs) were
magnetic stimulation study (Sack et al., 2002), we were able to analyzed separately: Left and right SPL, SMA, and left and right
further fractionate specialized processing components in the right premotor areas. The results showed that the observed activation
PPC and revealed the existence of highly dynamic compensatory in premotor areas was likely related to the very execution of the
mechanisms between the left and right hemisphere during the exe- mental rotation task (Richter et al., 2000).
cution of spatial imagery (Sack et al., 2005). This study suggested These studies thus indicate the potential functional contribu-
that although the left PPC is predominantly specialized in men- tion of prefrontal and premotor brain areas during spatial imagery.
tal image generation and the right PPC in spatial comparisons of The question remained, however, whether these prefrontal and
imagined content, the right hemisphere is also able to immediately premotor activities during mental rotation tasks are more related
compensate for (virtual) lesions of the left hemisphere by taking to the potential involvement of visual working memory rather
over its specific function, but not vice versa. Hence, in case of left than being critical neural structures for the visual imagery process
parietal functional lesion, the right PC will now subserve both per se. Or in other words, what would happen to the here iden-
functions, mental image generation, and spatial analysis of the tified premotor and prefrontal activations in case of pure visual
mental image. Discrepancies across studies concerning the hemi- imagery, i.e., when generating mental representations of objects
spheric lateralization during mental imagery likely arise because that have never been perceived before? In such cases, the gener-
different aspects of imagery are carried out by different parts of ation of visual images does not result from the reactivation of
a bi-hemispheric neural network. The fact that an isolated deficit previously stored memories but does result from an online con-
of the ability to generate inner visual images following unilateral struction of images based on the processing of, e.g., verbal instruc-
lesion is clinically hardly reported could also be explained on the tions and their encoding in a visual format. Mellet et al. (1996,
basis of the compensatory processes revealed in our study (Sack, 1998, 2000) used PET to monitor regional cerebral blood flow
2009, 2010). Interestingly, such hemispheric asymmetries between variations while participants were constructing mental images of
left and right PPC apply to both, the direct functional relevance objects made of three-dimensional cube assemblies from acousti-
(only right parietal disruption leads to behavioral impairments) cally presented instructions. Compared to a control condition, the
as well as ability of inter-hemispheric compensation (right PC can mental construction task specifically activated a bilateral occipito-
compensate for left PC, but not nice versa). parietal-frontal network, including the superior occipital cortex,
the inferior PC, and again also the PMC. These studies thus sug-
DYNAMIC ANTERIOR-TO-POSTERIOR BRAIN NETWORK gest that in addition to the well-established functional role of
CONNECTIVITY DURING SPATIAL IMAGERY posterior parietal cortices during spatial imagery, also the pre-
The execution of various spatial imagery paradigms consistently frontal and premotor activations revealed during imagery task are
activates core areas of the dorsal fronto-parietal visual path- of direct functional relevance for the imagery performance and
way, including bilateral parietal, prefrontal, and premotor areas likely also sub serve specific cognitive sub functions within the
(Kawashima et al., 1995;Mellet et al., 1995, 1996; Cohen et al., multifaceted cognitive-psychological construct of imagery. How-
1996; Richter et al., 1997; Tagaris et al., 1997; Kosslyn et al., 1998; ever, as described above, while some of the previous functional

imaging studies on spatial imagery tried to subscribe different might sub serve the necessary maintenance of the spatially rotated
cognitive (sub) functions to, e.g., left versus right PPC (Formisano visual mental object in spatial short-term working memory (Tul-
et al., 2002;Sack et al., 2002, 2005), a systematic investigation ving et al., 1994; Buckner et al., 1995; Moscovitch et al., 1995;
of the specific functional contribution and/or the exact spatio- Roland and Gulyas, 1995; Smith et al., 1995; Courtney et al., 1996;
temporal interactions with the always co-activated prefrontal and Cohen et al., 1997). These findings suggest that the activation flow
premotor brain regions during spatial imagery is missing. All underlying the construction and spatial transformation of visual
studies discussed so far either focused exclusively on the PPC mental images first recruits premotor regions which then project
and neglected all additionally activated brain regions (Trojano to, or receive information on demand from, parietal regions in a
et al., 2000; Formisano et al., 2002;Sack et al., 2002, 2005) or top-down manner, putting prefrontal, and PMC into a new and
simply descriptively reported the co-activation of anterior precentral focus also during higher cognitive functions (Sack et al.,
motor and prefrontal brain regions within large fronto-parietal 2008). Such functionally coupled activations of the parietal and
networks (Mellet et al., 1998, 2000; Carpenter et al., 1999; Richter premotor cortices have also been described for other cognitive
et al., 2000), without analyzing the network dynamics between functions (Abe and Hanakawa, 2009) and visuospatial tasks, such
these anterior premotor/prefrontal and posterior parietal activa- as spatial localization (Haxby et al., 1994) or shifting of spatial
tions during spatial imagery. These shortcomings were mainly due attention (Corbetta et al., 1993), and in situations explicitly involv-
to respective boundaries in spatial and temporal resolution of the ing the spatial working memory (Jonides et al., 1993; Courtney
functional imaging techniques being used, and due to the unavail- et al., 1996). The exchange of information between the premotor
ability of more advanced analyses tools for functionally segregating regions and the dorsal route thus appears to be a general feature
the acquired brain imaging data into separate networks of effective during spatial processing, whatever the nature of the initial input.
brain connectivity. It is thus likely that the parietal “perceptual” pole and the frontal
In our recent work, our group aimed to address this short- “motor” pole systematically exchange spatial information, whether
coming (Sack et al., 2008). Participants were asked to mentally a motor action is envisioned or not, thus executing the encoding
construct an inner image of either an entirely new object or of a spatial environment in its descriptive and behavioral aspects.
a new configuration of objects, simply based on either visually
or verbally presented instructions. Importantly, participants were INTEGRATION BETWEEN DORSAL AND VENTRAL PATHWAYS
required to construct and spatially rotate these abstract mental DURING SPATIAL IMAGERY
images generated from sequentially presented instructions, mean- In the previous sections, we have described and shown that
ing that only pieces of the final mental object were provided in within the dorsal fronto-parietal network activated during spa-
successive steps, sequentially building up the final mental object tial imagery, a direction of information flow seems to exist from
that had never been perceptually encountered as a whole. This anterior frontal/premotor to posterior parietal cortices. However,
behaviorally controlled spatial imagery paradigm was investigated in addition to the dorsal fronto-parietal network, all imagery par-
using time-resolved event-related functional magnetic resonance adigms also activate several nodes of the ventral occipito-temporal
imaging and analyzed based on data-driven and multivariate fMRI visual processing pathway (Ishai et al., 2000, 2002; O’Craven and
analysis tools. By accounting for the full spatial pattern of brain Kanwisher, 2000), including superior occipital areas (Mellet et al.,
activity measured simultaneously at many locations, we function- 1995, 1996; D’Esposito et al., 1997; de Borst et al., 2012), inferior
ally segregated an early from a late premotor-parietal-occipitotemporal regions (Carpenter et al., 1999; Mechelli et al., 2004; de
temporal-cortex (OTC) and late premotor-prefrontal activation Borst et al., 2012), parahippocampal cortex (de Borst et al., 2012),
network. We revealed, using effective brain connectivity analy- and in some tasks even primary visual cortex (Kosslyn et al., 1999;
ses, that the information coming from sensory brain regions was de Borst et al., 2012). This seems to make perfect sense because any
first sent to bilateral PMC and then to bilateral medial dorsal PC. spatial operation or spatial processing in spatial imagery requires
The early left PC received additional input from bilateral occipito- some sort of mental (object) representation upon which the spa-
parietal regions. One might thus speculate that this early bilateral tial inspection or manipulation can be based and performed on.
premotor-(medial dorsal) parietal activation network underlies Subsequently, the core neural network of spatial imagery includes
the online processing of the sequentially presented modality- brain areas of both the dorsal fronto-parietal as well as ventral
independent spatial instructions (Corbetta et al., 1993; Jonides occipito-temporal processing pathway, representing the “what”
et al., 1993; Haxby et al., 1994; Courtney et al., 1996; Prabhakaran and “where” aspects of spatial imagery (see Figure 1).
et al., 2000). In contrast, the identified late premotor-parietal net- An open question, however, then is how the necessary dynamic
work showed a clear hemispheric difference with only the late left interaction between the “what” and “where” aspects of spatial
PMC projecting back to bilateral parietal regions while at the same imagery is integrated in the brain. de Borst et al. (2011) directly
time sending neural signals to bilateral OTC. This specific effec- addressed this question by investigating the functional role of the
tive connectivity network might thus represent the juxtaposing frontal regions and their interaction with the “what,” “where” and
of the sequentially presented stimuli and thus the successive con- early visual regions during complex visuospatial scene imagery.
struction of the slowly emerging final mental object representation Participants were required to perform detailed scene imagery that
during imagery (Roland and Gulyas, 1995;Mellet et al., 1996, 1998; captured both object and spatial mental imagery aspects, while
D’Esposito et al., 1997). During this late premotor-parietal activity measuring all imagery-relevant network nodes in the brain and
network, neural input was also sent to bilateral prefrontal cortex their relative temporal onset of activation using event-related
(PFC). This late bilateral premotor-prefrontal activation network fMRI. In addition, electroencephalography (EEG) was recorded to

validate the fMRI latency results and to derive more information the frontal regions during the later stages of image construction,
on the underlying functional roles from the involved frequency rather than being a dorsal starting point of spatial imagery. In
bands. This study could nicely demonstrate that the “what” and line with this integrative role of the anterior premotor and pre-
“where” aspects of spatial imagery are integrated into one visu- frontal regions in spatial imagery, Abe and Hanakawa (2009) also
ally imagined scene by the frontal regions, including PMC, right suggested that the functional interplay through the prefrontal-
MFG, and mesial SFG (mSFG). The mSFG seemed to be most premotor connections may mediate the integration of specific
crucial for this integration process because it was activated earliest sub-operations for multi-step cognitive manipulation.
and predicted later imagery performance. The early and behav-
iorally relevant involvement of the frontal regions suggest that DYNAMIC NETWORK MODEL OF SPATIAL IMAGERY
these frontal regions indeed “orchestrate” the ventral occipital- In an attempt to summarize, converge, and integrate the brain
temporal “what” and dorsal parietal “where” regions during spatial imaging findings of our own and other groups on spatial imagery
imagery. Hence, the frontal regions and in particular the mSFG over the past 10 years, Figure 2 depicts the here proposed new
seem to integrate those areas encoding the detailed mental rep- dynamic network model of spatial imagery.
resentation with those areas encoding the spatial layout, in order Spatial imagery consistently activates several nodes within
to form and maintain the subjective experience of one coherent widely distributed cortical networks in the brain. Importantly,
mental picture. In line with this interpretation, several working these nodes include areas within both, the dorsal fronto-parietal
memory studies have also suggested that mSFG plays a role in as well as ventral occipito-temporal visual processing pathway (all
the integration of visual and spatial features during visual short- silver-shaded areas in Figure 2). Most prominent regions within
term memory (VSTM; Mitchell et al., 2000; Prabhakaran et al., the dorsal pathway during spatial imagery include bilateral PC,
2000; Munk et al., 2002). In accordance with the already described PMC, and PFC. Likewise, most prominent regions of the ventral
anterior-to-posterior information flow within the dorsal fronto- pathway activated during spatial imagery are located along the
parietal network during spatial imagery (Sack et al., 2008) this OTC, and include inferior temporal regions and parahippocam-
study also showed a rather late involvement of bilateral PPC, sug- pal cortex, but also superior occipital areas and in some conditions
gesting again that these PPC activations seem to be drawn on by even primary visual cortex (see Figure 2, silver-shaded areas).
FIGURE 2 | Dynamic network model of spatial imagery. This figure left-lateralized PMC-PC-OTC activation network (solid curve color-coded in
depicts the here proposed dynamic network model of spatial imagery. red). Spatial imagery thus first recruits bilateral anterior premotor cortices,
Spatial imagery consistently activates several nodes within both, the dorsal which then send neural information to, or receive on demand neural
fronto-parietal as well as ventral occipito-temporal visual processing information from, bilateral parietal cortices. This model thus proposes that
pathway (silver-shaded areas). Most prominent regions within the dorsal the well-established PPC activation during spatial imagery seems to be
pathway during spatial imagery include bilateral PC, Parietal Cortex; PMC, drawn on by the frontal regions at later stages in the course of imagery,
Premotor Cortex; and PFC, Prefrontal Cortex. Likewise, most prominent rather than being a dorsal starting point of spatial imagery, as previously
regions of the ventral pathway activated during spatial imagery are located suggested. Moreover, in order to form one coherent mental visuospatial
along the Occipito-Temporal-Cortex (OTC), and include inferior temporal picture, all these content and spatial aspects and segregated processing
regions and parahippocampal cortex, but also superior occipital areas and in stages of spatial imagery need to be integrated at a brain system level. This
some conditions even primary visual cortex (silver-shaded areas). During integration process is done by the mesial SFG. Mesial SFG orchestrates
spatial imagery, these two pathways can be labeled as representing the remote ventral occipital-temporal “what” regions and dorsal parietal
CONTENT (curve color-coded in light blue) versus SPATIAL (curve “where” regions in order to integrate areas encoding the detailed mental
color-coded in dark blue) aspects of spatial imagery. The areas within the visual representation with those areas encoding the spatial layout or
fronto-parietal dorsal network dynamically exchange information during manipulation. Finally, the late neural dynamic information flow between
spatial imagery with a direction of information flow from anterior bilateral PMC and PFC (curve color-coded in light red) represents the
frontal/premotor regions to posterior parietal cortices. An early bilateral necessary maintenance of the now spatially processed or manipulated and
PMC-PC (solid curve color-coded in dark red) can be segregated from a later thus integrated mental object in (spatial) visual short-term working memory.

The parallel processing within dorsal and ventral networks cognitive-psychological constructs, it is undisputed that the abil-
makes perfect sense in case of spatial imagery. It shows that ity to maintain visual information online in working memory
indeed the imagination and mental representation of any spe- largely depends on mental imagery, and likewise, performing spa-
cific mental object or scene always results in neural activity within tial manipulations upon mentally generated visual images shares
category-specific occipital-temporal regions of the ventral visual large conceptual overlap with the concept of (covert) visuospa-
processing pathway. This is true for all forms and aspects of visual tial attention and memory. In an attempt to combine aspects of
imagery, whether or not it includes a spatial processing compo- the so-called analog (James, 1890; Paivio, 1971; Kosslyn, 1980;
nent. In case of spatial imagery, however, additional brain regions Kosslyn and Ochsner, 1994) versus propositional (Anderson and
within the dorsal visual processing pathway are recruited because Bower, 1973; Pylyshyn, 1973) theory of mental imagery, Kosslyn
of the here required processing of spatially coded information (1980) proposed a computational hybrid imagery model consist-
and/or spatial manipulation of these mental representations. In ing of three basic components: a visual buffer, long-term stored
this sense, the two pathways can be labeled as representing the representations, and image-processing operations. Visual images
CONTENT (Figure 2; curve color-coded in light blue) versus SPA- are generated by retrieving information from long-term visual
TIAL (Figure 2; curve color-coded in dark blue) aspects of spatial memory (LTM) and constructing them in a spatial format in
imagery (Figure 2). the visual buffer. Although the LTM contains both analog and
Importantly, the areas within the fronto-parietal dorsal network propositional components, the final visual image is an analog rep-
dynamically exchange information during spatial imagery with resentation. Images in the buffer can be mentally manipulated and
a direction of information flow from anterior frontal/premotor transformed (e.g., by rotation or scanning) and then be inspected
regions to posterior parietal cortices. A spatio-temporal segrega- for new information (Palmer, 1999); a process likely largely medi-
tion has been suggested, dissociating an early bilateral PMC-PC ated by spatial attention mechanisms. In line with this depictive
(Figure 2; curve color-coded in dark red) from a later left- view of visual mental imagery, Slotnick et al. (2005) demonstrated
lateralized PMC-PC-OTC activation network (Figure 2; curve that visual mental imagery can evoke cortical activity with precise
color-coded in red). Importantly, spatial imagery thus first recruits visual field topography, i.e., imagery-induced retinotopic maps
bilateral anterior premotor cortices, which then send neural infor- that are similar to the perception maps.
mation to, or receive on demand neural information from, bilat- As evident from the attempts to segregate and subscribe dif-
eral parietal cortices. This model thus proposes that the well- ferent cognitive sub functions to the revealed sub networks dur-
established PPC activation during spatial imagery seems to be ing spatial imagery using functional imaging, these psychological
drawn on by the frontal regions at later stages in the course labels or cognitive-psychological constructs of the revealed brain
of imagery, rather than being a dorsal starting point of spatial activation pattern often have to fall back on other cognitive-
imagery, as previously suggested. psychological constructs such as attention and memory. In fact,
So far, the model describes the parallel processing within attention and memory processes are consistently used to explain
the ventral CONTENT and dorsal SPATIAL network of spatial and describe the various mental sub processes and their underly-
imagery, as well as the dynamic anterior-to-posterior informa- ing neural activations during the execution of (spatial) imagery.
tion flow within the dorsal fronto-parietal activation network. This also holds true the other way around: attentional modula-
However, in order to form one coherent mental visuospatial pic- tion occurs in the absence of any phenomenal experience, e.g., in
ture, all these aspects and segregated processing stages need to be form of expectations. Kastner et al. (1999) described attention-
integrated at a brain system level. We could recently suggest that modulated activity in fronto-parietal areas and in primary visual
this integration process is likely done by the co-activated frontal cortex during a visuospatial attention task without visual stim-
regions, in particular the mSFG. mSFG seems to literally orches- ulation. Regions in parietal and frontal cortex responded when
trate remote ventral occipital-temporal “what” regions and dorsal observers covertly pay attention to a peripheral location in expec-
parietal “where” regions in order to integrate areas encoding the tation of the stimulus occurrence. Therefore, attention is not
detailed mental visual representation with those areas encoding necessarily associated with visual perception. However, although
the spatial layout or manipulation. In line with this idea, neural several neuroimaging studies have shown that attention can mod-
dynamic information flow during spatial imagery was revealed ulate extrastriate cortical regions within the dorsal and ventral
also late between bilateral PMC and PFC, likely representing the processing streams (Kanwisher and Wojciulik, 2000; Martinez
necessary maintenance of the spatially processed or manipulated et al., 2001; Yantis et al., 2002), the role of the primary visual cor-
and thus integrated mental object in (spatial) visual short-term tex remains controversial (Kastner et al., 1999; Posner and Gilbert,
working memory (Figure 2; curve color-coded in light red). 1999; Sengpiel and Hubener, 1999). But what psychological and
neural mechanisms are responsible for the maintenance of spatial
QUO VADIS SPATIAL IMAGERY? attention in the absence of visual stimuli? And what are the con-
Besides the specific questions on the neurobiology, lateraliza- ceptual differences to spatial imagery and spatial working memory
tion, and/or spatio-temporal dynamics within and across the processes? Mental imagery is defined as a perceptual experience in
described fronto-parietal and occipito-temporal network activity the absence of an appropriate physical stimulus (Finke, 1989).
during spatial imagery, a striking conceptual uncertainty occurs Conceptually, attention and imagery can both be characterized
on how to differentiate the concept of spatial imagery to related by specific top-down processes that modulate extrastriate cor-
phenomena such as spatial attention or spatial visual working tex activity in the absence of visual stimulation. On a neuronal
memory. While these related processes are often treated as separate level both, spatial imagery and spatial attention show immense

functional overlapping: mental imagery is generally, like attention, Hence, it might very well be that despite all the conceptual and
accompanied by an activation of a fronto-parietal network (Tro- apparent neurobiological overlap between imagery, attention, and
jano et al., 2000; Formisano et al., 2002; Sack et al., 2002). Several memory, clear differences and neurobiological segregations can
imagery studies have shown that mental imagery, like attention, be found and described on the basis of, e.g., cleaner experimental
can modulate extrastriate cortices (Le Bihan et al., 1993; Roland designs directly comparing the different processes, and/or more
and Gulyas, 1995; D’Esposito et al., 1997). Finally, some imagery elaborated multivariate brain imaging analysis tools. Another
studies have even, like attention, revealed primary visual cortex elegant approach might be to use functional brain interference
activity during visual imagery (Kosslyn et al., 1999; de Borst et al., techniques such as transcranial magnetic stimulation (TMS) to
2011; Slotnick et al., 2005). However, the involvement of occipital chart the time point at which the identified similar neural struc-
areas during imagery is, like in attention, still a matter of debate tures activated by imagery and memory are functionally relevant
(Kosslyn and Ochsner, 1994; Roland and Gulyas, 1995; Mellet for either or both of these processes. Using this approach, Catta-
et al., 2000). The ability to maintain spatial information online neo et al. (2009), e.g., compared the functional contribution of
in memory depends on spatial attention and might be mediated EVC in short-term memory retention and visual mental imagery
by spatial imagery. Imagery might in general aid any cognitive at different moments in time using chronometric TMS. They
functions in the respective sensory domain. It has, for example, could show that short pulses of functional interference applied
been claimed that visual imagery ability correlates with visuospa- to EVC at the start of the retention interval increased reaction
tial memory span (Kail, 1997). Keogh and Pearson (2011) showed times for the memory trials, but not for the imagery trials; while
that performance in visual working memory – but not iconic visual later TMS pulses over EVC affected both processes, memory and
memory – can be predicted by the strength of mental imagery. imagery. In other words, while both visual imagery and VSTM
This raises the question whether spatial imagery, spatial atten- recruited identical neural structures within EVC, the time point of
tion, and memory in the end represent very similar or even functional recruitment was significantly different between these
identical neuronal processes that can only be differentiated on two processes. This enabled the authors to segregate between
a psychological level as servants for a thinking framework among imagery and memory in the temporal rather than spatial neural
neuroscientists. On a neuronal level, these two or three so far rather domain. More studies like these, directly comparing and segregat-
independently investigated constructs might prove to recruit sim- ing the neurobiology underlying imagery, attention, and memory
ilar or even identical neural structures. Hence, what might be a within one experimental session are needed to clearly separate
very crucial and important semantic and conceptual difference and differentiate the different mental sub processes involved in
for psychology and neuroscience might for our brain in the end spatial imagery from those involved in attention and memory.
simply require the recruitment of identical networks of in this In the end, we should be able to explain the individual cogni-
sense identical processes. tive, phenomenological, psychological, and behavioral differences
However, while this might in the end indeed show to be the case, within spatial imagery (see, e.g., Borst and Kosslyn, 2010), as well
such a conclusion would at this point be premature. One prob- as the differences between imagery, attention, and memory, by
lem of most paradigms investigated so far in the context of visual linking each of these processes to different underlying neurobi-
imagery and/or VSTM is the fact that they are usually conceptu- ological processes and spatio-temporal network dynamics in the
ally confounded by not representing pure operationalizations of brain.
the cognitive-psychological construct under investigation. Most
cognitive and neuroscientific studies of mental imagery required ACKNOWLEDGMENTS
participants to learn and memorize visual images, or recall The research leading to these results has received funding from the
images from past autobiographical experiences. Such “imagery” European Research Council under the European Union’s Seventh
paradigms are by definition confounded by processes involving Framework Programme (FP7/2007-2013)/ERC Grant agreement
visual working or episodic memory. no. (263472).
REFERENCES Functional anatomical studies Cichy, R. M., Heinzle, J., and Y., Rosen, B. R., and Belliveau, J. W.
Abe, M., and Hanakawa, T. (2009). of explicit and implicit memory Haynes, J. D. (2011). Imagery (1996). Changes in cortical activity
Functional coupling underlying retrieval tasks. J. Neurosci. 15, and perception share corti- during mental rotation. A mapping
motor and cognitive functions of 12–29. cal representations of content study using functional MRI. Brain
the dorsal premotor cortex. Behav. Carpenter, P. A., Just, M. A., Keller, T. A., and location. Cereb. Cortex 22, 119(Pt 1), 89–100.
Brain Res. 198, 13–23. Eddy, W., and Thulborn, K. (1999). 372–380. Corbetta, M., Miezin, F. M., Shul-
Anderson, J. R., and Bower, G. H. Graded functional activation in the Cohen, J. D., Perlstein, W. M., Braver, man, G. L., and Petersen, S. E.
(1973). Human Associative Memory. visuospatial system with the amount T. S., Nystrom, L. E., Noll, D. (1993). A PET study of visu-
New York: Winston. of task demand. J. Cogn. Neurosci. 11, C., Jonides, J., and Smith, E. ospatial attention. J. Neurosci. 13,
Borst, G., and Kosslyn, S. M. (2010). 9–24. E. (1997). Temporal dynamics of 1202–1226.
Individual differences in spatial Cattaneo, Z., Vecchi, T., Pascual-Leone, brain activation during a work- Courtney, S. M., Ungerleider, L. G., Keil,
mental imagery. Q. J. Exp. Psychol. A., and Silvanto, J. (2009). Con- ing memory task. Nature 386, K., and Haxby, J. V. (1996). Object
(Hove) 63, 2031–2050. trasting early visual cortical activa- 604–608. and spatial visual working mem-
Buckner, R. L., Petersen, S. E., Oje- tion states causally involved in visual Cohen, M. S., Kosslyn, S. M., Breiter, H. ory activate separate neural systems
mann, J. G., Miezin, F. M., Squire, imagery and short-term memory. C., Digirolamo, G. J., Thompson, W. in human cortex. Cereb. Cortex 6,
L. R., and Raichle, M. E. (1995). Eur. J. Neurosci. 30, 1393–1400. L., Anderson, A. K., Brookheimer, S. 39–49.

de Borst, A. W., Sack, A. T., Jansma, famous faces: effects of mem- Thompson, W. L., Ganis, G., Sukel, Mellet, E., Tzourio, N., Denis, M., and
B. M., Esposito, F., De Martino, F., ory and attention revealed K. E., and Alpert, N. M. (1999). The Mazoyer, B. (1995). A positron emis-
Valente, G., Roebroeck, A., Di Salle, by fMRI. Neuroimage 17, role of area 17 in visual imagery: sion tomography study of visual, and
F., Goebel, R., and Formisano, E. 1729–1741. convergent evidence from PET and mental, spatial exploration. J. Cogn.
(2011). Integration of “what” and Ishai, A., Ungerleider, L. G., and Haxby, rTMS. Science 284, 167–170. Neurosci. 7, 433–445.
“where” in frontal cortex during J. V. (2000). Distributed neural sys- Lamm, C., Windischberger, C., Mishkin, M., and Ungerleider, L.
visual imagery of scenes. Neuroim- tems for the generation of visual Leodolter, U., Moser, E., and Bauer, G. (1982). Contribution of stri-
age 60, 47–58. images. Neuron 28, 979–990. H. (2001). Evidence for premotor ate inputs to the visuospatial func-
de Borst, A. W., Sack, A. T., Jansma, James, W. (1890). The Principles of Psy- cortex activity during dynamic tions of parieto-preoccipital cortex
B. M., Esposito, F., De Martino, F., chology, Vol. 1 and 2. New York: visuospatial imagery from single- in monkeys. Behav. Brain Res. 6,
Valente, G., Roebroeck, A., Di Salle, Henry Holt and Co, Inc. trial functional magnetic resonance 57–77.
F., Goebel, R., and Formisano, E. Jonides, J., Smith, E. E., Koeppe, R. A., imaging and event-related slow Mishkin, M., Ungerleider, L. G., and
(2012). Integration of “what” and Awh, E., Minoshima, S., and Mintun, cortical potentials. Neuroimage 14, Macko, K. A. (1983). Object vision
“where” in frontal cortex during M. A. (1993). Spatial working mem- 268–283. and spatial vision: two cortical path-
visual imagery of scenes. Neuroim- ory in humans as revealed by PET. Le Bihan, D., Turner, R., Zeffiro, T. ways. Trends Neurosci. 6, 414–417.
age 60, 47–58. Nature 363, 623–625. A., Cuenod, C. A., Jezzard, P., Mitchell, K. J., Johnson, M. K., Raye,
D’Esposito, M., Detre, J. A., Aguirre, G. Kail, R. (1997). Processing time, and Bonnerot, V. (1993). Activa- C. L., Mather, M., and D’Esposito,
K., Stallcup, M., Alsop, D. C., Tip- imagery, and spatial memory. J. Exp. tion of human primary visual cor- M. (2000). Aging and reflective
pet, L. J., and Farah, M. J. (1997). Child Psychol. 64, 67–78. tex during visual recall: a mag- processes of working memory: bind-
A functional MRI study of mental Kanwisher, N., and Wojciulik, E. (2000). netic resonance imaging study. ing and test load deficits. Psychol.
image generation. Neuropsychologia Visual attention: insights from brain Proc. Natl. Acad. Sci. U.S.A. 90, Aging 15, 527–541.
35, 725–730. imaging. Nat. Rev. Neurosci. 1, 11802–11805. Moscovitch, C., Kapur, S., Kohler, S.,
Farah, M. J., Gazzaniga, M. S., Holtz- 91–100. Lee, S. H., Kravitz, D. J., and and Houle, S. (1995). Distinct neural
man, J. D., and Kosslyn, S. M. (1985). Kastner, S., Pinsk, M. A., De Weerd, Baker, C. I. (2011). Disentangling correlates of visual long-term mem-
A left hemisphere basis for visual P., Desimone, R., and Ungerleider, visual imagery and perception of ory for spatial location and object
mental imagery? Neuropsychologia L. G. (1999). Increased activity in real-world objects. Neuroimage 59, identity: a positron emission tomog-
23, 115–118. human visual cortex during directed 4064–4073. raphy study in humans. Proc. Natl.
Farah, M. J., Hammond, K. M., Levine, attention in the absence of visual Levine, D. N., Warach, J., and Farah, M. Acad. Sci. U.S.A. 92, 3721–3725.
D. N., and Calvanio, R. (1988). stimulation. Neuron 22, 751–761. (1985). Two visual systems in mental Munk, M. H., Linden, D. E., Muckli,
Visual and spatial mental imagery: Kawashima, R., Roland, P. E., and imagery: dissociation of “what” and L., Lanfermann, H., Zanella, F. E.,
dissociable systems of representa- O’Sullivan, B. T. (1995). Functional “where” in imagery disorders due to Singer, W., and Goebel, R. (2002).
tion. Cogn. Psychol. 20, 439–462. anatomy of reaching and visuomo- bilateral posterior cerebral lesions. Distributed cortical systems in visual
Finke, R. A. (1989). Principles of Men- tor learning: a positron emission Neurology 35, 1010–1018. short-term memory revealed by
tal Imagery. Cambridge, MA: MIT tomography study. Cereb. Cortex 5, Martinez, A., Di Russo, F., Anllo-Vento, event-related functional magnetic
Press. 111–122. L., Sereno, M. I., Buxton, R. B., and resonance imaging. Cereb. Cortex 12,
Formisano, E., Linden, D. E., Di Salle, F., Keogh, R., and Pearson, J. (2011). Hillyard, S. A. (2001). Putting spa- 866–876.
Trojano, L., Esposito, F., Sack, A. T., Mental imagery and visual work- tial attention on the map: timing O’Craven, K. M., and Kanwisher, N.
Grossi, D., Zanella, F. E., and Goebel, ing memory. PLoS ONE 6, e29221. and localization of stimulus selec- (2000). Mental imagery of faces
R. (2002). Tracking the mind’s image doi:10.1371/journal.pone.0029221 tion processes in striate and extras- and places activates corresponding
in the brain I: time-resolved fMRI Knauff, M., Kassubek, J., Mulack, T., triate visual areas. Vision Res. 41, stimulus-specific brain regions. J.
during visuospatial mental imagery. and Greenlee, M. W. (2000). Corti- 1437–1457. Cogn. Neurosci. 12, 1013–1023.
Neuron 35, 185–194. cal activation evoked by visual men- Mechelli, A., Price, C. J., Friston, K. Paivio, A. (1971). Imagery and Verbal
Goebel, R., Khorram-Sefat, D., Muckli, tal imagery as measured by fMRI. J., and Ishai, A. (2004). Where Process. New York: Holt, Rinehart,
L., Hacker, H., and Singer, W. Neuroreport 11, 3957–3962. bottom-up meets top-down: neu- and Winston.
(1998). The constructive nature of Kosslyn, S. M. (1980). Image and Mind. ronal interactions during percep- Palmer, S. E. (1999). Vision Science: Pho-
vision: direct evidence from func- Cambridge, MA: Harvard University tion and imagery. Cereb. Cortex 14, tons to Phenomenology. Cambridge:
tional magnetic resonance imag- Press. 1256–1265. MIT Press.
ing studies of apparent motion and Kosslyn, S. M., Behrmann, M., and Mellet, E., Briscogne, S., Tzourio- Posner, M. I., and Gilbert, C. D. (1999).
motion imagery. Eur. J. Neurosci. 10, Jeannerod, M. (1995). The cogni- Mazoyer, N., Ghaem, O., Petit, Attention and primary visual cor-
1563–1573. tive neuroscience of mental imagery. L., Zago, L., Etard, O., Berthoz, tex. Proc. Natl. Acad. Sci. U.S.A. 96,
Haxby, J. V., Grady, C. L., Horwitz, Neuropsychologia 33, 1335–1344. A., Mazoyer, B., and Denis, M. 2585–2587.
B., Ungerleider, L. G., Mishkin, Kosslyn, S. M., Digirolamo, G. J., (2000). Neural correlates of topo- Prabhakaran, V., Narayanan, K., Zhao,
M., Carson, R. E., Herscovitch, P., Thompson, W. L., and Alpert, N. graphic mental exploration: the Z., and Gabrieli, J. D. (2000). Inte-
Schapiro, M. B., and Rapoport, S. M. (1998). Mental rotation of impact of route versus survey per- gration of diverse information in
I. (1991). Dissociation of object objects versus hands: neural mech- spective learning. Neuroimage 12, working memory within the frontal
and spatial visual processing path- anisms revealed by positron emis- 588–600. lobe. Nat. Neurosci. 3, 85–90.
ways in human extrastriate cortex. sion tomography. Psychophysiology Mellet, E., Petit, L., Mazoyer, B., Pylyshyn, Z. W. (1973). What the mind’s
Proc. Natl. Acad. Sci. U.S.A. 88, 35, 151–161. Denis, M., and Tzourio, N. eye tells the mind’s brain: a critique
1621–1625. Kosslyn, S. M., Ganis, G., and Thomp- (1998). Reopening the mental of mental imagery. Psychol. Bull. 80,
Haxby, J. V., Horwitz, B., Ungerleider, son, W. L. (2001). Neural founda- imagery debate: lessons from 1–24.
L. G., Maisog, J. M., Pietrini, P., and tions of imagery. Nat. Rev. Neurosci. functional anatomy. Neuroimage 8, Richter, W., Somorjai, R., Summers, R.,
Grady, C. L. (1994). The functional 2, 635–642. 129–139. Jarmasz, M., Menon, R. S., Gati, J.
organization of human extrastriate Kosslyn, S. M., and Ochsner, K. N. Mellet, E., Tzourio, N., Crivello, F., S., Georgopoulos, A. P., Tegeler, C.,
cortex: a PET-rCBF study of selec- (1994). In search of occipital activa- Joliot, M., Denis, M., and Mazoyer, B. Ugurbil, K., and Kim, S. G. (2000).
tive attention to faces and locations. tion during visual mental imagery. (1996). Functional anatomy of spa- Motor area activity during mental
J. Neurosci. 14, 6336–6353. Trends Neurosci. 17, 290–292. tial mental imagery generated from rotation studied by time-resolved
Ishai, A., Haxby, J. V., and Ungerleider, Kosslyn, S. M., Pascual-Leone, A., Feli- verbal instructions. J. Neurosci. 16, single-trial fMRI. J. Cogn. Neurosci.
L. G. (2002). Visual imagery of cian, O., Camposano, S., Keenan, J. P., 6504–6512. 12, 310–320.

Richter, W., Ugurbil, K., Georgopoulos, Sack, A. T., Sperling, J. M., Prvulovic, D., representations in human visual cor- memory: positron emission
A., and Kim, S. G. (1997). Time- Formisano, E., Goebel, R., Di Salle, F., tex. Neuroimage 56, 1540–1545. tomography findings. Proc. Natl.
resolved fMRI of mental rotation. Dierks, T., and Linden, D. E. (2002). Tagaris, G. A., Kim, S. G., Strupp, J. Acad. Sci. U.S.A. 91, 2016–2020.
Neuroreport 8, 3697–3702. Tracking the mind’s image in the P., Andersen, P., Ugurbil, K., and Yantis, S., Schwarzbach, J., Serences, J.
Riesenhuber, M., and Poggio, T. (2000). brain II: transcranial magnetic stim- Georgopoulos, A. P. (1997). Mental T., Carlson, R. L., Steinmetz, M. A.,
Models of object recognition. Nat. ulation reveals parietal asymmetry rotation studied by functional mag- Pekar, J. J., and Courtney, S. M.
Neurosci. 3(Suppl.), 1199–1204. in visuospatial imagery. Neuron 35, netic resonance imaging at high field (2002). Transient neural activity in
Roland, P. E., and Gulyas, B. (1995). 195–204. (4 Tesla): performance and corti- human parietal cortex during spa-
Visual memory, visual imagery, and Sengpiel, F., and Hubener, M. (1999). cal activation. J. Cogn. Neurosci. 9, tial attention shifts. Nat. Neurosci. 5,
visual recognition of large field pat- Visual attention: spotlight on the 419–432. 995–1002.
terns by the human brain: func- primary visual cortex. Curr. Biol. 9, Thompson, W. L., Slotnick, S. D.,
tional anatomy by positron emission R318–R321. Burrage, M. S., and Kosslyn, S. Conflict of Interest Statement: The
tomography. Cereb. Cortex 5, 79–93. Seurinck, R., De Lange, F. P., Achten, E., M. (2009). Two forms of spatial authors declare that the research was
Sack, A. T. (2009). Parietal cortex and and Vingerhoets, G. (2011). Mental imagery: neuroimaging evidence. conducted in the absence of any com-
spatial cognition. Behav. Brain Res. rotation meets the motion afteref- Psychol. Sci. 20, 1245–1253. mercial or financial relationships that
202, 153–161. fect: the role of hV5/MT+ in visual Trojano, L., and Grossi, D. (1994). A could be construed as a potential con-
Sack, A. T. (2010). Using non-invasive mental imagery. J. Cogn. Neurosci. critical review of mental imagery flict of interest.
brain interference as a tool for mim- 23, 1395–1404. defects. Brain Cogn. 24, 213–243.
icking spatial neglect in healthy vol- Shepard, R. N., and Metzler, J. Trojano, L., Grossi, D., Linden, D. E., Received: 27 April 2012; accepted: 08 June
unteers. Restor. Neurol. Neurosci. 28, (1971). Mental rotation of three- Formisano, E., Goebel, R., Cirillo, 2012; published online: 28 June 2012.
485–497. dimensional objects. Science 171, S., Elefante, R., and Di Salle, F. Citation: Sack AT and Schuhmann
Sack, A. T., Camprodon, J. A., Pascual- 701–703. (2002). Coordinate and categorical T (2012) Hemispheric differences
Leone, A., and Goebel, R. (2005). Slotnick, S. D., Thompson, W. L., judgements in spatial imagery. An within the fronto-parietal net-
The dynamics of interhemispheric and Kosslyn, S. M. (2005). Visual fMRI study. Neuropsychologia 40, work dynamics underlying spatial
compensatory processes in mental mental imagery induces retino- 1666–1674. imagery. Front. Psychology 3:214. doi:
imagery. Science 308, 702–704. topically organized activation of Trojano, L., Grossi, D., Linden, D. E., 10.3389/fpsyg.2012.00214
Sack, A. T., Jacobs, C., De Martino, early visual areas. Cereb. Cortex 15, Formisano, E., Hacker, H., Zanella, This article was submitted to Frontiers in
F., Staeren, N., Goebel, R., and 1570–1583. F. E., Goebel, R., and Di Salle, Perception Science, a specialty of Frontiers
Formisano, E. (2008). Dynamic Smith, E. E., Jonides, J., Koeppe, R. F. (2000). Matching two imagined in Psychology.
premotor-to-parietal interactions A., Awh, E., Schuhmacher, E. H., clocks: the functional anatomy of Copyright © 2012 Sack and Schuhmann.
during spatial imagery. J. Neurosci. and Minoshima, S. (1995). Spatial spatial analysis in the absence of This is an open-access article distributed
28, 8417–8429. vs. objects working memory: PET visual stimulation. Cereb. Cortex 10, under the terms of the Creative Commons
Sack, A. T., Lindner, M., and Linden, investigations. J. Cogn. Neurosci. 7, 473–481. Attribution Non Commercial License,
D. E. (2007). Object- and direction- 337–356. Tulving, E., Kapur, S., Craik, F. I., which permits non-commercial use, dis-
specific interference between man- Stokes, M., Saraiva, A., Rohenkohl, G., Moscovitch, M., and Houle, S. tribution, and reproduction in other
ual and mental rotation. Percept. and Nobre, A. C. (2011). Imagery for (1994). Hemispheric encod- forums, provided the original authors and
Psychophys. 69, 1435–1449. shapes activates position-invariant ing/retrieval asymmetry in episodic source are credited.

ORIGINAL RESEARCH ARTICLE
published: 15 August 2012
doi: 10.3389/fpsyg.2012.00296
Unmasking the Perky effect: spatial extent of image

interference on visual acuity
Adam Reeves 1 * and Catherine Craver-Lemley 2
1
Department of Psychology, Northeastern University, Boston, MA, USA
2
Department of Psychology, Elizabethtown College, Elizabethtown, PA, USA
Edited by: We have previously argued that visual mental images are not substitutable for visual per-
cepts, because the interfering effects of visual stimuli such as line maskers on visual targets
differ markedly in their properties from the interfering effects of visual images (the “Perky
Reviewed by:
John Cass, University of Western effect”). Imagery interference occurs over a much wider temporal and spatial extent than
Sydney, Australia masking, and unlike masking, image interference is insensitive to relative orientation. The
Adriane E. Seiffert, Vanderbilt lack of substitutability is theoretically interesting because the Perky effect can be compared
University, USA
meaningfully to real line masking in that both types of interference are visual, not due to
*Correspondence:
optical factors (accommodative blur or poor fixation) or to high-level factors (attentional
Adam Reeves, Department of
Psychology, Northeastern University, distraction, demand characteristics, or effects of uncertainty). In this report, however, we
360 Huntingdon Avenue, Boston, MA question our earlier position that spatial extents of interference are markedly different:
02115, USA. when images and real lines are matched in contrast, which was not done previously, their
e-mail: a.reeves@neu.edu
interference effects have very similar spatial extents. These data add weight to the view
that spatial properties of images and percepts are similar in respect to extent. Along with
the wider temporal extent and the insensitivity to orientation, the new results remain com-
patible with our older hypothesis that to create a clear mental image in a region of visual
space, incoming signals from the eye must be suppressed (Craver-Lemley and Reeves,
1992). We have pursued this idea in this report using “unmasking,” in which adding ele-
ments to the visual image in the region beyond the zone of suppression reduces the Perky
effect.
Keywords: visual imagery, Perky effect, spatial vision, unmasking
INTRODUCTION location of the visual target. Thus when the target was flashed,
Whether or not visual mental images act like visual percepts is a it was presumably presented at the same location in visual space
fascinating and important question in cognitive science. Should that the image already occupied (the image-on-target, or “ON”
they have the same functions and share the same anatomy, as condition). Such images always interfere with perception of the
has been argued from imaging studies of cortical area V1, one target by lowering sensitivity, not by making the response crite-
might expect them to be mutually substitutable or at least, anal- rion sub-optimal. Interference occurs in the brain, not the eye,
ogous to each other. That images and percepts both have spatial as controls have ruled out optical effects such as poorer accom-
layout, both contain rich visual information, and both share the modation of the lens during imagery or poorer fixation during
functional property of reduced acuity in the periphery (Kosslyn, imagery (Craver-Lemley and Reeves, 1987). Moreover, the Perky
1980), may be taken to argue in favor of substitutability; in Koss- effect does not stem from demand characteristics; participants told
lyn’s array theory, for example, images and percepts are thought that imagery aided perception produced as great a Perky effect as
to share the same representational array, and in Finke’s (1980) those told it would hinder perception (Craver-Lemley and Reeves,
theory, images and percepts are functionally equivalent for high 1987, Experiments 1a and 1b). Other experiments have ruled out
imagers. In our earlier work, we argued in favor of analogous spa- mere distraction, the notion that having any image would add a
tial behavior but against substitutability (Arterberry et al., 2002), load to the cognitive system and thus lower visual performance, as
based on empirical findings concerning the so-called Perky effect, auditory images have virtually no effect on visual target (Craver-
in which visual images, analogous to real masking stimuli, can Lemley and Reeves, 1992; although they do on auditory ones: Segal
suppress perception of real visual targets (as discovered by Perky, and Fusella, 1970), visual images located far away from the visual
1910). The Perky effect is ubiquitous in studies of percept-image target have almost no effect (Craver-Lemley and Reeves, 1987;
interactions, with the imagined stimulus depressing sensitivity to Craver-Lemley and Arterberry, 2001), and diverting attention to a
briefly flashed visual targets by 15% or better (0.8 d0 units or more; distracting light does not increase the interference (Craver-Lemley
Segal and Fusella, 1970; Craver-Lemley and Reeves, 1987, 1992; and Reeves, 1992). Thus we have used the Perky effect as a test-
Ishai and Sagi, 1997). In the Perky experiments, the experimenter bed for discovering whether images and real visual stimuli have
requested that the subject project his or her visual image in the the same functional properties. We adopted an acuity target as the
www.frontiersin.org August 2012 | Volume 3 | Article 296 | 16

Reeves and Craver-Lemley Perky effect
visual stimulus, since the visual properties of acuity stimuli are has been terminated, long after any masking by real lines is com-
well documented1 . plete. However, we were concerned to repeat our work on the
Experiments using real (rather than imagined) stimuli have lateral extent of interference. We therefore compared the effects of
revealed several well-known characteristics of masking of line tar- images of lines against not only the real black lines used before,
gets, including that masking is reduced rapidly as the masking but also against real gray lines matched in contrast with the par-
stimuli are moved away laterally from the target; that masking is ticipant’s own imagined lines (Experiment 1). We continued to
reduced if the orientation of the mask differs from that of the tar- use lines rather than other acuity targets for continuity with our
get, disappearing if they are at right-angles to each other (although previous work; however, we note here that the spatial structure of
this general rule may not be true above threshold for dichop- the stimulus may affect the lateral extent of interference. While
tic low-spatial frequency gratings: Meese and Hess, 2004); that Ishai and Sagi (1997), like us, found considerable interference
masking is eliminated if the target-to-mask temporal asynchrony by imagery in the “ON” condition when using Gabor (wavelet)
is more than 200 ms; and that patterned masking is cortical – targets, they also found a small but significant enhancement in
unlike noise masks, patterned masks are as effective when the sensitivity when the mental image was displaced just one wave-
target is presented to the opposite as to the same eye (see, e.g., length away from the target center. With lines, interference is
Westheimer, 1965; Westheimer and Hauske, 1975). Relying on also reduced by increasing target-image distance, but we have not
this literature, we argued that interference due to images (the found enhancement.
Perky effect) had quite different characteristics from real mask-
ing (Craver-Lemley and Reeves, 1992), in that the Perky effect is EXPERIMENT 1
independent of the relative orientation of the image and the real This experiment was conducted to find out whether gray lines
lines, and the Perky effect persists up to 6 s after the image has that were matched in contrast to images of lines would interfere
been removed (Craver-Lemley and Reeves, 1987). Moreover, the with acuity in the same manner as the mental images. Black real
Perky effect apparently covers a wider region of visual space than lines were also used to replicate previous work. Target-image and
does the interfering effects of real lines (Craver-Lemley and Reeves, target-real line distances were varied to permit comparison of the
1992). lateral extents of image interference and real line masking.
In this past research, we asked participants to create visual
images but we have not matched the contrasts of the mental images MATERIALS AND METHODS
to the contrast of the physical stimuli. Typically, we employed PARTICIPANTS
high contrast physical stimuli, whereas participants created men- Eleven undergraduate volunteers enrolled in an introductory psy-
tal images ad lib, and our participants had informally reported chology course participated in this study for payment. All par-
that their imagined lines were lighter and less distinct than those ticipants had normal or corrected-to-normal vision and had no
pictured (Craver-Lemley and Reeves, 1987, 1992). We therefore previous experience in vision or imagery experiments.
wondered if our complete rejection of substitutability was too
hasty, the difference being, not between real and imaged stimuli, APPARATUS AND STIMULI
but between high and low contrasts. We did not repeat our ear- Experiments were conducted with Model T-2B-1 two-field Ger-
lier work on image orientation because it is already established brands tachistoscopes. The fixation and test fields were 58 cm from
that masking by low-contrast gratings, like masking by high con- the eyes, superimposed by a half-silvered mirror, and surrounded
trast ones, disappears at the orthogonal orientation, implying that by complete darkness (the inner walls of the apparatus were cov-
our earlier result (orthogonal images interfered as much as par- ered in black velvet). The fixation field was a white 17 cd/m2
allel ones) would differ from real line masking no matter what rectangular index card that subtended 10˚ × 15˚ at the eye. Cen-
real line contrast was used. We also did not repeat the work com- tral fixation was aided by two small (1.22 mm) black dots placed
paring timing, as the Perky effect lasts for 4–6 s after an image 5.2˚ apart, symmetrically above and below the center of this field.
Small guide dots (Experiments 1 and 2) or guide lines (Experiment
3) were added at top and bottom of the fixation field to aid the
1 Recent research has re-emphasized Hume’s view that the vividness of mental
subjects in locating their images when images were requested.
imagery is functional. David Hume had argued that images were less vivid than the
There were two real line conditions, in which imaginary lines
reality they represented, and that such a difference was sufficient to distinguish truth
from fantasy. Perky (1910), however, showed the participants do not always reliably were not requested, namely, Black Lines and Matched Gray Lines.
distinguish images from percepts, and perhaps more tellingly, reported image vivid- In these conditions, 1.0 cm wide and 7.6 cm high black or gray
ness does not in general correlate with perceptual ability though it may with memory lines, subtending 0.1 × 7.5˚, were added to the fixation field. Being
(Hilgard, 1981). Indeed, we (Craver-Lemley and Reeves, 1987) found no correlation part of the fixation field, these lines were presented continuously,
between the self-reported vividness of visual imagery and the extent of the Perky
effect across 125 participants. However, Rodway et al. (2006) recently reported that
unlike the vast majority of studies of masking in which masking
while self-reported high and low vividness participants did not differ in overall lev- lines are flashed; however, continuous presentation was necessary
els of change detection accuracy, high vividness participants were more accurate to mimic the imaginary lines, which were imagined throughout
at detecting salient changes to pictures. Moreover, self-reports of image vividness each trial, as closely as possible.
correlate r = 0.73 with image strength as documented by the fMRI (Cui et al., 2007), The test field was used to flash the vernier acuity targets, as
and stronger visual images are more likely to survive size-scaling than are weaker
ones (D’Angiulli and Reeves, 2002). The strength of simple line images, as manip-
in Craver-Lemley and Reeves (1992); Craver-Lemley et al. (1997).
ulated by the participants themselves, therefore seemed like a possible factor in our The test field was dimmed to 6 cd/m2 so that flash duration at
experiments. threshold would not be lower than 10 ms for the best participants.
Frontiers in Psychology | Perception Science August 2012 | Volume 3 | Article 296 | 17

Each target was made of two, thin black (75% contrast) lines, each
line subtending 2.2˚ vertically and 0.1˚ horizontally, with a vertical
gap of 0.5˚ between them. As the vertical gap between the acu-
ity targets is relatively large, hyperacuity was not achieved. The
targets (black tape by Chart-Pak) were mounted on white index
cards. The lower of the two lines in each target was, with equal
probability, offset to the left or right of the upper line. There were
10 different offsets ranging from 4.2 to 22 min visual angle, with a
mean offset of 9.7 min. The target presentation was randomized.
The participant’s task was to report whether the lower line was to
the right or left of the upper line; they were told that each was
equally likely to occur. The test field was jittered randomly left or
right by up to 17 arc min from trial to trial, to encourage the par-
ticipant to judge the lines relative to each other rather than relative
to the fixation point.
Conditions
There were 10 conditions, run in separate blocks: FIGURE 1 | Two-line acuity targets surrounded by imaginary lines, ON,
CLOSE, or FAR away, as directed by small guide spots or lines at top
Imaged lines. During imagery trials, participants were asked to and bottom of the fixation field (not shown). Real black or gray lines
were positioned in the same places as the imagined ones.
evoke a mental image of four black vertical lines. Participants were
shown a picture of the lines they were to imagine and asked to
locate their image in reference to small guide marks penciled in at
PROCEDURE
the top and bottom of the fixation field. Each participant’s acuity
was measured while they were instructed to project mentally ver- Participants were individually tested during two 1 h sessions. The
tical line images either “ON” the acuity target (nearest line 0.1 cm, duration of target presentation was determined during an addi-
or 6 min arc, away from the target), “CLOSE” to the target (near- tional 15 min practice session. Participants were told to fixate
est line 0.8 cm, or 0.8˚, away), or “FAR” from it (innermost line between two fixation points where the target would be presented.
2.0 cm, or 2.0˚, away; see Figure 1). Guide marks indicated to the The target duration was adjusted until accuracy in the baseline no
participants where to place their mental images. They were given imagery acuity task was close to 90%. Target duration remained
ample time to comply, and none reported difficulty in locating fixed after the practice session; it averaged a nominal 8 ms (range:
their imaginary lines with respect to the guide marks. They were 3–25 ms; as these values are well below the critical duration, tar-
instructed to hold their images throughout the trial; if reported get duration exerts its effect via energy). Participants reported the
losing the image, the trial was re-run. direction of offset (left, right) on each trial.
Conditions were run in separate blocks of 25 trials in each of
two sessions. The order of the blocks was randomized over partic-
Black lines. During black line trials, the four real, thin black lines ipants. There were 25 trials per participant in the black lines ON
were shown continuously from top to bottom of the fixation field. condition, which, being near chance in the first session was not
These lines appeared either “ON” the acuity target, “CLOSE” to it, re-run, however 50 trials were included for each of the other nine
or “FAR” from it, with the same spacing as for the imaged lines. conditions.
Instructions were reviewed before each block. In the No Image
Matched gray lines. During the initial practice session, we asked and the two real line conditions, the experimenter initiated each
each participant to select lines from a prepared set (ranging in color trial by saying,“go” and then the acuity target was presented. In the
from light gray to black) which best matched his or her imagined imagery condition, participants indicated their image was ready by
lines. All participants selected real gray lines that were lighter than saying “go,” and the experimenter then presented the acuity target.
the black lines, matching images to Munsell color chips between Participants were requested to re-create or re-evoke their images
10B 8/1 and N8/5. As in the Imagery and Black Lines conditions, on each of the 25 imagery trials during a block. Participants were
the Matched Gray Lines (at the level selected by each participant) allowed to rest between blocks. They did not receive any feedback
were presented in the same ON, CLOSE, and FAR conditions. until after the second session had been completed.
Our experimenters were undergraduate research assistants who
No imagery. In the baseline condition, accuracy was measured were naïve as to the predicted outcomes of the experiments and
for reporting the acuity target, without either real or imagined had never before collected data in imagery experiments. They
lines. could not see the acuity target until after each trial had been
Thus the 10 conditions comprised the baseline with no lines and completed, and so were unaware of what the correct response
no imagery, three conditions with imagined lines (ON, CLOSE, should be prior to stimulus presentation and the participant’s
FAR), three with Black Lines (ON, CLOSE, FAR), and three with response. Individual data were not tabulated or analyzed until
Matched Gray Lines (again, ON, CLOSE, FAR). after the experiment was completed.

RESULTS AND DISCUSSION

Accuracy in the baseline (No Image) condition averaged 91.4%
correct across participants, reflecting successful choices of target
durations during initial practice. Accuracy in all other conditions
was lower than this, reflecting the interfering effects of real and
of imagined lines. An omnibus analysis of variance (ANOVA)
revealed a main effect of both distance [F (2,9) = 191.54, p < 0.001,
η2p = 0.95] and line type [F (2,9) = 23.03, p < 0.001, η2p = 0.70], as
well as an interaction [F (4,7) = 45.66, p < 0.001, η2p = 0.82]. Mean
percent correct in each of the conditions is plotted in Figure 22 .
As expected, the results show a clear Perky effect, as imagined lines
reduced acuity. The reduction is 17.6% when the imagined lines
are “ON” the target, in line with Segal and Fusella (1970), Reeves
(1980), and Craver-Lemley and Reeves (1987, 1992). The Perky
effect was attenuated if the image was shifted off target, in the
present research from 17.6 to 6.3% (i.e., baseline No Image – image
OFF), also in line with Craver-Lemley and Reeves (1987).
The critical new result is that (real) Matched Gray lines have FIGURE 2 | Mean correct performance for the acuity task with imaged
rather similar effects on acuity as imagined lines, reducing perfor- lines, with real gray lines matched to the participant’s images, with
mance by 13.2% in the ON condition and 5% in the FAR condition, real black lines, and in the baseline no image, no mask (“none”)
condition. The interfering lines are ON the acuity target, CLOSE to it, or
compared to 17.6–6.3%, respectively. Given the difficulties inher-
FAR from it.
ent in exactly matching the contrast and spatial position of the
real lines, one is struck by the congruence between these effects
and those for imagined lines. This result causes us to re-think
our original claim that the spatial extent of interference by images
exceeds that of real lines; contrast, not reality, appears to be the
critical variable, and when contrasts are matched, the reductions
in interference with distance from the target are similar.
EXPERIMENT 2
Experiment 1 suggests that contrast plays a major role in the Perky
effect, but does contrast polarity also matter? We had not compared
images with positive and negative contrasts in previous studies of
the Perky effect, nor have we located a study that has. We there-
fore asked participants to create images of white vertical lines for
comparison with the images of black vertical lines used before,
projecting both onto a gray field. With real stimuli, acuity depends
strongly on the absolute value of the contrast, but when targets
2A detailed analysis of the data of Experiment 1 reveals the effects expected from
eyeballing Figure 2. In the ON conditions, black lines showed significantly more
interference than both matched lines [t (10) = 7.41, p < 0.001], and imagined lines FIGURE 3 | Mean correct performance for the acuity task with ON and
[t (10) = 5.71, p < 0.001], and imagined lines showed significantly more interference FAR images, each of four lines, with ALL eight imagined lines, and in
than matched lines [t (10) = −6.71, p < 0.001]. In the CLOSE conditions, matched the baseline no image condition.
lines showed significantly more interference than black lines [t (10) = −2.86,
p = 0.017], imagined lines showed significantly more interference than black lines
[t (10) = −3.32, p = 0.008], and matched lines and imagined lines were not signif-
icantly different, t (10) = 1.44, p = 0.181. In the FAR conditions, black lines and are presented on a constant background, acuity is only weakly
matched lines were not significantly different [t (10) = −1.35, p = 0.208], but imag- dependent on contrast polarity (Pointer, 2001), as is metacontrast
ined lines showed significantly more interference than black lines [t (10) = −2.71,
p = 0.022] and matched lines [t (10) = −2.28, p = 0.046]. Images ON the target
(Breitmeyer et al., 2008: note the same polarity masks and tar-
showed more interference than both images CLOSE to it [t (10) = 4.58, p = 0.001] gets in his Figure 3) and object substitution masking (Backmann
and FAR from it [t (10) = 5.14, p < 0.001]; also, CLOSE images showed more inter- and Luiga, 2008). (We note that earlier studies of polarity com-
ference than FAR images [t (10) = 4.50, p = 0.001]. Matched Gray Lines ON and pared acuity for white targets on a black background with acuity
CLOSE to the target did not differ [t (10) = 1.36, p = 0.204], but they interfered for black targets on a white background, but these backgrounds
more than those FAR from it [t (10) = 3.9, p = 0.003, and t (10) = 4.54, p = 0.001].
Real Black Lines ON the target interfered more than both those CLOSE to it
alter adaptation level, light scatter, and pupil size; a constant back-
[t (10) = 11.73, p < 0.001] and FAR from it [t (10) = 14.66, p < 0.001], and those ground controls these factors.) If the analogy between imagery and
CLOSE interfered more than those FAR [t (10) = 7.61, p < 0.001] real lines holds true of contrast, we therefore expect that contrast

polarity of the image would leave the magnitude of the Perky effect we have not studied the effects of contrast polarity with real lines
relatively unchanged, given that we use a constant lit background. in our equipment and our target-line distances.
MATERIALS AND METHODS EXPERIMENT 3

PARTICIPANTS If the analogy between real and imagined lines does hold up, at
Twenty-two undergraduates with normal or corrected-to-normal least for extent if not for orientation, then it is possible that imag-
vision volunteered to participate in exchange for course credit ined lines, like real ones, might show “unmasking” (also termed
in an introductory psychology course. None of the students had “release from masking.”) In this phenomenon, a real mask super-
previously participated in an imagery experiment before. imposed on the target has its masking effect reduced by adding a
MATERIALS AND PROCEDURE distant mask, even when the latter had little direct effect on tar-
The stimuli and apparatus were as in Experiment 1. Participants get visibility (Haber, 1970). It is as if the distant mask releases
were again tested individually in single 45 min. sessions. The acuity the target from masking, either by inhibiting the superimposed
targets were presented for a duration that resulted in close to 90% mask (Dember and Purcell, 1967), or by strengthening the tar-
correct performance without imagery, chosen for each participant get (Briscoe et al., 1983). Herzog (2007) reviews unmasking in a
during an initial 15 min practice session. The mean nominal target more complex spatial arrangement when the vernier is masked by
duration was 9.3 ms (range = 3–30 ms). The task was, as before, to a spatial grating. We wondered whether the same phenomenon of
report whether the lower line was offset to the left or right of the unmasking could be found with visual images.
upper line.
There were three conditions: No Image (the baseline), Imagine MATERIALS AND METHODS
Black Lines, and Imagine White Lines, ON the target. The three PARTICIPANTS
conditions were not blocked but were randomly intermixed. There Sixteen undergraduates with normal or corrected-to-normal
were 50 trials per condition for a total of 150 trials for each partic- vision volunteered to participate in exchange for course credit
ipant. Participants were cued prior to the beginning of each trial in an introductory psychology course. None of the students had
as to whether they were to imagine four vertical white lines, four previously participated in an imagery experiment before.
vertical black lines, or to have no image at all. During imagery
trials, participants indicated when they were ready with the image, MATERIALS
and then the experimenter presented the target. Participants were A Gebrands two-field tachistoscope was used, of the same type as
instructed to maintain the image for the duration of the trial. On the one used in Experiments 1 and 2. Stimuli were the same as
No Image trials, the experimenter indicated it was a no image before. On imagery trials, participants were asked to evoke four
trial and then said, “go” to alert the participant that the trial was or eight black vertical line images. Small guide lines in the fixa-
beginning. Participants received a 5 min break after 75 trials. tion field helped the participants locate their images. Participants
imagined four vertical lines in the ON and FAR conditions illus-
RESULTS AND DISCUSSION trated in Figure 1, but in the new condition, ALL-8, they were
Correct performance, averaged over targets and participants was requested to generate an image of eight black lines, as indicated by
89.1% with no image, 80.8% for Imagine Black Lines, and the guide lines. Ideally this new image is the sum of ON and FAR
79.0% for Imagine White Lines. A significant condition effect, images.
F (2,42) = 23.71, p < 0.001, was found in a one-way repeated mea-
sures ANOVA, with both Perky effects significant by Tukey’s hon- PROCEDURE
estly significant difference test (p < 0.01). However, accuracy with As before, participants were tested individually in a single session
Black line images (80.8%) did not differ from that with White line lasting approximately 50 min. The acuity targets were presented
images (79.0%). for a duration that resulted in close to 90% correct performance
The Perky effect for Imagine Black Lines, being 8.3%, was in the baseline (no imagery) condition, which was chosen for each
smaller than the 17.6% effect found for ON images in Experiment participant during an initial 15 min practice session and thereafter
1. Since otherwise these conditions were the same, this difference fixed. The mean duration of targets was 13 ms (range = 4–33 ms).
in magnitude may be due to randomizing rather than blocking All three imagery conditions were practiced until the participant
the conditions. Randomizing may reduce magnitude because the was familiar with the procedure. Participants were asked to main-
Perky effect lasts several seconds after the participant reports turn- tain central fixation during the experiment. The task was, as in
ing off the image (Craver-Lemley and Reeves, 1987), and therefore Experiments 1 and 2, to report whether the lower line was offset
some image-generated interference may leak into those baseline to the left or right of the upper line. The three imagery and one no
trials which directly follow imagery trials. However, randomiz- imagery conditions were randomly intermixed with 50 trials per
ing has the advantage that the lack of a difference between black condition for each participant. Participants were cued prior to the
images and white images is unlikely to be due to a change in strat- beginning of each trial as to what to image, or to have no image at
egy between conditions. We conclude that the lack of a difference all. During imagery trials, participants indicated when they were
is genuine, and therefore that contrast, but not contrast polarity, ready with the image and then the experimenter presented the
influences the Perky effect. This finding for imagery appears to target. Participants were instructed to maintain the image for the
agree with the literature concerning the effects of contrast polarity duration of the trial. On baseline trials, the experimenter indicated
on acuity for real stimuli, but this agreement is not definitive as it was a no image trial and then said, “go” to alert the participant

that the trial was beginning. Participants received 5 min breaks to the area in which the target will be presented, as opposed to
after each block of 50 trials. FAR images which draw attention away, so distraction would pre-
dict more, not less, of a Perky effect in FAR than in ON. Moreover,
RESULTS Craver-Lemley and Reeves (1992) explicitly manipulated attention
The differences between the four conditions were highly signifi- by having participants attend to a blinking light in the periphery;
cant [F (3,45) = 9.48, p < 0.01] by a repeated measures, one-way this lowered accuracy overall but left the Perky effect unaltered.
ANOVA applied to the scores of the 16 participants. Mean accu- Again, Craver-Lemley and Reeves (1987) found that a vertical line
racy was 89.4% in the baseline “No imagery” condition, reflecting image produced a Perky effect for 4–6 s after the subject indicated
successful manipulation of the individual target durations. Accu- that he or she had removed the image, and in this situation there
racy with ON images was 75.3%, giving a reduction in accuracy, was no image to distract attention. Finally, Craver-Lemley et al.
or Perky effect, of 14.2% (Figure 3). When the lines were imaged (1997) found that having vertical line images in front of the visual
far from the target, accuracy was 83.6%, for a much smaller Perky target produced a Perky effect, but having them behind the target
effect of 5.8%. These two Perky effect magnitudes are similar to had no interfering effect at all. Since to control location in depth,
those reported before by Craver-Lemley and Reeves (1987) for the same image was requested on one or other face of an outlined
these two conditions. They suggested to us that the Perky effect cube, it is unlikely that differential attention to the image could
is spatially localized. The new condition is ALL-8, in which both explain this result. We regard all this as good evidence that the
ON and FAR lines were imaged; in this case, mean accuracy was Perky effect is not entirely due to distraction of attention.
81.6%, for a Perky effect of 7.8%. By“suppression,”we invoke a hypothetical mechanism in which
The Perky effect in ALL-8 would be greater than in ON if the the inner four lines in ALL-8 have their interfering effects on
effect of the inner lines summed with that of the outer lines, the target reduced by the outer four lines. Although it is some-
because the inner lines were the same in ON as in ALL-8, and what counterintuitive that adding masking elements could reduce
the outer lines had a weak (7.6%) effect of their own. However, masking, this effect is known in the masking of visual targets by
the results showed a clear diminution of the Perky effect in ALL-8, real maskers (Dember and Purcell, 1967; Herzog, 2007), and the
to a mere 7.8%, which is much less than additive. same idea has been applied before to the unmasking of real stimuli
by visual images (Reeves, 1980). Either the outer lines “unmask”
DISCUSSION the target by suppressing the interfering effect of the inner lines,
There are three possible explanations for our new result, that the or the outer lines act to facilitate the target. Facilitation by dis-
Perky effect with eight imaged lines is less than additive: suppres- tant imagery is theoretically possible and has been observed using
sion, distraction, and weakening. We will ultimately argue for an Gabor-like stimuli and images by Ishai and Sagi (1997); however,
explanation in terms of suppression, but the other two possibilities their effect, thought theoretically important, was relatively small.
merit consideration, especially as the result from Experiment 3 is We are inclined to conclude that unmasking occurs when the active
singular and we therefore do not have an adequate sampling of part of the mask is itself masked due to some form of inhibition
experimental conditions to permit a definitive conclusion. by additional mask elements, both for real lines and for imaged
By “weakening” we mean that relative to the strength of the ones. However, the idea that the far away image elements inhibit
ON image, that of the ALL-8 image is reduced, either by graying, the ones close to or on the acuity target is based on this single
blurring, making less vivid, decreasing image duration, or other- result, and this may not survive further testing; and even if it does,
wise lowering image quality. The ALL-8 image could in principle the properties of image unmasking may turn out quite different
be weaker because maintaining eight vertical line images is sim- from those of real line unmasking.
ply more taxing than maintaining four. However, both in initial
practice and in debriefing, we found no evidence for this: all of GENERAL DISCUSSION
the participants reported being able to imagine eight lines as well Craver-Lemley and Reeves (1992) suggested that the Perky effect
as four. In debriefing, we again asked whether the three images is a consequence, not of the image directly, but of a mechanism
(ON, FAR, and ALL-8) had equal vividness, and had kept their designed to protect the image by inhibiting visual input from the
vividness throughout the experiment; participants reported that region of the visual field in which the subject was requested to
they had. Such retrospective reports do not disprove the possibility have the image. Since we found the Perky effect was insensitive
of some weakening, but as explicitly weakening the ON image by to the relative orientation of imaged lines and real line acuity tar-
asking participants to gray-out their imagery reduced the Perky gets, we suggested that the protection was provided by inhibitory
effect by only 4%, as we found in a pilot experiment, we think the cortical feedback to the LGN, the last relay in the visual path-
consequences of any slight weakening would be marginal. way which is not sensitive to orientation. Our measurements
By“distraction”we mean that having a vertical line image might suggested that the feedback effect was weak, amounting to a reduc-
distract attention from the primary task of reporting the acuity tar- tion in stimulus contrast of 0.24 log units, but this was enough
get. We originally argued that images required the same amount to account for the reduction in acuity that we had reported. As
of attention, and were equally distracting, whether they were near already remarked, we have rejected other explanations of interfer-
the acuity target or far from it, so differences between them in the ence, such as poorer optical accommodation, shifts of gaze away
magnitude of the Perky effect could not be explained by attention from the target, confusion of the target with the image, distraction
(Craver-Lemley and Reeves, 1987, 1992). One might dispute this of attention, response bias, and experimenter effects, all based on
logic, but note that ON images demand that attention be paid empirical data.

However, the notion of a protective mechanism operating at of real and imagined features that makes the real features more
the level of the LGN did not predict the finding of Craver-Lemley difficult to extract. This notion is attractive in many ways, but
et al. (1997) that having vertical line images behind the visual it presupposes that imagined features have much more featural
target eliminates the Perky effect, because LGN neurons are insen- spread than real ones, since, unlike real lines, horizontal line
sitive to depth as well as to orientation. The model could be images interfere as much with acuity for vertical lines as do verti-
modified such that the feedback signal itself is dependent on cal line images. It also supposes that some unknown after-effects
depth, such that only frontal images need protection from visual of imaged lines, not just concurrent images, combine with stim-
input, although this is quite ad hoc. However, the new result, of ulus features, as images continue to interfere for 4–6 s after being
unmasking, cannot be explained in such a manner. An alternative removed (Craver-Lemley and Reeves, 1987). It may be that some
hypothesis suggested by Perky (1910) and by Segal (1971), and combination of the two hypotheses, or perhaps an altogether dif-
resurrected by Craver-Lemley et al. (1997) in order to explain the ferent one, will eventually explain the intricacies of the Perky
effect of depth, is that the Perky effect results from a combination effect.
REFERENCES reduces vernier acuity. Perception 16, Ishai, A., and Sagi, D. (1997). Visual Westheimer, G. (1965). Visual acuity.
Arterberry, M. E., Craver-Lemley, C., 599–614. imagery facilitates visual perception: Annu. Rev. Psychol. 16, 359.
and Reeves, A. (2002). Visual Craver-Lemley, C., and Reeves, A. psychophysical evidence. J. Cogn. Westheimer, G., and Hauske, G. (1975).
imagery is not always like visual (1992). How visual imagery inter- Neurosci. 9, 476–489. Temporal and spatial interference
perception. Behav. Brain Sci. 25, feres with vision. Psychol. Rev. 99, Kosslyn, S. M. (1980). Image and Mind. with vernier acuity. Vision Res. 15,
183–184. 633–649. Cambridge, MA: Harvard University 1137–1141.
Backmann, T., and Luiga, I. (2008). Cui, X., Jeter, C. B., Yang, D., Mon- Press.
Luminance processing in object sub- tague, P. R., and Eagleman, D. M. V. Meese, T. S., and Hess, R. F. (2004). Low Conflict of Interest Statement: The
stitution masking. Vision Res. 48, (2007). Vividness of mental imagery: spatial frequencies are suppressively authors declare that the research was
937–945. individual variability can be mea- masked across spatial scale orienta- conducted in the absence of any com-
Breitmeyer, B. G., Tapia, E., Kafaligönül, sured objectively. Vision Res. 47, tion, field position, and eye of origin. mercial or financial relationships that
H., and Ögmen, H. (2008). Meta- 474–478. J. Vis. 4, 843–859. could be construed as a potential con-
contrast masking and stimulus D’Angiulli, A., and Reeves, A. Perky, C. (1910). An experimental study flict of interest.
contrast polarity. Vision Res. 48, (2002). Generating visual men- of imagination. Am. J. Psychol. 21,
2433–2438. tal images: latency and vividness 422–452. Received: 05 May 2012; accepted: 30 July
Briscoe, G., Dember, W., and Warm, inversely related. Mem. Cognit. 30, Pointer, J. S. (2001). The influence of 2012; published online: 15 August 2012.
J. S. (1983). Target recovery in 1179–1188. level and polarity of figure-ground Citation: Reeves A and Craver-Lemley
visual backward masking: no clear Dember, W. N., and Purcell, D. G. contrast on vision. Acta Ophthalmol. C (2012) Unmasking the Perky effect:
explanation in sight. J. Exp. Psy- (1967). Recovery of masked visual Scand. 79, 422–425. spatial extent of image interference on
chol. Hum. Percept. Perform. 9, targets by inhibition of the masking Reeves, A. (1980). Visual imagery in visual acuity. Front. Psychology 3:296.
898–911. stimulus. Science 757, 1335–1336. backward masking. Percept. Psy- doi: 10.3389/fpsyg.2012.00296
Craver-Lemley, C., and Arterberry, M. Finke, R. A. (1980). Levels of equiv- chophys. 28, 118–124. This article was submitted to Frontiers in
E. (2001). Imagery-induced interfer- alence in imagery and perception. Rodway, P., Gillies, K., and Schepman, Perception Science, a specialty of Frontiers
ence on a visual detection task. Spat. Psychol. Rev. 87, 113–132. A. (2006). Vivid imagers are better at in Psychology.
Vis. 14, 101–119. Haber, R. N. (1970). Note on how to detecting salient changes. J. Individ. Copyright © 2012 Reeves and Craver-
Craver-Lemley, C., Arterberry, M. E., choose a visual noise mask. Psychol. Differ. 27, 218–228. Lemley. This is an open-access article dis-
and Reeves, A. (1997). Effects of Bull. 74, 373–376. Segal, S. J. (1971). Imagery: Current Cog- tributed under the terms of the Creative
imagery on vernier acuity under Herzog, M. H. (2007). Spatial process- nitive Approaches. New York, NY: Commons Attribution License, which
conditions of induced depth. J. Exp. ing and visual backward masking. Academic Press. permits use, distribution and reproduc-
Psychol. Hum. Percept. Perform. 23, Adv. Cogn. Psychol. 3, 85–92. Segal, S. J., and Fusella, V. (1970). Effects tion in other forums, provided the original
3–13. Hilgard, E. R. (1981). Imagery and of imaging on signal-to-noise ratio, authors and source are credited and sub-
Craver-Lemley, C., and Reeves, A. imagination in American psychol- with varying signal conditions. Br. J. ject to any copyright notices concerning
(1987). Visual imagery selectively ogy. J. Ment. Imagery 5, 5–66. Psychol. 60, 459–464. any third-party graphics etc.

published: 10 July 2012
doi: 10.3389/fpsyg.2012.00224
Training visual imagery: improvements of metacognition,

but not imagery strength
Rosanne L. Rademaker 1,2,3 * and Joel Pearson 2,3,4
1
Cognitive Neuroscience Department, Maastricht University, Maastricht, Netherlands
2
Psychology Department, Vanderbilt University, Nashville, TN, USA
3
Vanderbilt Vision Research Center, Vanderbilt University, Nashville, TN, USA
4
School of Psychology, The University of New South Wales, Sydney, NSW, Australia
Edited by: Visual imagery has been closely linked to brain mechanisms involved in perception. Can
Stephen Michael Kosslyn, Stanford
visual imagery, like visual perception, improve by means of training? Previous research has
University, USA
demonstrated that people can reliably evaluate the vividness of single episodes of imag-
Reviewed by:
Stephen Michael Kosslyn, Stanford ination – might the metacognition of imagery also improve over the course of training?
University, USA We had participants imagine colored Gabor patterns for an hour a day, over the course of
Gregoire Borst, Université Paris five consecutive days, and again 2 weeks after training. Participants rated the subjective
Descartes, France
vividness and effort of their mental imagery on each trial. The influence of imagery on
*Correspondence:
subsequent binocular rivalry dominance was taken as our measure of imagery strength.
Rosanne L. Rademaker , Cognitive
Neuroscience Department, We found no overall effect of training on imagery strength. Training did, however, improve
Maastricht University, participant’s metacognition of imagery. Trial-by-trial ratings of vividness gained predictive
Universiteitssingel 40, 6229 ER power on subsequent rivalry dominance as a function of training. These data suggest
Maastricht, Netherlands.
that, while imagery strength might be immune to training in the current context, people’s
e-mail: rosanne.rademaker@
maastrichtuniversity.nl metacognitive understanding of mental imagery can improve with practice.
Keywords: visual imagery, training, learning, metacognition, introspection, binocular rivalry, consciousness
INTRODUCTION Sasaki et al., 2010); can imagery also improve with daily prac-
Mental imagery can be described as the retrieval of perceptual tice? There is some evidence to suggest that perceptual learning
information from memory, and the subsequent examination of can occur from training without physical stimulation. Repeti-
this information in the “minds eye.” Research has provided a tively imagining the crucial part of a visual bisection stimulus
growing body of behavioral and neuroimaging evidence that (visual spatial judgment) or imagining a low-contrast Gabor
there is considerable overlap between the “minds eye” and actual pattern (contrast judgment) can improve performance on sub-
perception (Chen et al., 1998; Kreiman et al., 2000; O’Craven sequent perceptual tasks (Tartaglia et al., 2009). Similarly, imag-
and Kanwisher, 2000; Zatorre and Halpern, 2005). For examining motor-acts facilitates performance on corresponding tasks
ple, behavioral studies have demonstrated that imagery content by training relevant parts of motor cortex, and by strengthen-
can selectively influence perception (Perky, 1910; McDermott and ing associations between processes and actions (Driskell et al.,
Roediger, 1994; Pearson et al., 2008). Imagery has been shown to 1994; Weiss et al., 1994; Feltz and Landers, 2007). To date,
affect visual detection thresholds (Ishai and Sagi, 1997), perfor- research has mainly focused on the effects imagery training
mance on a visual acuity task (Craver-Lemley and Reeves, 1992), has on subsequent perceptual tasks. Here, we look directly at
and to induce negative aftereffects (Gilden et al., 1995) in much the the influence of imagery training on the strength of imagery
same way as a sensory stimulus. Recent neuroimaging studies show itself.
that there is considerable spatial overlap between activated areas One of the hallmarks of mental imagery is the considerable
of the brain during both visual perception and visual imagery, for difference in reported imagery strength and vividness observed
example information about a pattern held in mind during work- across individuals (Galton, 1883; McKellar, 1965; Marks, 1973;
ing memory or imagery can be present in visual sensory cortex Amedi et al., 2005; Cui et al., 2007). Some individuals claim
(Kosslyn et al., 1995; Slotnick et al., 2005; Harrison and Tong, veridical, vivid imagery, while others doubt its entire existence
2009; Serences et al., 2009; Stokes et al., 2009). Like perception, (McKellar, 1965). The factors causing such differences in imagery
visual imagery is impaired when visual cortical activity is dis- strength remain largely unknown. One hypothesis is that individ-
turbed by means of transcranial magnetic stimulation (Kosslyn uals who actively practice, or whose everyday activities involve
et al., 1999). strong use of imagery, might have strengthened their imagery
If visual imagery can indeed be defined as the recreation of through training and practice (Sacks, 2010). We sought to exam-
a perceptual representation in the absence of retinal input (Ishai ine such a proposal in the lab by engaging individuals in
and Sagi, 1995), one may wonder exactly how similar imagery is an imagery task daily, over a period of 5 days. Can repeated
to perception. Specifically, prolonged visual practice can improve instances of forming visual imagery lead to improved imagery
perceptual skill (Fahle and Poggio, 2002; Fine and Jacobs, 2002; strength?
www.frontiersin.org July 2012 | Volume 3 | Article 224 | 23

Rademaker and Pearson The training of imagery and metacognition
To address this question researchers must be able to reli- be stable and task independent, suggesting a common cognitive
ably measure imagery strength from 1 day to the next. Previous process (Song et al., 2011).
work demonstrated that sustained imagery has a pronounced Little is known regarding the stability and independence of
and visually specific impact on subsequent perception (Pearson metacognition of mental imagery. If metacognition for percep-
et al., 2008, 2011). These studies utilized a visual phenomenon tual tasks originates from a common cognitive process, might
called binocular rivalry; when two different patterns are pre- a similar process allow people to have metacognition of mental
sented one to each eye, only one of the patterns is consciously imagery? Despite the highly subjective and volitional nature of
perceived. Subtle experimental manipulations, such as attention imagery, people are reasonably good at imagery metacognition
(Meng and Tong, 2004; Mitchell et al., 2004; Chong and Blake, (Pearson et al., 2011). Is this ability stable, or might metacognition
2006; Kamphuisen et al., 2007), sensory memory (Pearson and of imagery improve with repeated practice? Here, we also investi-
Brascamp, 2008), or imagery (Pearson et al., 2008), can bring about gated the degree of imagery metacognition as a function of daily
a slight imbalance in the neural states, creating a bias that helps training.
one pattern win the race for dominance at the expense of the To assess metacognition we use a method derived from signal
other. detection theory (Swets, 1986; Macmillan and Creelman, 1991;
We have previously demonstrated that imagery can alter future Galvin et al., 2003; Kornbrot, 2006) that has been successfully
competitive visual interactions in favor of the imagined stimulus employed in a variety of recent metacognition studies (Fleming
on a large percentage of trials (Pearson et al., 2008, 2011), while et al., 2010; Song et al., 2011). Using this method, we looked at the
catch-trial presentations of mock rivalry stimuli do not reveal such likelihood that imagery biased subsequent rivalry, given a certain
bias, ruling out the possibility of demand characteristics (Pylyshyn, level of imagery vividness. Signal detection allows us to estimate a
2003). Indeed, scores on offline imagery questionnaires predict single quantitative “sensitivity” measure of metacognitive ability,
imagery strength measured using rivalry (Pearson et al., 2011) and derived from these objective (amount of perceptual bias) and sub-
rivalry has been utilized to examine the role of imagery during jective (ratings of vividness) variables. This measure of sensitivity
visual working memory (Keogh and Pearson, 2011). Thus, there is is criterion free, which means that it is not prone to changes in cri-
compelling evidence that rivalry bias (or“perceptual bias”) is a use- terion (rating-magnitude), and it is not affected by irregular use
ful way to measure imagery strength in general (e.g., encompassing of the rating scale (which generally results in unequal numbers of
perceptual elements and sensations of vividness). In the current observations across the various conditions).
study, imagery strength is the underlying construct of interest, By way of preview, here we report that imagery strength – mea-
and the extent to which imagery biases perception is taken as a sured as the extent to which imagery biases perception during
reliable measure of imagery strength. The subjective experiences binocular rivalry – did not increase over the 5-day training period.
associated with imagery strength are probed by having participants Interestingly, participant’s metacognition of imagery did signifi-
report the “vividness” of their mental images. cantly improve over the training period. This dissociation between
Can people evaluate the phenomenal qualities of internally imagery strength and metacognitive ability suggests a degree of
generated experiences, such as whether a mental image is vivid independence between the two processes.
or detailed? Recently, an attempt was made to answer the ques-
tion of knowing ones own thoughts (exemplifying the problem MATERIALS AND METHODS
of “metacognition;” Flavell, 1979) in relation to mental imagery PARTICIPANTS
(Pearson et al., 2011). This study provided compelling new evi- Nine observers (six female) participated in the experiment. All had
dence that people have accurate metacognitive knowledge at normal or corrected-to-normal visual acuity and normal stere-
fine-grained scale, regarding specific instances of imagery: On ovision, and all provided written informed consent. Observers
individual trials, higher ratings of imagery vividness predicted a received payment for their participation ($10 per hour, plus a $5
greater likelihood that the imagined pattern would appear domi- per hour bonus upon completion) with the exception of a partici-
nant during subsequent rivalry (Pearson et al., 2011). Interestingly, pating author (RR) and participant BW. The study was carried out
repeated attempts to form a particular visual image can lead to dif- with the approval of the Institutional Review Board at Vanderbilt
ferent degrees of success with each try, causing imagery strength University.
to fluctuate from one moment to the next. Despite this variance in
imagery strength, people demonstrate good metacognitive under- MATERIALS
standing of their imagery, and can readily evaluate how vivid their Observers viewed the stimuli on a luminance-calibrated CRT
mental images are on a particular occasion. monitor with 1152 × 870 resolution and a 75-Hz refresh rate in
At a general level, there has been a growing interest in metacog- an otherwise darkened room. Visual stimuli were generated with
nitive judgments of memory and sensory decision-making (Kiani Matlab 7.5.0 (R2007b) and the Psychophysics toolbox (Brainard,
and Shadlen, 2009; Fleming et al., 2010; Rounis et al., 2010; Song 1997; Pelli, 1997) under Mac OSX. Observers sat at a viewing
et al., 2011). Frontal brain regions are important for introspec- distance of 56 cm, and used a chinrest to maintain a stable head
tive or metacognitive ability (Kepecs et al., 2008; Fleming et al., position. A mirror stereoscope was used to present a different pat-
2010), which suggests that the neural substrates of metacogni- tern to each eye, and binocular convergence of the two images was
tive ability are distinct from those supporting primary perception. aided by a white bull’s eye fixation dot (0.95˚) at the center of each
Although the ability to introspect varies substantially across indi- monocular half-image. Participants were instructed to maintain
viduals, within a single individual metacognitive ability seems to steady fixation throughout all experimental trials.
Frontiers in Psychology | Perception Science July 2012 | Volume 3 | Article 224 | 24

Rivalry stimuli consisted of a green and a red grating (spatial into the lab at or around the same time on each day of training,
frequency = 1.23 c/˚) surrounding a central fixation point, pre- and were dark-adapted for a couple of minutes before the start of
sented against a black background with a mean luminance of each experimental session.
0.09 cd/m2 . CIE color values of the stimuli were as follows – green: During the experiment, participants were briefly presented with
x = 0.293, y = 0.572; red: x = 0.602, y = 0.353. Gratings were pre- a randomly chosen (equal number of both) central cue (“G” for
sented at 75% contrast and had a Gaussian-shaped luminance green, or “R” for red) at the beginning of each imagery-trial
profile (mean luminance = 6.95 cd/m2 ) that faded to black at the (Figure 1A). Subsequently, participants would engage in visual
stimulus edge (Gaussian σ = 4.29˚). Five observers were trained imagery of the cued pattern for an 8-s period. After completing
with a green grating of orientation 112.5˚ and a red grating this imagery period, the word “vividness?” cued participants to
of orientation 22.5˚, while on generalization blocks they were first report the quality of their imagery by means of left-handed
presented with 67.5˚ green, and 157.5˚ red gratings. The oppo- button presses (1 = almost no imagery, 2 = some weak imagery,
site was true for the remaining four observers, meaning that 3 = moderate imagery, 4 = strong imagery almost like perception),
we counterbalanced which grating-pairs were used for training after which they were cued by the word “effort?” to report the
and generalization between participants. On catch trials, a mock amount of vigor with which they had tried to imagine the pattern
rivalry stimulus was presented consisting of a physical blend (1 = almost no effort, 2 = some effort, 3 = moderate effort, 4 = tried
of the green and red rivalry patterns. This stimulus was pre- very hard to form a mental image). Observers were instructed to
sented to both eyes simultaneously in order to avoid interocular use the full range of the rating scale to the best of their abilities.
competition. Presentation of the mock-stimulus allowed us to As soon as a participant had responded to both questions, a
test for decisional bias and demand characteristics (Landsberger, rivalry display (90% of trials) or a mock display (10% of trials)
1958). was presented for 750 ms. On rivalry trials, the green grating was
The dominant eye plays a key role in determining which of presented to the left eye, and the red grating to the right eye. On
two monocular images is likely to be perceived at the onset of mock trials, the plaid-stimulus was presented to both eyes simul-
binocular rivalry. Therefore, individual fine-tuning of stimulus taneously. Participants reported which image had appeared most
contrast was done before the start of the experiment, and before dominant, by pressing one of three buttons (1 = green, 2 = mixed,
each daily session, to control for differences in ocular dominance 3 = red). For this response, the right hand was used in order to
between observers. We used the same procedure as in previous minimize potential response conflict between the two hands. A
research (Pearson et al., 2008, 2011; Keogh and Pearson, 2011), “mixed” response could be made on all trials (rivalry and mock
matching the relative strength of the rivalry gratings to the point trials). On rivalry trials, the observer could give a mixed response
at which perceptual competition is most balanced, and thus most in case he or she was unable to distinguish which grating had
susceptible to disruption. appeared more dominant due to binocular combination or piece-
meal rivalry. This type of mixed percept was reported on 6.49% of
PROCEDURE rivalry trials (SEM = 2.49%).
To investigate whether visual imagery can be improved by means A single training session consisted of two blocks of 70 trials
of training, and to see how this relates to metacognition of imagery each. Within each block, seven catch trials were randomly inter-
over time, we had observers perform a visual imagery task on five leaved between the rivalry trials. We tested potential generalization
consecutive days, for about an hour a day. A sixth follow-up ses- of learning to non-trained orientations on day 1 and 5 of train-
sion was conducted 2–3 weeks after training. Participants came ing, and during follow-up. On these days, observers performed
A B
p(self-report | perceptual bias)
Cue Imagery Subjective ratings Rivalry 0.8
0.6
right eye
0.4
R vivid? effort?
left eye 0.2
0
1s 8s ~1s ~1s 0.75s 1s 0 0.2 0.4 0.6 0.8 1
p(self-report | no perceptual bias)
FIGURE 1 | Sample trial sequence and example ROC II curve. (A) of the two. On 10% of the trials – instead of the rivalry display – a
Participants were presented with a random cue (“G” or “R”) after which they mock-stimulus was presented to both eyes simultaneously, consisting of a
formed a mental image of the green or red grating over an 8-s period. physical combination of both the green and red grating. (B) To determine how
Participants were then cued to report the vividness of the imagined item, and well subjective ratings predict perceptual bias, type II ROC sensitivity was
the effort exerted while imagining the item, on an absolute scale from 1 to 4. calculated by taking the area under the ROC curve (Aroc ). This area is the sum
After a brief flash of the rivalry display, participants reported which grating had of the area of the half-square triangle (dark-gray shaded region) and the area
appeared perceptually dominant, or whether their percept was an equal mix between the diagonal and the ROC function (light-gray shaded region).

twice the amount of trials, with training and generalization blocks is the sum of the area of the half-square triangle (dark-gray shaded
presented separately and in a randomized order. region in Figure 1B) and the area between the diagonal and the
ROC function (light-gray shaded region in Figure 1B):
ANALYSES
To assess the strength of visual imagery, we looked at the per- 4 h
X 2 2 i
ceptual facilitation (or bias) of imagery on rivalry. This was Aroc = 0.25 hi+1 − fi − hi − fi+1 + 0.5 (1)
calculated as the percentage of trials in which the imagined grat- i=1
ing matched subsequent perception during rivalry (Pearson et al.,
The bias of the ROC II curve (B roc ) was defined as the ratio
2008), excluding trials on which a mixed percept was reported. A
K B /K A , where K B is the area between the ROC curve and the
perceptual bias greater than 50% (chance) on the rivalry trials but
major diagonal (dashed line in Figure 1B) to the right of the
not on the catch trials suggests facilitation due to imagery con-
minor diagonal (dotted line in Figure 1B), and K A is the area
tent. Due to experimenter error, a small number of runs (7 out
between the ROC curve and the major diagonal to the left of the
of 108) were missing from the data. Where necessary, we used tri-
minor diagonal. A neutral bias would give B roc equal to zero, while
linear interpolation to infer the mean percentage of bias. For the
a negative or positive B roc indicates a bias toward lower or higher
day-by-day analysis (Figure 2) only one data point was interpo-
ratings respectively.
lated (percentage perceptual bias for participant CB on day 4); the
session-by-session analysis of the same data required interpolation  2 h

2 2 i
of all seven missing runs.
P
 0.25 h i+1 − fi − hi − fi+1 
Data obtained from subjective ratings of vividness (and effort) Broc = ln 
 i=1 
(2)
4 h

were analyzed by constructing type II receiver operating charac-  P 2 2 i 
0.25 hi+1 − fi − hi − fi+1
teristic (ROC) curves for each participant on each day of training. i=3
This method of assessing metacognitive ability is derived from
signal detection methods (Swets, 1986; Macmillan and Creelman, Previous research has shown that subjective ratings of vivid-
1991; Galvin et al., 2003; Kornbrot, 2006) and has been success- ness – but not effort – are predictive of how much perceptual bias
fully employed in a variety of recent inquiries about metacognition someone experiences (Pearson et al., 2011). To confirm this, we
(Fleming et al., 2010; Song et al., 2011). Essentially, the only differ- also applied the ROC II methods described above to participant’s
ence between type I and type II tasks lies in the event that is being ratings of exerted effort. This effort-based ROC thus characterizes
discriminated: Type I decisions are about the occurrence of events the probability of a participant being perceptually biased given a
independent of the observer (so, distinguishing between signal and certain level of self-reported effort. Finally, to determine whether
noise), whereas type II decisions are about whether a decision was the ROC II model did a good job accounting for our metacognitive
correct or incorrect (so, making a metacognitive judgment). data, we fit a linear regression model:
Since it is not possible to be “correct” or “incorrect” about
z (h) = β0 + β1 z f + ε

an internally generated image, we adapted the definition of the (3)
type II decision to include judgments about the vividness of
single episodes of mental imagery and its effect on the per- Where z is the inverse of the cumulative normal distribu-
ceptual outcome during brief instances of subsequent binocular tion function. The ROC II model provided a good fit to the
rivalry. Applying the signal detection logic, we categorized trials self-reported vividness (mean R 2 = 0.976 ± 0.004) and effort data
where participants reported high vividness and where imagery (mean R 2 = 0.981 ± 0.007).
subsequently biased perception as “hits.” Trials where partici-
pants reported high vividness but perception was not subsequently RESULTS
biased were categorized as “false alarms.” Here, the ROC II char- IMAGERY TRAINING
acterizes the probability of a participant being perceptually biased Sustained mental imagery can bias the perception of an ambigu-
during rivalry, given a certain level of self-reported vividness. ous display, resulting in a reliable measure of imagery strength on
To construct the ROC II curves, we calculated p(rating = i | a trial-to-trial basis (Pearson et al., 2008, 2011). When people rate
perceptual bias) and p(rating = i | no perceptual bias) for all i, their imagery as more vivid, the likelihood that imagery influences
and transformed these into cumulative probabilities before plot- perception is larger (Pearson et al., 2011). Thus, if training men-
ting them against each other (anchored at [0,0] and [1,1]). tal imagery would result in more vivid images, one would expect
Distribution-free methods were employed to characterize type to see an increase of perceptual bias over time. Figure 2 shows
II ROC sensitivity by calculating the area under the ROC curve the mean imagery strength (or “perceptual bias”) as a function of
(A roc ), and type II ROC bias (B roc ). These parameters are derived days of training and again 2 weeks later. A within-subjects ANOVA
from simple geometry and do not make assumptions about the revealed that training did not increase the amount of perceptual
shape of the distribution (Kornbrot, 2006). The area under the bias over time [F (5,40) < 1].
ROC curve (A roc ) quantifies the extent to which metacogni- Mental imagery did bias perception in favor of the imagined
tive judgments are predictive of perceptual bias during rivalry grating [F (5,40) = 8.861; p = 0.018] which is consistent with pre-
(Figure 1B); a diagonally flat ROC function indicates little previous work demonstrating the effect of mental imagery on rivalry
dictive value of the metacognitive judgment on the subsequent (Pearson et al., 2008, 2011). Unsurprisingly – considering the
perceptual outcome during rivalry. The area under the ROC curve lack of a training effect – gratings of both trained and untrained

negligible influence on participant’s reports of rivalry dominance,

70 as previously documented (Pearson et al., 2011).
Perceptual bias (% of trials)
65 METACOGNITIVE JUDGMENTS
To assess whether people’s metacognitive insights about imagery
strength improve over the course of training, we constructed ROC
60
II curves for each individual observer, on each day of training
(Figure 4; Materials and Methods). The extent to which metacog-
55 nitive judgments of vividness predict perceptual bias was quan-
tified as the area under each ROC II curve. Data presented in
50 Figure 4 demonstrate that on earlier days of training (darker green
training lines) the area under the curve is smaller than on later days of
generalization training (lighter green lines). The upward bowing profile of the
45
1 2 3 4 5 2-3 weeks curves observable in over half of our participants demonstrates
Day of training that vividness judgments indeed predict perceptual bias.
There is a clear trend toward more metacognitive abil-
FIGURE 2 | Imagery biased perception during rivalry in favor of the ity over time for both the trained [F (5,40) = 1.742; p = 0.147]
previously imagined grating, but this perceptual bias did not
and untrained (generalization) orientation [F (2,16) = 7.416;
significantly change over the course of a 5-day training. The blue line
represents the mean training data, and the blue shaded regions represent p = 0.005]. Trained and untrained orientations do not statisti-
±1 SEM. The parts of the plot depicted in red represent data from blocks cally differ [F (1,8) < 1]. This lack of orientation specificity may
where we tested training generalization to other orientations. not be surprising considering that metacognition for perception
is something presumably supported by higher-level frontal areas
of the brain (Fleming et al., 2010), where responses are invariant
(generalization) orientations yielded similar perceptual biases: a to fine-grained orientation information. Hence, we simplified our
within-subjects ANOVA for training days 1, 5, and follow-up analysis by collapsing the data from all orientations before con-
revealed no main effect of orientation [F (1,8) < 1]. Analyzing the structing the ROC II curves as displayed in Figure 4. Estimates of
data by session did not unveil any hidden differences in perceptual the type II ROC sensitivity A roc are therefore slightly more reliable
bias over time [F (11,88) = 1.106; p = 0.366], which excludes the on day 1, day 5, and during follow-up, since they are constructed
possibility that most learning took place between the first couple based on more data.
of sessions. The information from Figure 4 is summarized in Figure 5,
Additional evidence that mental imagery was not improved showing the main effect of training: vividness judgments pre-
by training comes from participant’s introspective judgments dict perceptual bias increasingly better over time [F (5,40) = 3.075;
of imagery vividness. Mean self-reported vividness of men- p = 0.019]. This trend is linear when only looking at training
tal imagery was statistically the same on all days of training days 1–5 [F (1,8) = 5.846; p = 0.042] but becomes quadratic when
[F (5,40) = 1.224; p = 0.316]. Self-reports of exerted effort did not follow-up is included [F (1,8) = 8.778; p = 0.018], indicating a drop
change over the course of training either [F (5,40) < 1]. In summary, of the proportion A roc at follow-up. Nevertheless, planned com-
neither the perceptual measure of imagery strength (“perceptual parisons (uncorrected t -tests) show that – with the exception of
bias”) nor ratings of vividness showed any significant change over day 1 – the predictive value of self-reported vividness on the per-
the 5-days of training. Thus, it appears that training in this study ceptual outcome is larger than would be expected by chance (one-
was unable to increase imagery strength over time. tailed one-sample t -test day 1: p = 0.243; all others: p < 0.029),
and this ability is still present 2–3 weeks after training (p = 0.021).
CATCH TRIALS In previous work we demonstrated that people can reliably
Catch trials were presented in a randomly interleaved fashion on evaluate the vividness of their mental imagery from one trial
10% of all experimental trials, to determine whether observers to the next (Pearson et al., 2011). Participants in that previous
showed response bias in favor of the imagined grating. On these study were tested only once. Thus, one might expect to find evi-
trials, a mock rivalry display was presented consisting of a balanced dence for metacognition of imagery strength on day 1 of training
physical combination of the green and red gratings shown to both in the current experiment. However, the difference in analyses
eyes simultaneously. If the effects observed during rivalry were due used to determine metacognition in our previous and current
to decisional bias or demand characteristics, we expect to find the work (within-subjects analysis of variance, and area under ROC II
same degree of response bias on catch trials. We analyzed bias by curve respectively), make it hard to directly compare the findings.
coding veridical “mixed” responses to the catch trials as 50%, while A within-subjects analysis of variance performed on the current
responses that matched the cued pattern were coded as 100%, and data shows that on day 1 of training, participants marginally (but
responses opposite to the cued grating were coded as 0%. The not significantly) showed a main effect of vividness on perceptual
percentage of catch trials during which participant’s responses bias [F (3,15) = 2.83; p = 0.074]. However, a lack of observed power
were biased in favor of the cued grating are shown in Figure 3 (0.558) indicates that at this sample size there is only a small (44%)
(for all days of training). On average, this bias was 50.79%. This chance of finding a significant effect (at α = 0.05) when assuming
indicates that demand characteristics and decisional bias have a that people have metacognitive insights into their own imagery

100
day 1
90 day 2
day 3
Perceptual bias (% of trials)
80 day 4
day 5
70 2−3 weeks
60
50
40
30
20
10
RR BW RS EE CB DB EY MM AA
0
FIGURE 3 | Mean percentage of catch trials on which participant’s response to the fake-rivalry display was biased in favor of the imagined grating. A
score of 50% indicates a lack of bias. Gray shaded areas show each individual participant, the colored bars represent the 5 days of training plus follow-up
2–3 weeks later.
1 1 1
RR BW RS
0.8 0.8 0.8
0.6 0.6 0.6
0.4 0.4 0.4
0.2 0.2 0.2
0 0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
1 1 1
EE CB DB
p(vividness | perceptual bias)
0.8 0.8 0.8
0.6 0.6 0.6
0.4 0.4 0.4
0.2 0.2 0.2
0 0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
1 1 1
EY MM AA
0.8 0.8 0.8
day 1
0.6 0.6 0.6 day 2
day 3
0.4 0.4 0.4 day 4
day 5
0.2 0.2 0.2 2−3 weeks
0 0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
p(vividness | no perceptual bias)
FIGURE 4 | Participant’s metacognition of imagery over the course of training. ROC II curves based on the vividness ratings reported by our participants
(collapsed over Gabor orientations). Each plot represents an individual participant; the colored lines represent the ROC curves for the different days of training.

strength at the population level. An a priori power analysis indi- that, as expected, effort did not predict perceptual bias [F (1,8) < 1].
cates that, assuming a medium effect size, 21 subjects would be The area under the ROC II curve (A roc ), which quantifies the
required to obtain a power of 0.95. degree to which self-reported effort predicts perceptual bias during
Vividness ratings are predictive of the efficacy that mental rivalry, did not differ from chance on any of the training days (two-
imagery has at biasing the perception of rivaling stimuli. By tailed one-sample t -tests all p > 0.081). Neither did we observe a
contrast, self-reported effort for imagery was not hypothesized to change over time for the trained [F (5,40) < 1], untrained (general-
predict perceptual bias. Attempts to exert greater effort do not nec- ized) [F (2,16) = 2.711; p = 0.097], or collapsed [F (5,40) < 1] grating
essarily result in highly effective imagery, as demonstrated by pre- orientations.
vious work (Pearson et al., 2011). To ensure that our findings were Vividness appears to predict perceptual bias more strongly over
specific to introspective vividness – and not effort – we constructed the course of training, whereas effort does not predict perceptual
ROC II curves (as in Figure 4; Materials and Methods) based on bias at all. Can this finding be explained by the way participants
the effort ratings reported by our participants. The pooled (across used the rating scales? Participant’s average reported vividness
participants) curves per day are shown in Figure 6A; the diagonally (2.57 ± 0.21) and effort (2.86 ± 0.14) did not significantly dif-
flat function indicates a weak link between self-reported effort rat- fer (p = 0.261). In other words, subjective vividness – but not
ings and perceptual bias during rivalry. Figure 6B demonstrates effort – is predictive of how well something was imagined inde-
pendent of rating-magnitude. Signal detection theory considers
metacognitive ability (sensitivity) and rating-magnitude (bias) as
two independent properties (Galvin et al., 2003). In accordance
0.7
with this notion, we find that individuals with higher self-reported
Proportion A roc vividness
0.65 vividness were not better at evaluating their imagery strength

and vice versa. Specifically, participant’s ability to make accurate
0.6 metacognitive judgments of their mental imagery (A roc vivid-
ness) and participant’s mean vividness ratings were not correlated
0.55 (r = 0.063; p = 0.873).
The type II bias of the ROC curve (B roc ) provides us with a
0.5 distribution-free estimate of the criterion used by participants to
provide their subjective ratings. A neutral bias would give B roc
0.45
equal to zero, while a negative or positive B roc indicates a bias
0.4 toward lower or higher ratings respectively. This estimate corre-
1 2 3 4 5 2-3 weeks sponded very well with the actual rating-magnitude collected dur-
Day of training ing the experiment: self-reported vividness ratings and estimated
vividness bias (B roc vividness) were highly correlated (r = 0.913;
FIGURE 5 | Imagery metacognition improves over the course of p < 0.001), as were self-reported effort and estimated effort bias
training. The extent to which self-reported vividness predicts perceptual (r = 0.889; p = 0.001). The close resemblance between these two
bias (quantified as the area under the ROC II curve after collapsing all
variables – both measures of participant’s criterion – helps validate
grating orientations, as also shown in Figure 4) plotted against day of
training. Shaded areas represent ±1 SEM. the distribution-free approach used to determine ROC estimates
in the current paradigm.
A B 0.7
1
p(effort | perceptual bias)
Proportion Aroc effort
0.65
0.8
0.6 day 1 0.6

day 2
0.4 0.55
day 3
day 4
0.2 0.5
day 5
2−3 weeks 0.45
0
0 0.2 0.4 0.6 0.8 1
0.4
p(effort | no perceptual bias) 1 2 3 4 5 2-3 weeks
Day of training
FIGURE 6 | Participant’s self-reported effort with which the effort predicts perceptual bias (quantified as the area under the ROC II
gratings were imagined does not predict perceptual bias. (A) ROC curve) plotted against each day of training. Self-reported effort does not
II curves based on the effort ratings pooled across participants. Green predict perceptual bias, and this does not change over time. Data was
colored lines represent the different days of training. The diagonally flat collapsed over Gabor orientations, since outcomes did not differ
functions indicate a weak link between self-reported effort ratings and between the two sets of orientations [F (1,8) < 1]. Shaded areas
perceptual bias during rivalry. (B) The extent to which self-reported represent ±1 SEM.

Strikingly, Figure 4 shows large differences between individu- through imagery training, yet, changes at this cortical level may
als: the degree to which metacognitive vividness judgments predict not be sufficient to improve imagery strength itself.
perceptual bias varies quite a bit from one person to the next Historically, mental imagery has been considered a fainter form
(A roc = 0.48–0.71). This type of variability is not uncommon, and of perception (Hume, 1739). Evidence to support this notion
previous studies have reported similarly large individual differ- comes from functional magnetic resonance imaging (fMRI) stud-
ences in metacognitive ability for perceptual tasks (Song et al., ies demonstrating that the magnitude of brain activity is lower
2011). Besides large differences related to metacognition, also the during imagery than during bottom-up perception (Goebel et al.,
overall amount of perceptual bias experienced by our partici- 1998; O’Craven and Kanwisher, 2000). Likewise, single neuron
pants varied widely (46–76%). Nonetheless, participant’s overall recordings in the medial temporal lobe of humans found fewer
metacognitive ability and the percent perceptual bias they experi- neurons that were recruited during imagery than during per-
enced throughout the experiment, were uncorrelated (r = −0.027; ception, and that the firing rate of these cells was lower during
p = 0.945). This suggests that participant’s metacognitive ability imagery compared to perception (Kreiman et al., 2000). In the
in this task is independent of imagery strength, as measured by case of perceptual bias during rivalry, imagery is presumed to
binocular rivalry. influence or boost the memory trace that exists between one
rivalry presentation and the next, and the location and orienta-
DISCUSSION tion specificity of this memory trace implies that it is composed
The research presented here suggests that mental imagery strength of primarily low-level characteristics (Ishai and Sagi, 1995; Pear-
does not improve over the course of our 5-day training regime. son et al., 2008; Slotnick, 2008). Mechanisms such as a gain in
Over the 5-days, no changes were observed relating to imagery sensitivity for the imagined pattern, or the strengthening of sen-
strength as measured by rivalry, nor were there any changes in sory traces, would be needed to modify population activity in
the average introspective judgments of imagery vividness. We fur- lower visual areas mediating alternations of conscious percep-
ther demonstrated that self-reported vividness of mental imagery tion during rivalry. Imagery may simply lack sufficient impact
predicts the perceptual consequences of single epochs of imagery. to induce permanent plastic changes at these lowest sensory levels.
More importantly, this prediction becomes stronger with practice, Future research directions aiming to improve imagery strength
implying increased metacognition of imagery over the course of could investigate the necessity of bottom-up information for
training. Self-reported effort of mental imagery on the other hand, learning. Specifically, it would be interesting to see if there is a
did not predict perceptual outcomes. transfer from improving visual perception by means of prolonged
There have been reports of visual imagery increasing perfor- training with actual sensory stimuli, to improvements of imagery
mance on subsequent perceptual tasks (Tartaglia et al., 2009). Yet strength.
we were unable to find an increase in facilitation of rivalry domi- One could hypothesize that imagery strength is liable to
nance after 5 days of training. The question is of course, why? The improvement, but we simply failed to find any in this study due to
emphasis of the research presented here was on improving imagery the configuration of our task. Research into the process of improv-
strength over time. This is a notably different emphasis from stud- ing perceptual skill – or perceptual learning – provides useful con-
ies that have investigated how imagery training changes perceptual text in support of this hypothesis. One influential view known as
skills (Tartaglia et al., 2009). One obvious explanation for the lack the reverse hierarchy theory (Ahissar and Hochstein, 2004), states
of an imagery training effect in this study is that imagery strength that learning is gated by top-down, task-related factors: Learning
simply cannot improve with practice. This idea is corroborated begins at high-level areas of the brain, after which it trickles down
by the fact that neither imagery bias, nor subjective ratings of the hierarchy, fine-tuning the read out from lower level areas. This
imagery strength showed a significant increase as a function of theory invokes a number of detailed predictions, namely, early
training. Introspective ratings of imagery strength are reflected (fast) learning should be related to high-level changes, whereas
in the perceptual outcomes during rivalry, and the close relation- asymptotic (slow) learning should involve plasticity in low-level
ship between the two implies they measure the same underlying sensory areas – if required by the task. There is considerable evi-
construct (Pearson et al., 2011). Thus, several aspects of the data dence supporting this view (Ahissar and Hochstein, 1993, 1997;
support the hypothesis that it is not possible – or very difficult – to Dosher and Lu, 1998; Dupuis-Roy and Gosselin, 2007).
improve imagery strength by means of training. In light of the reverse hierarchy framework, our training regime
The idea that training cannot easily change imagery strength is suspect to a critical vulnerability. Namely: training duration. Five
might be explained by the manner in which imagery strength is days may have been insufficient time to reach the asymptotic learn-
linked to brain anatomy. The Tartaglia study (Tartaglia et al., 2009) ing phase. The Tartaglia study previously mentioned (Tartaglia
had participants repetitively imagine the crucial part of a bisection et al., 2009) trained participant’s imagery for 10 days, twice as long
stimulus (spatial judgment) or a low-contrast Gabor pattern (con- as in our study, and found an improvement on perceptual tasks.
trast judgment). They found improved perceptual performance Assuming that specific cellular plastic processes at the hierarchical
on a subsequent perceptual bisection task and a Gabor detection level of ocular dominance columns can only occur during asymp-
task after imagery training, and this improvement generalized to totic learning, longer training might be necessary when aiming to
untrained orientations. This lack of orientation specificity implies influence rivalry perception.
that learning through imagery did not involve plastic changes in Recent research has demonstrated that perceptual learning can
early visual cortex, but probably involved higher-level extra-striate also occur without a specific task and outside of awareness, as
areas. Higher-level changes may boost perceptual performance long as the information of interest is paired with feedback or

a reward signal (Seitz and Watanabe, 2003, 2005; Seitz et al., (Fleming et al., 2010), and there is a marked dissociation between
2009) or with online-feedback via decoded fMRI signals (Shi- metacognitive ability and performance on visual perceptual tasks
bata et al., 2011). Our experimental design lacked a direct reward (Lau and Passingham, 2006; Lau, 2008; Rounis et al., 2010; Song
signal. Perhaps if successful epochs of imagery were paired et al., 2011). Thus, metacognitive ability can be viewed as a stable
with a reward signal, this could facilitate learning. In practice and task independent cognitive process that can be improved with
the implementation of a reward may prove difficult to realize. practice, independent of performance on other tasks. Changes
Often, measures of imagery strength are dependent on subjec- in high-level neuronal populations are likely candidates for this
tive reports, and offering rewards based on only self-reports learning.
could induce strong response and observer biases. Neverthe- The ability to introspect on private thoughts is key to human
less, it is possible that our training was insufficient to obtain an subjective experience. Yet, people’s ability to evaluate internally
effect, and providing feedback, rewards, or some manner of get- generated experiences – such as imagery – is not as self-evident as it
ting participants to intentionally try and increase their imagery may appear. Although a large number of studies now demonstrate
strength, could have been a more effective way to train mental that something as private as a mental image can be successfully
imagery. studied from a third-person perspective (Ishai and Sagi, 1995,
During memory consolidation, initially fragile memory traces 1997; Kosslyn et al., 2001; Pearson et al., 2008; Tartaglia et al.,
become stabilized due to practice-induced plasticity in task rele- 2009), research has only recently begun to tackle issues related
vant brain areas (Karni, 1996; Dudai, 2004). Can the ineffectiveness to the first-person perspective (Pearson et al., 2011). The core
of imagery training be due to somehow disrupted memory consol- problem from the first-person perspective of the imaginer is that
idation? Classically, consolidation has been defined as a time lim- self-generated instances of imagery, unlike perception, cannot be
ited process directly following learning (Dudai, 2004). However, directly compared with a perceptual template. Nevertheless, peo-
recent studies indicate that interference is rather time indepen- ple seem quite capable of knowing if a mental image is accurate,
dent, and can occur at long intervals after training (Goedert and vivid, or detailed. And practice further improves this first-person
Willingham, 2002; Caithness et al., 2004; Zhang et al., 2008). Inter- introspective ability. Why might such metacognitive knowledge be
ference can be considered strongly stimulus dependent, resulting important?
from similarity between the learned and interfering stimulus, and Introspective or “metacognitive” sensitivity is important to
the corresponding neuronal populations recruited by these stimuli guide actions and to make decisions (Vickers, 1979; Daw et al.,
(Seitz et al., 2005; Been et al., 2011). Specifically, for Gabor pat- 2005; Dayan and Daw, 2008) and being able to adequately esti-
terns most interference occurs when interfering stimuli differ from mate ones confidence can help drive adaptive behavior (Kepecs
the learned orientation by 30˚, while no interference is observed et al., 2008). In its simplest form, low confidence that a recent
from orthogonal orientations (Been et al., 2011). Considering the decision was correct may prompt reexamination of the evidence,
orthogonal training orientations of our experiment, disruption of or seeking a second-opinion. In the event of internally gener-
consolidation seems an unlikely explanation for the ineffectiveness ated experiences such as mental imagery, low confidence that an
of imagery training. image was veridical and life-like may lead someone to recon-
Can people become better at knowing their own thoughts? We sider such an experience. A better metacognitive understanding
were able to improve subject’s ability to judge the vividness of their may help the imaginer bridge the gap between first and third-
imagery. This improvement was still present during a follow-up person perspective. For example, people can resolve potential
test, implying a long lasting effect of training on metacognitive ambiguities about perception by comparing their own percep-
evaluation of mental imagery. Furthermore, training of metacog- tual experience with the subjective experience of another person
nition was not orientation specific: metacognition was improved (Bahrami et al., 2010). Similarly, when the imaginer has a better
for both trained and untrained sets of orientations. It is likely understanding of the authenticity of his or her mental image, it
that the improvement of metacognition reported here originates will be easier to communicate its content to another person. In
from higher-level brain areas. This is in concordance with the sum, increasing the efficiency with which people introspect the
suspected high-level neural locus of metacognitive ability for per- quality of their mental images can prove a novel and important
ception (Kepecs et al., 2008; Kiani and Shadlen, 2009; Fleming finding.
et al., 2010) as well as the idea that networks in high-level cortical In conclusion, we discussed a variety of reasons why train-
regions orchestrate strategic choices during early learning, allocat- ing did not lead to an improvement of imagery strength in the
ing attention and motivation in response to specific task demands current study. Such an improvement may simply be very dif-
(Willingham, 1999; Hochstein and Ahissar, 2002; Doyon et al., ficult to document, or our task may not have been optimally
2003). suited to detect improvements of imagery strength. Neverthe-
Our study demonstrated improvements of the metacogni- less, we demonstrated that people’s ability to introspect their
tion of imagery, whereas no changes in imagery strength itself own imagery strength does improve with training, which sug-
were observed. This dissociation suggests distinct brain mech- gests distinct mechanisms underlying imagery and metacognition.
anisms underlying metacognition and visual imagery respec- Being able to improve metacognition by means of practice can
tively. Similar distinctions have been made regarding metacogni- have important implications for real-life situations. It would be
tion of perception: Neuroanatomical substrates of introspective interesting to know if training metacognition could help people
ability are distinct from those supporting primary perception improve certain cognitive functions, such as decision-making or

planning actions. If so, this may prove especially helpful for specific ACKNOWLEDGMENTS
patient populations. Finally, future investigations of prolonged We thank Frank Tong for discussions and helpful input with design
training of imagery can prove advantageous in outlining the over- and execution of this work. We also thank Jan Brascamp, Elias
lap between mechanisms of perception and imagery. Imagery as Cohen, Janneke Jehee, Jan Schepers, and Jascha Swisher for helpful
defined here is a highly voluntary process that allows introspec- comments and discussions regarding this research. This work was
tion in the absence of direct perceptual input. As such, imagery funded by grants from the National Health and Medical Research
can provide a unique gateway to understanding how perceptual Council of Australia CJ Martin Fellowship 457146 and project
and introspective processes are represented in the brain. grant APP1024800 to Joel Pearson.
REFERENCES Daw, N. D., Niv, Y., and Dayan, P. in the theory of signal detectability: correlates, computation and behav-
Ahissar, M., and Hochstein, S. (1993). (2005). Uncertainty-based competi- discrimination between correct and ioural impact of decision confi-
Attentional control of early perception between prefrontal and dorso- incorrect decisions. Psychon. Bull. dence. Nature 455, 227–231.
tual learning. Proc. Natl. Acad. Sci. lateral striatal systems for behavioral Rev. 10, 843–876. Kiani, R., and Shadlen, M. N. (2009).
U.S.A. 90, 5718–5722. control. Nat. Neurosci. 8, 1704–1711. Gilden, D., Blake, R., and Hurst, G. Representation of confidence asso-
Ahissar, M., and Hochstein, S. (1997). Dayan, P., and Daw, N. D. (2008). Deci- (1995). Neural adaptation of imag- ciated with a decision by neurons
Task difficulty and the specificity sion theory, reinforcement learning, inary visual motion. Cogn. Psychol. in the parietal cortex. Science 324,
of perceptual learning. Nature 387, and the brain. Cogn. Affect. Behav. 28, 1–16. 759–764.
401–406. Neurosci. 8, 429–453. Goebel, R., Khorram-Sefat, D., Muckli, Kornbrot, D. E. (2006). Signal detec-
Ahissar, M., and Hochstein, S. (2004). Dosher, B. A., and Lu, Z. L. (1998). L., Hacker, H., and Singer, W. tion theory, the approach of choice:
The reverse hierarchy theory Perceptual learning reflects exter- (1998). The constructive nature of model-based and distribution-free
of visual perceptual learning. nal noise filtering and internal vision: direct evidence from func- measures and evaluation. Percept.
Trends Cogn. Sci. (Regul. Ed.) 8, noise reduction through channel tional magnetic resonance imag- Psychophys. 68, 393–414.
457–464. reweighting. Proc. Natl. Acad. Sci. ing studies of apparent motion and Kosslyn, S. M., Ganis, G., and Thomp-
Amedi, A., Malach, R., and Pascual- U.S.A. 95, 13988–13993. motion imagery. Eur. J. Neurosci. 10, son, W. L. (2001). Neural founda-
Leone, A. (2005). Negative BOLD Doyon, J., Penhune, V., and Ungerlei- 1563–1573. tions of imagery. Nat. Rev. Neurosci.
differentiates visual imagery and der, L. G. (2003). Distinct contri- Goedert, K. M., and Willingham, D. B. 2, 635–642.
perception. Neuron 48, 859–872. bution of the cortico-striatal and (2002). Patterns of interference in Kosslyn, S. M., Pascual-Leone, A., Feli-
Bahrami, B., Olsen, K., Latham, P. E., cortico-cerebellar systems to motor sequence learning and prism adap- cian, O., Camposano, S., Keenan,
Roepstorff, A., Rees, G., and Frith, skill learning. Neuropsychologia 41, tation inconsistent with the consol- J. P., Thompson, W. L., Ganis, G.,
C. D. (2010). Optimally interacting 252–262. idation hypothesis. Learn. Mem. 9, Sukel, K. E., and Alpert, N. M.
minds. Science 329, 1081–1085. Driskell, J., Copper, C., and Moran, 279–292. (1999). The role of area 17 in
Been, M., Jans, B., and De Weerd, P. A. (1994). Does mental practice Harrison, S., and Tong, F. (2009). visual imagery: convergent evidence
(2011). Time-limited consolidation enhance performance? J. of Appl. Decoding reveals the contents of from PET and rTMS. Science 284,
and task interference: no direct link. Psychol. 79, 481–491. visual working memory in early 167–170.
J. Neurosci. 31, 14944–14951. Dudai, Y. (2004). The neurobiology visual areas. Nature 458, 632–635. Kosslyn, S. M., Thompson, W. L., Kim,
Brainard, D. H. (1997). The psy- of consolidations, or, how stable is Hochstein, S., and Ahissar, M. (2002). I. J., and Alpert, N. M. (1995). Topo-
chophysics toolbox. Spat. Vis. 10, the engram? Annu. Rev. Psychol. 55, View from the top: hierarchies and graphical representations of men-
433–436. 51–86. reverse hierarchies in the visual sys- tal images in primary visual cortex.
Caithness, G., Osu, R., Bays, P., Chase, Dupuis-Roy, N., and Gosselin, F. (2007). tem. Neuron 36, 791–804. Nature 378, 496–498.
H., Klassen, J., Kawato, M., Wolpert, Perceptual learning without signal. Hume, D. (1739). A Treatise on Human Kreiman, G., Koch, C., and Fried, I.
D. M., and Flanagan, J. R. (2004). Vision Res. 47, 349–356. Nature, Vol. 1. London: John Noon. (2000). Imagery neurons in the
Failure to consolidate the consolida- Fahle, M., and Poggio, T. (2002). Percep- Ishai, A., and Sagi, D. (1995). Common human brain. Nature 408, 357–361.
tion theory of learning for sensori- tual Learning. Cambridge, MA: MIT mechanisms of visual imagery Landsberger, H. A. (1958). Hawthorne
motor adaptation tasks. J. Neurosci. Press. and perception. Science 268, Revisited. Ithaca, NY: Cornell Uni-
24, 8662–8671. Feltz, D. L., and Landers, D. M. (2007). 1772–1774. versity.
Chen, W., Kato, T., Zhu, X. H., Ogawa, S., The Effects of Mental Practice on Ishai, A., and Sagi, D. (1997). Visual Lau, H. C. (2008). A higher order
Tank, D. W., and Ugurbil, K. (1998). Motor Skill Learning and Perfor- imagery: effects of short-and long- Bayesian decision theory of con-
Human primary visual cortex and mance: A Meta-analysis. Champaign, term memory. J. Cogn. Neurosci. 9, sciousness. Prog. Brain Res. 168,
lateral geniculate nucleus activation IL: Human Kinetics. 734–742. 35–48.
during visual imagery. Neuroreport Fine, I., and Jacobs, R. A. (2002). Com- Kamphuisen, A., Van Wezel, R., and Lau, H. C., and Passingham, R. E.
9, 3669–3674. paring perceptual learning tasks: a Van Ee, R. (2007). Inter-ocular trans- (2006). Relative blindsight in normal
Chong, S., and Blake, R. (2006). Exoge- review. J. Vis. 2, 190–203. fer of stimulus cueing in dom- observers and the neural correlate
nous attention and endogenous Flavell, J. H. (1979). Metacognition and inance selection at the onset of of visual consciousness. Proc. Natl.
attention influence initial domi- cognitive monitoring: a new area binocular rivalry. Vision Res. 47, Acad. Sci. U.S.A. 103, 18763–18768.
nance in binocular rivalry. Vision of cognitive-developmental inquiry. 1142–1144. Macmillan, N. A., and Creelman, C. D.
Res. 46, 1794–1803. Am. Psychol. 34, 906–911. Karni, A. (1996). The acquisition of per- (1991). Detection Theory: A User’s
Craver-Lemley, C., and Reeves, A. Fleming, S. M., Weil, R. S., Nagy, Z., ceptual and motor skills: a mem- Guide. Cambridge: Cambridge Uni-
(1992). How visual imagery inter- Dolan, R. J., and Rees, G. (2010). ory system in the adult human cor- versity Press.
feres with vision. Psychol. Rev. 99, Relating introspective accuracy to tex. Brain Res. Cogn. Brain Res. 5, Marks, D. F. (1973). Visual imagery dif-
633–649. individual differences in brain struc- 39–48. ferences in the recall of pictures. Br.
Cui, X., Jeter, C. B., Yang, D., Mon- ture. Science 329, 1541–1543. Keogh, R., and Pearson, J. (2011). J. Psychol. 64, 17–24.
tague, P. R., and Eagleman, D. M. Galton, F. (1883). Inquiries into Human Mental imagery and visual work- McDermott, K. B., and Roediger, H.
(2007). Vividness of mental imagery: Faculty and its Development. Lon- ing memory. PLoS ONE 6, e29221. L. (1994). Effects of imagery on
individual variability can be mea- don: Macmillan. doi:10.1371/journal.pone.0029221 perceptual implicit memory tests. J.
sured objectively. Vision Res. 47, Galvin, S. J., Podd, J. V., Drga, V., and Kepecs, A., Uchida, N., Zariwala, H. A., Exp. Psychol. Learn Mem. Cogn. 20,
474–478. Whitmore, J. (2003). Type 2 tasks and Mainen, Z. F. (2008). Neural 1379–1390.

McKellar, P. (1965). The Investigation Rounis, E., Maniscalco, B., Rothwell, Slotnick, S. D. (2008). Imagery: mental Willingham, D. B. (1999). Implicit
of Mental Images. Harmondsworth: J. C., Passingham, R. E., and Lau, pictures disrupt perceptual rivalry. motor sequence learning is not
Penguin Books. H. (2010). Theta-burst transcranial Curr. Biol. 18, R603–R605. purely perceptual. Mem. Cognit. 27,
Meng, M., and Tong, F. (2004). Can magnetic stimulation to the pre- Slotnick, S. D., Thompson, W. L., 561–572.
attention selectively bias bistable frontal cortex impairs metacognitive and Kosslyn, S. M. (2005). Visual Zatorre, R. J., and Halpern, A. R. (2005).
perception? Differences between visual awareness. Cogn. Neurosci. 1, mental imagery induces retino- Mental concerts: musical imagery
binocular rivalry and ambiguous 165–175. topically organized activation of and auditory cortex. Neuron 47,
figures. J. Vis. 4, 539–551. Sacks, O. (2010). The Mind’s Eye. New early visual areas. Cereb. Cortex 15, 9–12.
Mitchell, J., Stoner, G., and Reynolds, York: Alfred A. Knopf. 1570–1583. Zhang, J.-Y., Kuai, S.-G., Xiao, L.-
J. (2004). Object-based attention Sasaki, Y., Nanez, J. E., and Watanabe, T. Song, C., Kanai, R., Fleming, S. M., Weil, Q., Klein, S. A., Levi, D. M., and
determines dominance in binocular (2010). Advances in visual percep- R. S., Schwarzkopf, D. S., and Rees, Yu, C. (2008). Stimulus coding
rivalry. Nature 429, 410–413. tual learning and plasticity. Nat. Rev. G. (2011). Relating inter-individual rules for perceptual learning. PLoS
O’Craven, K., and Kanwisher, N. (2000). Neurosci. 11, 53–60. differences in metacognitive perfor- Biol. 6, e197. doi:10.1371/journal.
Mental imagery of faces and places Seitz, A., and Watanabe, T. (2005). A mance on different perceptual tasks. pbio.0060197
activates corresponding stimulus- unified model for perceptual learn- Conscious. Cogn. 20, 1787–1792.
specific brain regions. J. Cogn. Neu- ing. Trends Cogn. Sci. (Regul. Ed.) 9, Stokes, M., Thompson, R., Cusack, R., Conflict of Interest Statement: The
rosci. 12, 1013–1023. 329–334. and Duncan, J. (2009). Top-down authors declare that the research was
Pearson, J., and Brascamp, J. (2008). Seitz, A. R., Kim, D., and Watanabe, activation of shape-specific popu- conducted in the absence of any com-
Sensory memory for ambiguous T. (2009). Rewards evoke learning lation codes in visual cortex dur- mercial or financial relationships that
vision. Trends Cogn. Sci. (Regul. Ed.) of unconsciously processed visual ing mental imagery. J. Neurosci. 29, could be construed as a potential con-
12, 334–341. stimuli in adult humans. Neuron 61, 1565–1572. flict of interest.
Pearson, J., Clifford, C., and Tong, F. 700–707. Swets, J. A. (1986). Form of empirical
(2008). The functional impact Seitz, A. R., and Watanabe, T. (2003). ROCs in discrimination and diag- Received: 27 April 2012; accepted: 16 June
of mental imagery on con- Psychophysics: is subliminal learn- nostic tasks: implications for theory 2012; published online: 10 July 2012.
scious perception. Curr. Biol. 18, ing really passive? Nature 422, 36. and measurement of performance. Citation: Rademaker RL and Pearson J
982–986. Seitz, A. R., Yamagishi, N., Werner, Psychol. Bull. 99, 181–198. (2012) Training visual imagery: improve-
Pearson, J., Rademaker, R. L., and Tong, B., Goda, N., Kawato, M., and Tartaglia, E. M., Bamert, L., Mast, F. ments of metacognition, but not imagery
F. (2011). Evaluating the mind’s eye: Watanabe, T. (2005). Task-specific W., and Herzog, M. H. (2009). strength. Front. Psychology 3:224. doi:
the metacognition of visual imagery. disruption of perceptual learning. Human perceptual learning by 10.3389/fpsyg.2012.00224
Psychol. Sci. 22, 1535–1542. Proc. Natl. Acad. Sci. U.S.A. 102, mental imagery. Curr. Biol. 19, This article was submitted to Frontiers in
Pelli, D. G. (1997). The VideoToolbox 14895–14900. 2081–2085. Perception Science, a specialty of Frontiers
software for visual psychophysics: Serences, J. T., Ester, E. F., Vogel, E. Vickers, D. (1979). Decision Processes in Psychology.
transforming numbers into movies. K., and Awh, E. (2009). Stimulus- in Visual Perception. New York, NY: Copyright © 2012 Rademaker and Pear-
Spat. Vis. 10, 437–442. specific delay activity in human pri- Academic Press. son. This is an open-access article distrib-
Perky, C. (1910). An experimental study mary visual cortex. Psychol. Sci. 20, Weiss, T., Hansen, E., Rost, R., Beyer, uted under the terms of the Creative Com-
of imagination?. Am. J. Psychol. 21, 207–214. L., Merten, F., Nichelmann, C., and mons Attribution License, which per-
422–452. Shibata, K., Watanabe, T., Sasaki, Y., and Zippel, C. (1994). Mental practice of mits use, distribution and reproduction
Pylyshyn, Z. (2003). Return of the men- Kawato, M. (2011). Perceptual learn- motor skills used in poststroke reha- in other forums, provided the original
tal image: are there really pictures in ing incepted by decoded fMRI neu- bilitation has own effects on central authors and source are credited and sub-
the brain? Trends Cogn. Sci. (Regul. rofeedback without stimulus presen- nervous activation. Int. J. Neurosci. ject to any copyright notices concerning
Ed.) 7, 113–118. tation. Science 334, 1413–1415. 78, 157–166. any third-party graphics etc.

REVIEW ARTICLE
published: 02 October 2012
doi: 10.3389/fpsyg.2012.00360
New percepts via mental imagery?

Fred W. Mast 1,2 *, Elisa M. Tartaglia 3,4 and Michael H. Herzog 3
1
Department of Psychology, University of Bern, Bern, Switzerland
2
Center for Cognition, Learning and Memory, University of Bern, Bern, Switzerland
3
Laboratory of Psychophysics, Brain and Mind Institute, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
4
Laboratory of Neurophysics and Physiology, UMR 8119 CNRS, Université René Descartes, Paris, France
Edited by: We are able to extract detailed information from mental images that we were not explicitly
aware of during encoding. For example, we can discover a new figure when we rotate a
previously seen image in our mind. However, such discoveries are not “really” new but
Reviewed by:
Duje Tadin, University of Rochester, just new “interpretations.” In two recent publications, we have shown that mental imagery
USA can lead to perceptual learning (Tartaglia et al., 2009, 2012). Observers imagined the cen-
Mark M. Schira, University of New tral line of a bisection stimulus for thousands of trials. This training enabled observers to
perceive bisection offsets that were invisible before training. Hence, it seems that per-
*Correspondence:
ceptual learning via mental imagery leads to new percepts. We will argue, however, that
Fred W. Mast , Department of
Psychology, University of Bern, these new percepts can occur only within “known” models. In this sense, perceptual
Muesmattstrasse 45, Bern CH-3012, learning via mental imagery exceeds new discoveries in mental images. Still, the effects
Switzerland. of mental imagery on perceptual learning are limited. Only perception can lead to really
e-mail: fred.mast@psy.unibe.ch
new perceptual experience.
Keywords: ambiguous figure, bisection, combinatorics, interpolation, mental rotation, top-down processing,
perceptual learning
Imagination is more important than knowledge. For knowledge is mental images. For example, when participants indicate whether a
limited, whereas imagination embraces the entire world, stimulating previously seen object is higher than wide (e.g., a picture of a sail-
progress, giving birth to evolution. boat) they are able to extract this information from mental images
(Albert Einstein, 1929) even though they did not attend to the spatial dimensions when
they encoded the visual stimulus (Kosslyn et al., 1995). It has been
MENTAL IMAGERY argued that participants learned the stimuli beforehand, and that
Mental imagery is usually referred to as seeing with the mind’s tacit knowledge about the purpose of the experiment could have
eye. We are able to create images in our mind despite the absence led to the findings. Finke and Pinker (1982) showed participants
of appropriate sensory stimulation. Mental images are usually an array of dots. After the dots disappeared, an arrow was pre-
described introspectively, using a sensory terminology such as sented and participants decided whether or not the arrow pointed
form, color, spatial extent, and so forth. Even though introspec- to one of the dots they just saw. Response times increased linearly
tion does not necessarily allow for drawing conclusions about the with increasing distance between the arrow and the target dot. It
underlying mechanisms recent research has shown that people has been proposed that attentional crowding could account for the
have a good metacognitive understanding of the vividness of their distance effect, leading to more difficult discriminations for fur-
mental images (Pearson and Tong, 2011). ther distances (Pylyshyn, 2002). However, this possible explanation
The mechanisms that underlie mental imagery have become the has been ruled out by Dror and Kosslyn (1993) who conducted a
topic of the “Imagery Debate” (see for example Kosslyn et al., 2003; modified version of Finke and Pinker’s experiment. They replaced
Pylyshyn, 2003). Cognitive psychologists have carried out a large dots by black (3) and white (17) squares which were arranged in
amount of behavioral experiments to tap into the mechanisms a square-like configuration (six squares on each side). The task
that underlie mental imagery, and the results provide compelling remained the same as in Finke and Pinker’s experiment but the
evidence that mental imagery shares common mechanisms with distance between the black squares was chosen so that it exceeded
visual perception. For example, it has been shown that men- the distance known to produce attentional crowding (Intriliga-
tal images preserve spatial distances. In experiments on image tor and Cavanagh, 2001). As expected, Dror and Kosslyn (1993)
scanning, participants had to first memorize visual information demonstrated that response times increased with increasing dis-
shown in a picture (e.g., a map of an island) and later answer tance between the arrow and the square. Hence, their results
questions about landmarks (Kosslyn et al., 1978). Interestingly, confirmed that mental images embody metric properties even
response times were proportional to the actual distances between when no spatial relationships are encoded explicitly during the
the landmarks. It took participants more time to scan longer dis- presentation of the stimulus. Yet other evidence comes from neu-
tances. Hence, participants were able to extract metric distance roimaging studies showing a large overlap in activated areas during
information from memory. Yet other experiments have shown mental imagery and visual perception, including early visual cor-
that participants are well able to judge and compare distances in tex (e.g., Slotnick et al., 2005). Activation in the latter has been
www.frontiersin.org October 2012 | Volume 3 | Article 360 | 34

Mast et al. Perceptual learning via mental imagery
found when participants were engaged in mental imagery tasks imagery. Perceptual learning without perceptual input opens
that required the extraction of high-resolution visual information. new ways of using mental imagery in learning paradigms, thus
In addition, TMS over the occipital cortex disrupted performance challenging classical views on perceptual learning. The fact that
in those tasks (Kosslyn et al., 1999). Thus, there is compelling evi- mental imagery and visual perception share – at least in part –
dence for commonalities between mental imagery and perception. the same mechanisms extends to learning and memory. We
However, it has to be pointed out that there are also differences, gathered evidence that learning in imagery transfers to percep-
and, for example, only rarely do we confuse images with percepts. tual performance (Tartaglia et al., 2009, 2012). The next para-
The discovery of new information is yet another conceivable dif- graph will provide an overview of perceptual learning, and focus
ference between mental imagery and perception. When we inspect on the implications that learning via mental imagery has on
with our eyes a visual scene new information is picked up con- the current understanding of mental imagery and perception
tinuously. We discover more and more details that are part of the alike.
visual scene. It has been argued that new discoveries cannot be
made in mental imagery (Chambers and Reisberg, 1985, 1992). PERCEPTUAL LEARNING
A mental image contains nothing new besides the information Perceptual learning is learning to perceive. Sommeliers can taste
the observer is aware of when generating the image. Ambiguous not only the grape of a wine but often the year of its production
figures, for example, demonstrate compellingly that the percep- and vineyard. Years of training make the master. In the laboratory,
tual interpretation of the same visual stimulus can switch from training improves basic visual skills including the discrimination
one explanation to another. Observers usually do not notice the of motion direction (Ball and Sekuler, 1987), the detection of
alternative perceptual explanation when first viewing the ambigu- Gabor gratings (Sowden et al., 2002), stereoacuity (Fendick and
ous figure. Obviously, new discoveries occur in perception. What Westheimer, 1983; Kumar and Glaser, 1993), line orientation judg-
about mental images? Are people able to discover new information ments (Vogels and Orban, 1985), texture discrimination (Karni
in mental images? Mast and Kosslyn (2002) used an ambiguous and Sagi, 1991; Pourtois et al., 2008) and the discrimination of
figure in which the alternative perceptual explanation was upside fine spatial features, such as in vernier acuity (Fahle and Edelman,
down (young lady, old lady). It was almost impossible to discover 1993). Another example are bisection stimuli where a central line
the second explanation (e.g., young lady upside down) acciden- is closer to one of two outer lines (Figure 1). Observers indicate the
tally during encoding. However, roughly 50% of participants were offset direction (to the left or to the right). Training can strongly
able to discover in their mental image a new upright interpre- improve offset discrimination (Figure 1).
tation after having mentally rotated the image by 180˚. None of Without phylogenetic or ontogenetic perceptual learning, there
the participants was aware about the alternative explanation when is no perception (Herzog and Esfeld, 2009). Kittens that were raised
they learned the image (in one orientation only). Hence, observers in an environment missing horizontal lines during a critical period
discovered a new interpretation that they were not aware of when were blind for the horizontal orientation for the rest of their entire
encoding the image. Thus, the ability to extract new interpretations life (Blakemore and Cooper, 1970; Hirsh and Spinelli, 1970). In
is not bound to the process of perception as it can occur just as this respect, perceptual learning is often thought to be the most
well during mental imagery. The ability to discover new interpre- basic type of learning.
tations in images is not the crucial difference between imagery and The mechanisms that underlie perceptual learning are contro-
perception. Mental images are not interpreted entities as claimed versially debated. However, there is clear consensus that perceptual
by Fodor (1975) and they are not tied to their initial interpretation learning is driven by the repeated exposure to the stimuli. Only
assigned during image generation. Indeed, participants were not repeated wine tastings make a good sommelier. On the neural
aware of the second interpretation in the image even though all level, the presentation of a stimulus leads to changes of synaptic
the information was available in the stimulus they encoded. Hence, weights which may, for example, fine tune receptive fields (e.g.,
only a new interpretation but no new information was discovered. Fahle, 2005) or enhance gating to decision stages (e.g., Herzog
Here is another example. Imagine a letter “D” and rotate it coun- and Fahle, 1998; Lu and Dosher, 2004). Top-down aspects, such
terclockwise by 90˚, and then take the letter “J” and attach it to the as attention, may be crucial in perceptual learning, but they are
rotated “D” so that the upper end of the “J” is attached exactly at only modifying, but not driving the synaptic changes. For exam-
the middle of the horizontal line in the rotated “D.” What does ple, attention selects aspects of the stimuli to be learned, but the
it look like? An umbrella. The umbrella does not make part of its learning itself is driven by the stimuli.
individual constituents (D, J) but still people are able to extract the In all mathematical models of perceptual learning, there is a
shape from the configuration created in imagery. Indeed, a major stimulus related term essential for learning. In unsupervised learn-
strength of mental imagery is the ability to recombine informa- ing algorithms, as for example in Hebbian learning, the synaptic
tion, thus, going beyond perceptual experience. Therefore, mental changes ∆w j depend only on the concurrent activation of neurons:
imagery serves an important function in creative thinking. How-
µ
ever, the detection of emergent new shapes in imagery still remains ∆wj = η · y µ xj
in the realm of potential interpretations.
This article focuses on the relation between mental imagery µ
where xj is the activity of the j-th input neuron in response to
and perceptual learning. At first glance, these topics appear some- µ
what unrelated but we were able to demonstrate in a series stimulus µ and yj is the activity of the output neuron in response
µ
of experiments the existence of perceptual learning via mental to the same stimulus (η is the learning rate); yj is the weighted
Frontiers in Psychology | Perception Science October 2012 | Volume 3 | Article 360 | 35

A B C Pre-training
90 Post-training
100
vs 90 80
% co rre ct
% co rre ct
80
70
70
60
60
50 50
1 6 11 16 21 26 31 36 41 46 51
N um ber of B loc ks
FIGURE 1 | (A) In line bisection, a spatial interval, delineated by two outer discriminate offset sizes indiscriminable before training. (C) Another way of
lines, is bisected in two unequal components by a central line. The task of the measuring improvements of performance is to compare observers’
observer is to judge whether the central line is closer to the left or right outer performance before (“pre-training”) and after (“post-training”) the training
line. Observers are asked to give their response by using one of two buttons. phase. Post-training baselines are significantly higher than pre-training
(B) Performance improves with training, i.e., observers become able to baselines.
µ µ
sum of the input yj = Σwj *xj and hence fully determined by
the actual input and the synaptic weights w j . If no stimuli are A
vs
presented xj = 0, for all j, i.e., there is no activation of the input
layer and hence no learning since ∆wj = 0, for all j. Hence, no
perceptual learning is expected in the absence of stimulus pre- B
sentation. Similar considerations hold for other types of neural
networks of perceptual learning.
However, contrary to all previous thinking in perceptual learn-
ing research, our recent experiments showed that perceptual learn-
ing can occur in the absence of (proper) stimulus presentation
when participants imagine the missing perceptual information
C
(Tartaglia et al., 2009, 2012). For example, we used a modified
bisection stimulus in which only the outer lines were presented vs
during training (Figure 2). Observers were asked to imagine the
center line to be offset either to the left or right depending on
a tone provided in addition (a high tone was associated to the
D Pre-training
right offset, a low tone to the left one). Hence, the physical stim- 90 Post-training
ulus was always the same in all trials. There was “nothing” to
%correct
80
learn from the stimulus itself. Still, performance improved dur-
ing mental imagery, as determined by pre- and post-training 70
measurements in which the proper bisection stimulus was pre- 60
sented (Figure 2). Hence, mental imagery is sufficient to enable 50
perceptual learning.
The improvements during training are not caused by the pre-
and post-measurements with the full bisection stimulus because
measurements without training did not yield any performance
FIGURE 2 | (A,C) Pre- and Post-training baseline measurements with the
gains (see Tartaglia et al., 2009). But is it really mental imagery that
“proper” vertical bisection stimulus. (B) Imagery training. Only the two
improves performance in perceptual learning? Unspecific effects outer lines of the bisection stimulus are presented. Observers are asked to
such as better coping with attention and improved decision mak- imagine the absent central line to be offset either to the left or to the right,
ing need to be controlled for. We performed a control experiment depending on a tone, and to push buttons accordingly. We asked observers
with (almost) the same stimuli but without imagery during train- to imagine the smallest possible offset; the imagined line is indicated by
the dashed line (which was not shown in the actual display). (D)
ing. During the training phase, again only the two vertical outer Performance improved significantly through training. The error bars indicate
lines of the bisection were presented, together with the two dif- the standard error of the mean (SEM) for nine observers. Adapted from
ferent tones. Observers had to press the right button when a Tartaglia et al. (2009).
high-frequency tone was presented and the left button when a low-
frequency tone was presented. We did not ask observers to imagine
the central line of the bisection stimulus. To ensure observers’ end of each block, observers were required to report the number
attention to the stimuli, the outer lines were only half as long as of trials with shorter lines. The physical stimulation was identical
normal in a few trials (0, 1, or 2 within a block of 80 trials). At the to the imagery experiment (except for the 0–2 lines deviating per

A A B C
vs
B
or
or
C
FIGURE 4 | The dot world. (A) Three dots are presented that appear to be
vs aligned. (B) After extended training, observers can easily discriminate the
offset direction of the center dot. Perceptual learning has made spatial
differences visible that were not visible before. (C) Nine dots are presented.

D Pre-training In this display, 93 = 84 tasks with positional changes of three dots can be
defined (the other six dots constitute the background). In general, 29 tasks
Post-training can be defined for 9 dots. In displays with n dots, hence, 2n tasks can be
90 defined leading to 2n objects that one can potentially learn to perceive.
% correct
k = k (n−k )! with n! = 1*2*3*. . . *n. From Herzog and Esfeld (2009).

n
n!
70
50 COMBINATORICS
In Figures 4A,B a 3-dot vernier is shown for which training
improves performance (Fahle and Morgan, 1996). In Figure 4C,
a set of nine dots is shown for which 84 3-dot vernier tasks can
FIGURE 3 | (A,C) Pre and Post-training baseline measurements with the be defined and trained. Because of the specificity of perceptual
proper vertical bisection stimulus. (B) Counting lines. Only the two outer
lines were presented. Observers were asked to press the right button
learning, we expect no or very little transfer between tasks. Also 2
when a high-frequency tone was presented and the left button for a or 8-dot vernier tasks can be defined. In general, 256 vernier tasks
low-frequency tone. In addition, observers were asked to count the number can be defined by the 9-dot display. For 52 dots, there are more
of trials with shorter lines (0, 1, or 2 out of 80 trials per block). No imagery tasks than all the milliseconds in the universe that passed by since
was involved in this experiment. (D) No significant improvement of big bang. Hence, very small sets of elements can create very large
performance occurred through training for the vertical bisection. The error
bars indicate the standard error of the mean (SEM) for six observers.
combinatorial spaces.
Adapted from Tartaglia et al. (2009). We propose that mental imagery operates within such combi-
natorial spaces. The landmarks of the island that were memorized
in the Kosslyn et al. (1978) experiment span a combinatorial space.
block; Figure 3). Performance did not improve in this condition Imagery does not add any new basic elements (e.g., a light tower)
(Figure 3). Hence, it is the mental imagery training that leads to but computes, for example, second order aspects such as path
perceptual learning. Further experiments demonstrated percep- lengths. In this sense, imagery does not create anything new. It just
tual learning via mental imagery for Gabor and motion stimuli makes implicit aspects explicit. However, combinatorial spaces are
(Tartaglia et al., 2009, 2012). large, often virtually infinite. Hence, the majority of combinato-
Interestingly, perceptual learning via motion imagery improves rial facts will remain unknown because only a small margin can
coherent motion discrimination of moving dots when a blank be computed. In this sense, imagery produces new knowledge that
screen is presented. However, no improvement occurs when, may be unique to a certain person at a given moment in time.
instead of a blank screen, randomly moving dots are presented On the first glance, it may appear as if perceptual learning via
during training (Tartaglia et al., 2012). Contrary to motion stim- mental imagery can lead to really new percepts, as in the bisection
uli perceptual learning in the bisection task requires the presence stimulus where participants learn to perceive offsets that were not
of the two outer lines during training. Without the lines, perfor- visible before. However, as mentioned above, without perceptual
mance does not improve. In mental imagery studies with static assistance provided by the outer lines, mental imagery training
stimuli, it is often the case that a perceptual reference is needed did not lead to perceptual learning. It seems that perceptual learn-
(see the role of perceptual assistance shown in other studies, ing via mental imagery occurs within a model consisting of the
e.g., Intons-Peterson, 1981; Mast et al., 1999). Future research two outer lines and the metric space in between. It is possible
will better define the conditions under which mental imagery that the brain learns to better read out information from cod-
training will unfold its impact on perceptual learning. Taken ing the center line. Hence, perceptual learning via mental imagery
together, it is possible to learn to see things that were not vis- creates new percepts. However, these basic percepts are not so
ible before imagery training. Thus, is perceptual learning via much different from detecting a new face in a mental image that
mental imagery truly creating new percepts? We will argue: yes was not visible during encoding. Still, all these percepts are new
and no. in the virtually infinite combinatorial space of things that can
Frontiers in Psychology | Perception Science October 2012 | Volume 3 | Article 360 | 37

be learned and perceived. Once we have made the perceptual imagery relying on a generating model combining and changing
experience new knowledge can emerge from the combinatorial past percepts. This would allow for creating infinite numbers of
space. Really new percepts, however, are based on experience and new percepts within vast combinatorial spaces. We suggest that
they cannot be traced back to previous visual percepts. At this only perception can generate really new percepts in the sense of
point, however, we enter the realm of philosophy and, admit- model free extrapolation.
tedly, the definition of what a new percept is remains a debatable
issue. ACKNOWLEDGMENTS
This research is supported by SNF grants awarded to Fred
CONCLUSION W. Mast (135303), Michael H. Herzog (114404), and Elisa M.
Mental imagery can create new percepts via perceptual learning. Tartaglia (PBELP3-135838). We thank the reviewers for helpful
The examples we summarized here can be explained by mental suggestions.
REFERENCES Fodor, J. A. (1975). The Language of Kosslyn, S. M., Thompson, W. L., Kim, of luminance contrast detection:
Ball, K., and Sekuler, R. (1987). Thought. New York: Crowell. I. J., and Alpert, N. M. (1995). Topo- specific for spatial frequency and
Direction-specific improvement in Herzog, M. H., and Esfeld, M. (2009). graphical representations of mental retinal location but not orientation.
motion discrimination. Vision Res. How the mind constitutes itself images in primary visual cortex. Vision Res. 42, 1249–1258.
27, 953–965. through perceptual learning. Learn. Nature 378, 496–498. Tartaglia, E. M., Bamert, L., Herzog, M.
Blakemore, C., and Cooper, G. F. (1970). Percept. 1, 147–154. Kumar, T., and Glaser, D. (1993). Initial H., and Mast, F. W. (2012). Percep-
Visual experience modifies distrib- Herzog, M. H., and Fahle, M. (1998). performance, learning, and observer tual learning of motion discrimina-
ution of horizontally and vertically Modeling perceptual learning: diffi- variability for hyperacuity tasks. tion by mental imagery. J. Vis. 12,
oriented receptive fields in cats. Sci- culties and how they can be over- Vision Res. 33, 2287–2300. 1–10.
ence 168, 869–871. come. Biol. Cybern. 78, 107–117. Lu, Z. L., and Dosher, B. A. (2004). Per- Tartaglia, E. M., Bamert, L., Mast, F.
Chambers, D., and Reisberg, D. (1985). Hirsh, H. V. B., and Spinelli, D. N. ceptual learning retunes the percep- W., and Herzog, M. H. (2009).
Can mental images be ambiguous? J. (1970). Development of the brain tual template in foveal orientation Human perceptual learning by
Exp. Psychol. Hum. Percept. Perform. depends on the visual environment. identification. J. Vis. 4, 44–56. mental imagery. Curr. Biol. 19,
11, 317–328. Nature 228, 477–478. Mast, F., Kosslyn, S. M., and Berthoz, 2081–2085.
Chambers, D., and Reisberg, D. (1992). Intons-Peterson, M. J. (1981). Con- A. (1999). Visual mental imagery Vogels, R., and Orban, G. A. (1985).
What an image depicts depends on structing and using unusual and interferes with allocentric orienta- The effect of practice on the
what an image means. Cogn. Psychol. common images. J. Exp. Psychol. tion judgements. Neuroreport 10, oblique effect in line orienta-
24, 145–174. Hum. Learn. 7, 133–144. 3549–3553. tion judgements. Vision Res. 25,
Dror, I., and Kosslyn, S. M. (1993). Men- Intriligator, J., and Cavanagh, P. Mast, F. W., and Kosslyn, S. M. 1679–1687.
tal imagery and aging. Psychol. Aging (2001). The spatial resolution of (2002). Visual mental images can be
9, 90–102. visual attention. Cogn. Psychol. 43, ambiguous: insights from individual Conflict of Interest Statement: The
Einstein, A. (1929). What Life Means to 171–216. differences in spatial transformation authors declare that the research was
Einstein. Saturday Evening Post, 26, Karni, A., and Sagi, D. (1991). Where abilities. Cognition 86, 57–70. conducted in the absence of any com-
October. practice makes perfect in texture dis- Pearson, J., and Tong, F. (2011). Evalu- mercial or financial relationships that
Fahle, M. (2005). Perceptual learn- crimination: evidence for primary ating the mind’s eye: the metacogni- could be construed as a potential con-
ing: specificity versus generalization. visual cortex plasticity. Proc. Natl. tion of visual imagery. Psychol. Sci. flict of interest.
Curr. Opin. Neurobiol. 15, 154–160. Acad. Sci. U.S.A. 88, 4966–4970. 22, 1535–1542.
Fahle, M., and Edelman, S. (1993). Kosslyn, S. M., Ball, T. M., and Reiser, Pourtois, G., Rauss, K. S., Vuilleumier, Received: 01 May 2012; accepted: 04 Sep-
Long-term learning in vernier acu- B. J. (1978). Visual images preserve P., and Schwartz, S. (2008). Effects tember 2012; published online: 02 Octo-
ity: effects of stimulus orientation, metric spatial information: evidence of perceptual learning on primary ber 2012.
range and of feedback. Vision Res. 33, from studies of image scanning. J. visual cortex activity in humans. Citation: Mast FW, Tartaglia EM and
397–412. Exp. Psychol. Hum. Percept. Perform. Vision Res. 48, 55–62. Herzog MH (2012) New percepts via
Fahle, M., and Morgan, M. (1996). 4, 47–60. Pylyshyn, Z. W. (2002). Mental imagery: mental imagery? Front. Psychology 3:360.
No transfer of perceptual learn- Kosslyn, S. M., Ganis, G., and Thomp- in search of a theory. Behav. Brain doi: 10.3389/fpsyg.2012.00360
ing between similar stimuli in the son, W. L. (2003). Mental imagery: Sci. 25, 157–238. This article was submitted to Frontiers in
same retinal position. Curr. Biol. 6, against the nihilistic hypothesis. Pylyshyn, Z. W. (2003). Return of the Perception Science, a specialty of Frontiers
292–297. Trends Cogn. Sci. (Regul. Ed.) 7, mental image: are there pictures in Psychology.
Fendick, A., and Westheimer, G. (1983). 109–111. in the brain? Trends Cogn. Sci. 7, Copyright © 2012 Mast , Tartaglia and
Effect of practice and the separa- Kosslyn, S. M., Pascual-Leone, A., Feli- 113–118. Herzog . This is an open-access article dis-
tion of test targets on foveal and cian, O., Camposano, S., Keenan, J. Slotnick, S. D., Thompson, W. L., and tributed under the terms of the Creative
peripheral stereoacuity. Vision Res. P., and Thompson, W. L., Ganis, Kosslyn, S. M. (2005). Visual men- Commons Attribution License, which
23, 145–150. G., Sukel, K. E., and Alpert, N. tal imagery induces retinotopically permits use, distribution and reproduc-
Finke, R. A., and Pinker, S. (1982). Spon- M. (1999). The role of area 17 in organized activation of early visual tion in other forums, provided the original
taneous imagery scanning in mental visual imagery: convergent evidence areas. Cereb. Cortex 1570–1583. authors and source are credited and sub-
extrapolation. J. Exp. Psychol. Learn from PET and rTMS. Science 284, Sowden, P. T., Rose, D., and Davies, ject to any copyright notices concerning
Mem. Cogn. 8, 142–147. 167–170. I. R. (2002). Perceptual learning any third-party graphics etc.

REVIEW ARTICLE
published: 01 April 2013
HUMAN NEUROSCIENCE doi: 10.3389/fnhum.2013.00104
An emerging paradigm: a strength-based approach to

exploring mental imagery
Tadhg E. MacIntyre 1*, Aidan P. Moran 2 , Christian Collet 3 and Aymeric Guillot 3
1
Department of Physical Education and Sport Sciences, University of Limerick, Limerick, Ireland
2
School of Psychology, University College Dublin, Ireland
3
Centre de Recherche et d’Innovation sur le Sport, Université Claude Bernard Lyon 1, France
Edited by: Mental imagery, or the ability to simulate in the mind information that is not currently
Joel Pearson, The University of New perceived by the senses, has attracted considerable research interest in psychology
since the early 1970’s. Within the past two decades, research in this field—as in
Reviewed by:
cognitive psychology more generally—has been dominated by neuroscientific methods
South Wales, Australia that typically involve comparisons between imagery performance of participants from
Julie Krans, The University of New clinical populations with those who exhibit apparently normal cognitive functioning.
South Wales, Australia Although this approach has been valuable in identifying key neural substrates of visual
*Correspondence: imagery, it has been less successful in understanding the possible mechanisms underlying
Tadhg E. MacIntyre, Department of
another simulation process, namely, motor imagery or the mental rehearsal of actions
Physical Education and Sport
Sciences, University of Limerick, without engaging in the actual movements involved. In order to address this oversight, a
Limerick, Ireland. “strength-based” approach has been postulated which is concerned with understanding
e-mail: tadhg.macintyre@ul.ie those on the high ability end of the imagery performance spectrum. Guided by the
expert performance approach and principles of ecological validity, converging methods
have the potential to enable imagery researchers to investigate the neural “signature”
of elite performers, for example. Therefore, the purpose of this paper is to explain
the origin, nature, and implications of the strength-based approach to mental imagery.
Following a brief explanation of the background to this latter approach, we highlight
some important theoretical advances yielded by recent research on mental practice,
mental travel, and meta-imagery processes in expert athletes and dancers. Next, we
consider the methodological implications of using a strength-based approach to investigate
imagery processes. The implications for the field of motor cognition are outlined and
specific research questions, in dynamic imagery, imagery perspective, measurement,
multi-sensory imagery, and metacognition that may benefit from this approach in the
future are sketched briefly.
Keywords: expertise, mental imagery, metacognition, motor cogniton, converging methods, mental practice,
mental travel, mental rotation
INTRODUCTION array of disciplines for four decades? The answer to that question,
Since the classic study on mental rotations by Shepard and as we shall see, is perhaps as complex as the construct of mental
Metzler (1971) four decades ago, research on mental imagery, imagery itself.
or “an internal representation that gives rise to the experience Imagery, due to its ephemeral nature, has provided significant
of perception in the absence of the appropriate sensory input” challenges for psychology since it first received formal research
(Wraga and Kosslyn, 2002, p. 466), has flourished in cognitive attention over a century ago (e.g., Galton, 1883). Chief among
psychology (e.g., Paivio et al., 1968), cognitive neuropsychology these challenges were the empirical question of how best to
(e.g., Farah, 2000a), cognitive neuroscience (e.g., Kosslyn et al., measure imagery given the limitations of the introspective meth-
2006), and more recently, in the study of motor cognition (e.g., ods (Kosslyn, 1980) and the theoretical question of whether the
Jeannerod, 2001, 2006a). The common concern of researchers in mental representation underlying imagery is propositional or
these fields conveys the extent to which exploring mental imagery analogical in nature (Block, 1983). Kosslyn et al. (2006) have
has become part of the cognitive science zeitgeist (Cornoldi and provided a comprehensive proposal to resolve this latter issue.
De Beni, 2012). Furthermore, mental imagery has also been of Despite such challenges, scientists have made significant progress
interest to those concerned with its application in the domains of in capturing the intricacies of mental imagery processes and
skill acquisition (e.g., Sevdalis et al., 2013), rehabilitation sciences applications by using new approaches (e.g., neuroscientific meth-
(e.g., Malouin and Richards, 2013) and professional expertise ods such as fMRI; Slotnick et al., 2005). Building on this need for
(e.g., Kozhevnikov and Blajenkova, 2013). What is it about men- new paradigms, the purpose of the present paper is to explore the
tal imagery that has captured the interest of scientists across an possibility that a strength-based approach to imagery—one that
Frontiers in Human Neuroscience www.frontiersin.org April 2013 | Volume 7 | Article 104 | 39

MacIntyre et al. Strength-based approach to mental imagery
focuses explicitly on recruiting participants at the high-ability normative scores and even sophisticated statistical models are
end of the continuum of expertise—can augment conventional used to average date from neural imaging studies (Senior et al.,
approaches to this construct. We argue that a strength-based 2006). Individual differences have not been central to the men-
approach may be valuable in illuminating both fundamental tal imagery research program in recent decades although as we
and applied questions (e.g., do expert athletes and/or dancers shall see later in this paper, they provided an impetus for the
possess distinctive insights into meta-imagery processes in their strength-based approach. What is apparent from the aforemen-
domain?) which have so far evaded imagery researchers. In pur- tioned investigations is the paucity of research with participants
suit of this objective, the paper is organized as follows. In the at the high end of the continuum. Ability measures have been
first section, we shall review the strengths and weaknesses of con- used widely to assess handedness (e.g., lateralization invento-
ventional approaches to sampling that have dominated research ries) and they have been employed to discriminate good from
on mental imagery for over a century. Next, we shall present poor imagers (e.g., Isaac and Marks, 1994). Indeed, highly skilled
an alternative approach to mental imagery research that we call imagers may have been among the student samples in the clas-
the “strength-based” paradigm. The significant factors that led to sic imagery experiments of Kosslyn et al. (1978) and Shepard
the emergence of this paradigm will also be summarized. In the and Metzler (1971). The deficit-based approach which included
final section, we shall outline some potentially fruitful new direc- both patients and healthy subjects may have overlooked poten-
tions in imagery research that can be addressed by supplementing tial insights, as we will propose later, from those who are highly
traditional approaches with this strength-based paradigm. skilled on imagery ability measures.
Nevertheless, the success of the prevailing paradigm has been
KEY SAMPLES IN MENTAL IMAGERY RESEARCH the accumulation over four decades of substantive empirical
The study of imagery has profited greatly from neuroscientific evidence from case studies, behavioral data, and meta-analytic
methods (Behrmann, 2000). Given that it was a neglected topic reviews (Kosslyn et al., 2006). Furthermore, evidence has emerged
for almost 50 years, its return from obscurity is remarkable over the past two decades substantiating the neural basis of motor
(Kosslyn et al., 1995). Integral to the popularity of imagery as a imagery (Munzert and Zentgraf, 2009; Munzert et al., 2009),
legitimate scientific topic has been the increased strength in the olfactory imagery (Bensafi et al., 2003; Djordjevic et al., 2005),
inferences drawn from the research findings. These inferences are auditory imagery (Hubbard, 2010), gustatory imagery (Nitschke
based on two aspects of the research process, the methodological et al., 2006), spatial imagery (Kosslyn et al., 2007; Kozhevnikov
tools and the approach to sampling (Table 1). The latter of these and Blajenkova, 2013), and the role of emotions in imagery
will be discussed in light of the converging methods approach of (Kosslyn, 2010).
neuroscience (Kosslyn and Koenig, 1992). However, some limitations in the key methods of cognitive
Smith and Kosslyn (2007) describe the development of three neuroscience are also evident, and they may explain why our
main approaches in studying cognition: behavioral measures understanding of visual imagery is more comprehensive than
(e.g., latencies on computer-based tasks), correlational neural that of other modalities (Moran et al., 2012). The scanning tech-
methods (e.g., fMRI), and causal neural methods (e.g., neuropsy- nologies to date (e.g., fMRI) restrict movement and thus limit
chological studies). Typical of this latter approach are compar- maximal contractions during measurement. As a result, typically
isons between the performances of two groups on a task that only discrete movements can be performed (e.g., small finger or
requires imagery (i.e., behavioral data). The groups may comprise foot movements; Pascual-Leone et al., 1995; Brown et al., 2005).
those who exhibit normal cognitive functioning or participants Interestingly, transcranial magnetic stimulation (TMS) offers a
from clinical populations (Senior et al., 2006). Furthermore, the different approach to studying action-one that is far more flexible
methods are often combined (i.e., behavioral data from nor- as cortical functions can be disrupted even among normal con-
mal and patient samples during brain imaging). Studying the trols (Walsh and Cowey, 2000; Stewart and Walsh, 2006). These
deficits of patients with brain damage and impairments can methods may have obscured some aspects of imagery but we posit
establish that certain brain areas (e.g., by double dissociation) that in combination with the strength-based approach, they will
are at least partially responsible for a particular cognitive func- ultimately illuminate the construct further.
tion. Neuropsychological evidence of both deficits following brain
damage and brain activation patterns have been fundamental to ANTECEDENTS OF A “STRENGTH-BASED” APPROACH
the accumulating knowledge base of mental imagery (Kosslyn In many respects, the “strength-based” approach to understand-
et al., 2006). Robust inferences can obviously be determined from ing mental imagery is not a new phenomenon. In fact, we
monitoring changes in intact brain regions (Sarter et al., 1996), believe that it has three key antecedents—the individual dif-
particularly when combined with theory-driven experimental ferences approach to imagery (e.g., Galton, 1883), the idea of
paradigms (Kosslyn et al., 2006). Consequently, the neural-based imagery as a cognitive simulation process (e.g., Jeannerod, 1994),
studies with patients suffer from issues of generalizability but rely- and the emergence of a movement (known as “motor cognition”;
ing on converging evidence these limitations have largely been see Jeannerod, 2006a) which is committed to the investigation
overcome. of imagery-action relationships. As we shall explain below, we
From the above one could conclude that the cognitive neu- believe that these three antecedents have contributed certain
roscience approach has encompassed participants across the per- key ideas which come together in the strength-based approach
formance spectrum, from those with deficits to those with high (Table 2). For example, the individual differences tradition led to
abilities. The nomothetic method has ultimately emphasized the discovery that certain mental imagery processes are trainable.

Table 1 | Summary of traditional approaches to sampling in mental imagery research with human participants.
Samples Paradigm Method Goal Strengths Limitations Landmark study

MacIntyre et al.
Experimenters were Individual differences Introspection Understand Information appears to Highly subjective Perky (1910) study of visual
included in the sample approach phenomenology be easy to test imagery and perception
of healthy subjects
Healthy subjects Psychometric approach Questionnaires Focus on individual Subjective Marks (1973) Vividness of
Large no. of subjects differences and Potential response bias Visual Imagery
sub-scale performance (e.g., central tendency) Questionnaire (VVIQ)
Frontiers in Human Neuroscience

Healthy subjects and Experimental approach Response times (e.g., Normalized data Objective and easy to Trade-off between it took longer for subjects
typically right-handed Mental scanning task) test large N structure and process to consciously scan
(based on laterality Some criticism of between image features
tests) potential demand that were relatively further
Large no. of subjects characteristics (Pylyshyn, apart than between those
2003) and risk in use of that appeared close
sophisticated subjects together (Kosslyn et al.,
1978)
Cognitive Correlational neural Typically subtractive

neuropsychological methods methods with
approach normalized data
combined with
behavioral data
Healthy subjects and nomothetic EEG/ERP Records changes in Relatively non-invasive Imprecise in resolution EEG activity was
clinical patients electrical potentials via measure of brain activity Only taps cortical changes measured in four tasks
www.frontiersin.org
Systematic exclusion electrodes Relatively expensive after sample was divided
based on certain on the basis of VVIQ
characteristics (e.g., scores (Isaac and Marks,
imagery test 1994)
performance)
Healthy subjects and nomothetic PET/fMRI Differences in the High spatial extent and Restricted movement for Area 17 used in visual
clinical patients amounts of oxgyen resolution subject imagery and perception
Small no. of subjects consumed form basis Highly expensive (Kosslyn et al., 1999)
(<20) for technique.
Healthy subjects and nomothetic TMS Magnetic pulse can Possible to target precise Low spatial extent Used to inhibit imagery
clinical patients inhibit or facilitate area Highly expensive during rest condition prior
Small no. of subjects function to fMRI condition (Kosslyn
(<10) et al., 1999)
Clinical patients Neuropsychological Causal neural methods Observation of patterns Ability to relate brain loci Uniqueness limits ability Dissociation reported
1-2 subjects with rare case studies of dissociation and to functions to generalize between mental imagery
disorder idiographic association and object recognition in
patient with brain damage
(Behrmann et al., 1992)
Strength-based approach to mental imagery
April 2013 | Volume 7 | Article 104 | 41

Table 2 | Antecedents to the strength-based approach in mental imagery research.
Issue Source Contribution
Mental Practice (MP) effect Vandell et al., 1943 Richardson (1967a,b) first narrative review of MP
Trainability of imagery Shepard and Metzler, 1971 Subjects performed several thousand trials over 8–10 weeks
Mental travel effect Decety et al., 1989 Congruence found between the duration of locomotion task and imagery
Converging methods approach Kosslyn and Koenig, 1992 Kosslyn (1994) Neurally based computational model had superseded
computational account (Kosslyn, 1980)
Role of deliberate practice in expertise Ericsson et al., 1993 Expertise is domain-specific and is acquired through practice
Motor cognition approach Jeannerod, 1994 Imagery is integral to motor preparation and action (Jeannerod, 2001, 2006a,b)
Individual differences in imagery Kosslyn et al., 1998 MRT and rCBF study demonstrating two different ways to perform mental
ability rotation, one that involves processes that execute movements and one that
may not
Similarly, the theory of “imagery as simulation” has helped question arose for researchers: do different strategies influence
researchers to postulate theoretical explanations for certain robust performance on mental rotations ability? In 1998, an rCBF tech-
effects in imagery research (e.g., expertise effects in research on nique was employed while subjects mentally rotated either hands
the phenomena of “mental practice” and “mental travel”; see or the original 3-D block objects (Kosslyn et al., 1998). The results
explanation of terms below). Finally, the motor cognition move- highlighted that two mechanisms could be applied—“one mech-
ment has enabled theoretical bridges to be built between imagery anism that recruits processes that prepare motor movements and
research in cognitive sport psychology and that in neuroscience another that does not” (Kosslyn et al., 1998, p. 151). Findings on
(see also Moran et al., 2012). the neural basis of mental rotation, while dependent on the type
of stimuli (animate vs. inanimate), may be contaminated by some
INDIVIDUAL DIFFERENCES subjects using a motor-based strategy. In a follow-up study, sub-
It has long-been known (e.g., since Galton, 1883) that people jects either imagined rotating 3-D block isomers by hand or by a
differ reliably from each other in their imagery experiences and motor and the neural evidence demonstrated differential activa-
skills. For instance, whereas some people can report intense visual tion of the motor and visual cortex, respectively (Kosslyn et al.,
and motor images, akin to actually “seeing” the experience and 2001b).
“feeling” the effort (e.g., running up a stairs), others report little Another consequence of the individual differences movement
detail in their recollection of imagery. This discovery of individual was the range of expert samples used (e.g., pilots; Dror et al.,
differences in imagery has led to several important if somewhat 1993) and the range of methods expanded to explore in an eco-
under-appreciated breakthroughs in our understanding of this logical way the application of mental imagery in everyday life.
construct. For example, consider Shepard and Metzler’s (1971) For example, a weeklong diary study found that students reported
study of mental rotations or people’s ability to turn things over using imagery for a range of purposes including problem solving
in their mind in order to answer questions about the spatial ori- (e.g., navigation) and mental practice (Kosslyn et al., 1990).
entation of certain pairs of shapes. It is worth noting that the
participants in this study were not naïve subjects but highly prac- IMAGERY AND SIMULATION
ticed individual who had performed several thousand trials over The idea of imagery as a cognitive simulation technique may
8–10 weeks. Crucially, this study highlighted the fact that men- be traced back to William James’ (1890) ideomotor principle
tal rotation processes were not only measurable but trainable (Jeannerod, 2006b). Since then, two different imagery effects have
too. Interestingly, the extensive practice trials sparked follow-up come to the attention of researchers—firstly, “mental practice,”
research on the role of practice in mental rotation effects (Steiger and secondly, “mental travel.”
and Yuille, 1983). Results showed that mental rotation effects were Firstly, mental practice, or “the systematic use of mental
robust—regardless of the amount of training received by partici- imagery to rehearse an action in the minds eye without engaging
pants (Leone et al., 1993). Another notable finding from imagery the actual physical movements involved” (Moran, 2012, p. 349),
studies in this era concerns Kosslyn’s (1994) report that Jacky has been subjected to formal study since the 1940’s (Vandell
Metzler (co-author with Roger Shepard) had commented that et al., 1943). In the intervening decades several hundred stud-
some subjects had experienced what appeared to be “kinesthetic” ies have investigated the mental practice effect, which based on
sensations during imagery. It now seems that the mental rotations meta-analytic research has a significant positive effect on the per-
task, which was originally assumed to be visuo-spatial in nature, formance of motor skills (Driskell et al., 1994). A number of
may actually involve the motor system. This discovery of motor moderating variables were reported including the type of task
involvement in visuospatial imagery could account for the early and experience level. To explain, experienced athletes appear to
findings in neuroimaging studies of mental rotation that reported benefit more from mental practice than do novices. This finding
multiple brain areas activated during the task (Kosslyn et al., suggests a possible constraint on the efficacy of mental practice
2001a). Indeed, Kosslyn had argued that “visual mental images for novice learners. Specifically, as Driskell et al. (1994) proposed,
are transformed in part via motor processes” (1994, p. 345). A key that for “novices, who have not formed an approximation of the

skill, the symbolic rehearsal provided by mental practice may not imagery as a window into the representation of action have been
be sufficient to guide performance” (p. 489). One explanation for acknowledged by numerous researchers (e.g., Guillot and Collet,
this expertise effect within mental practice is that the enhanced 2005; Smith and Kosslyn, 2007; Bläsing et al., 2012). The adop-
imagery abilities of experts may enable them to develop greater tion of this new umbrella term, motor cognition, may now ensure
implicit knowledge of the spatial and kinesthetic requirements of that the study of action is within the realm of cognitive neuro-
the tasks than is possible for relative novices. Alternatively, experts science after a degree of neglect by researchers within psychol-
may simply possess greater meta-cognitive knowledge of how to ogy (Rosenbaum, 2005). By implication the domains of dance
employ imagery effectively for skill improvement as compared (Bläsing et al., 2010), sport (Kosslyn and Moulton, 2009) have all,
with novices. as a consequence, evolved as natural laboratories for the study of
Since the early 1990’s researchers have continued to show how motor cognition. Why the neglect of motor control by psychol-
mental practice can improve sport skills (e.g., flip-turn in swim- ogy? Rosenbaum (2005) argues that “motor control has had the
ming; Casby and Moran, 1998), surgical skills (Arora et al., 2011), status of Cinderella in psychological research” (p. 308). Among
and can accelerate the process of neurorehabilitation (McEwan the reasons for the failure of psychology to engage in attempts to
et al., 2009). However, many questions remain unanswered. For understand aspects of motor control were the “too hard to study
example, what are the exact psychological mechanisms underly- hypothesis.” Undoubtedly the complexity of psychological aspects
ing mental practice effects (Kosslyn and Moulton, 2009)? How of motor control was a barrier to research, but the raison d’être
best does one apply imagery optimally in motor skill rehearsal may be that the methodological barriers hindered its exploration.
(Weinberg, 2008)? And how do elite athletes employ mental prac- It seems plausible to propose that motor imagery has only become
tice in field settings (Moran, 2012)? Nevertheless, the mental a topic of study since researchers have had the methodological
practice literature highlighted the clever application of imagery tools to explore it (i.e., through mental kinematics). The afore-
among elite sport participants (Morris et al., 2005). And fur- mentioned factors, allied with the expertise paradigm, based on
thermore, the awareness of the mental practice effect created a the theory of deliberate practice (Ericsson et al., 1993; Ericsson,
common interest for sport psychologists and cognitive neurosci- 2009) provides the impetus for the strength-based approach.
entists alike. While the former were primarily concerned with
performance as the dependent variable, the latter were interested THE STRENGTH-BASED APPROACH TO STUDYING MENTAL
in the neural basis. In fact, researchers within the field of sport IMAGERY
psychology have called for models of mental imagery in sport to The “strength-based” approach provides an extension of the pre-
be grounded in the neuroscientific literature and attempts at such vailing paradigm of cognitive neuroscience, which has focused
theorizing have been tried (Holmes and Collins, 2001; Guillot and primarily upon investigations with both healthy subjects and
Collet, 2008; Wakefield et al., 2013). patient samples. Instead, the emphasis is on those at the high abil-
The second imagery effect that has been reported in the lit- ity end of the continuum. Furthermore, it posits that an expertise
erature is the mental travel effect, which is concerned with the paradigm should be employed, to ensure that a comprehensive
relationship between the duration of a simulated movement (i.e., rationale is provided for the selection of participants (i.e., they
through imagery) and its executed counterpart (Decety et al., meet multiple criteria to establish their level of performance).
1989). For example, in the early study by Decety et al. (1989), To tap this expertise efficiently, it is proposed that the converg-
participants had to perform a blindfold walk and then men- ing methods approach is applied and moreover, that ecological
tally simulate the walk. Congruence between the duration for the validity is considered in the experimental design (Table 3).
tasks in each condition was reported in this study. Since then a Two key issues integral to the application of the strength-based
robust mental travel effect has consistently been found while both approach are now highlighted. These are quantifying expertise
underestimation and overestimation may occur in certain con- and applying the principle of ecological validity. Subsequently,
ditions, the temporal coupling of simulated and executed action recent examples of this approach in music, sport, and dance
is strongly influenced by expertise. Reviews have indicated that samples are discussed. And finally, we highlight where we fore-
elite performers in sport are highly proficient at this skill (Guillot see the “strength-based” approach can illuminate the interaction
and Collet, 2005; Guillot et al., 2012a,b). For example, strong cor- between cognition and action (motor cogntion), the integra-
relations have been reported between the time taken to rehearse tion of multi-sensory information in perception and simulation,
completing a canoe-slalom course and the motor execution time and the role of conscious thought and knowledge in imagery
(Moran and MacIntyre, 1998). Conversely, patients with devel- (metacognition).
opmental movement disorder (e.g., Gabbard et al., 2012), and a
variety of neurological disorders of the motor system have been EXPERTISE PARADIGM
demonstrated to perform poorly on tests comparing the duration Expert performance is defined as consistently superior perfor-
of simulated and executed movements (Guillot and Collet, 2005). mance on a specified set of representative tasks for the domain
that can be administered to any subject (Ericsson, 2009). Ericsson
MOTOR COGNITION and Smith (1991) proposed the expert performance approach to
As a consequence of the robust findings on imagery effects, motor understand the critical mechanisms underlying expertise. Thus,
imagery rose to prominence at the interface between cognitive the “strength-based” approach advances the prevailing research
neuroscience and sport psychology (Moran et al., 2012). Since paradigms by explicitly exploring experts to enhance our under-
Jeannerod’s landmark paper in 1994, the implications of motor standing of mental imagery. As we have noted, this had been done

Table 3 | Proposed assumptions of the strength-based approach. Furthermore, comparisons across expert groups should be
applied rather than simply exploring expert-novice differences.
Assumption Implications
Experts and novices can easily be discriminated from one another
Expertise approach • Use the dimension of expertise to choose in many domains on such an array of variables that the com-
(Ericsson et al., 1993) samples parison can be meaningless. Instead the emphasis should be
• Employ multiple criteria to establish level on inclusion criteria that target participants on the expertise
of expertise spectrum based upon the research question. To explain, while
• Explore expertise across domains relevant the studies listed in Table 4 have samples that reflect a range
to mental imagery processes of abilities, it is evident that purposive sampling was applied.
Ecological validity • To preserve the domain-specific expertise For example, to understand learning a novice sample were used
(Neisser, 1976, 1978) elements of ecological validity should be (Bezzola et al., 2012) and on the other hand, a spectrum of abil-
included in the study design ities were represented in Ross et al.’s (2003) investigation of links
• This should occur across the different between activated brain areas and golf skill.
dimensions-nature of the setting, stimuli, The primacy of the expertise approach within the “strength-
and response set based” approach supplements the prevailing approach with the
Converging methods • This approach should be employed to intention of exploring the role of expertise in mental imagery
(Kosslyn and Koenig, 1992) explore the interaction between abilities, processes. Experts can be recruited, for instance, on the basis
the brain and computation of their special imagery abilities or their sport, or professional
Theory-based approach • Research questions should be guided by activity expertise. Moreover, choosing appropriate activities from
theory which to recruit samples should be based upon the cognitive
• Functional-equivalence and structural task-demands of the actions, rather than an ad-hoc decision. One
equivalent accounts of mental imagery paradox is that clinical patients may develop specialized abilities
provide a road map for research questions in order to cope with the demands of their deficits. One such
example is the case of IW, a patient with peripheral neuropathy,
who has shown diminished motor imagery ability but enhanced
previously but it occurred in an implicit fashion. For instance, the visual imagery ability relative to controls (ter Horst et al., 2012).
training in the original mental rotation study hinted at the devel- The inclusion of expertise as a variable in mental imagery research
opment of expertise (Shepard and Metzler, 1971). Only recently can shed light on both the processes underlying imagery and its
have experts been targeted consistently as samples by cognitive potential application.
neuroscientists (Table 4). Notable exceptions include the use of
expert samples (i.e., artists) by Blajenkova et al. (2006) in their ECOLOGICAL VALIDITY
validation of the Object-Spatial Imagery Questionnaire. Another The expert performance approach of Ericsson and Smith (1991)
example of the implicit expertise approach was the inclusion of proposes that field or laboratory tasks are designed in order to
US air force pilots in a study by Dror et al. (1993) on their visual- retain a high level of ecological validity. As Saarliluoma et al.
spatial ability. However, in that case, we would assert that their (2004) states “it is important to vary the way basic concepts such
expertise should be quantifiable across multiple criteria including as mental imagery are operationalized to avoid the metascien-
their memory, metacognitive skills, and not simply the number of tific Ebbinghaus effect” (p. 753–754). In other words, complex
flying hours accrued. and dynamic tasks in which imagery processes are important
The challenge for researchers, therefore, is to quantify pre- should be subject to scientific scrutiny or aspects of the under-
cisely what constitutes an expert. Standardized ranking systems lying processes may be overlooked. One persistent criticism of
within domains like chess make this possible (Saarliluoma et al., imagery research within the sport context is the use of tasks
2004) but in sport, dance, and music, it is more difficult to have lacking ecological validity (Morris et al., 2005; Moran, 2012).
a standard metric. Typically researchers in sport have defined This challenge applies to perhaps a greater degree within neu-
an expertise as a function of competitive level (e.g., novice, col- roscience where simple laboratory tasks (usually involving con-
legiate, elite, professional). This simple rubric, with elite being strained movements of the fingers) that are chosen not only for
denoted as those competing at the highest possible level (Van den their amenability to computational modeling but also for the
Auweele et al., 1993) is vague and may not adequately reflect the ease with which they can be mastered after a relatively small
nature of the expertise. For example, is an elite athlete equiva- amount of practice (Nielsen and Cohen, 2008; Yarrow et al.,
lent to a chess Grandmaster? Consequently, there is a need to 2009). Ecological validity is a necessary component of studies
fractionate experts from one another in terms of expertise. In that are targeting expertise as if we are to determine their abilities
domains such as music, dance, and sport “performance can be within a converging methods approach, some transfer of skill and
publically observed and even objectively measured in open com- process should occur (Moran, 2012). The dimensions of ecologi-
petition and public performances” (Ericsson, 2009, p. 18). And cal validity include the nature of the setting, the stimuli, and the
moreover, given that expertise is distinguishable according to response (Schmuckler, 2001). These interlinked dimensions can
criteria, including metacognition (Ericsson, 2009), the precise be considered within the converging methods approach that is
performance across a matrix of measures of expertise should be integral to the “strength-based” approach. For example, the diary
explored (e.g., declarative knowledge, predictive ability). study methodology conducted by Kosslyn et al. (1990) could be

Table 4 | Recent studies using neuroimaging methods in mental imagery with music, sport, and dance samples.
Question Method Authors Sample Findings
An attempt to define any fMRI of imagery of Ross et al., 2003 6 golfers from novice Decreased brain activation occurred with
association between activated golf swing to elite level increased golf skill level for the SMA and
brain areas and golf skill cerebellum with little activation of basal
ganglia
Investigation of the cortical fMRI Meister et al., 2004 12 music academy Activations of premotor areas and
network which mediates students precuneus were found in both conditions,
music performance compared contralateral M1 and posterior parietal
to music imagery cortex were active during performance only
Comparison of neural fMRI of imagined Milton et al., 2007 6 expert and 7 novice Extensive practice leads experts to develop
networks of expert and novice pre-shot routine golfers a focused and efficient organization of
golfers during simulation of task-related neural networks, whereas
pre-shot routine novices have difficulty filtering out
irrelevant information
To investigate differences in fMRI of motor Olsson et al., 2008 High-jumpers (12 Imagery training reduces the activity in
brain activity between groups imagery of a high elite and 12 novices) parietal cortex suggesting that imagery is
and to effect of the use of jump performed more automatic and results in a
internal vs. external more efficient motor representation more
perspective easily accessed during motor performance
Role of experience in TMS of familiar and Fourkas et al., 2008 3 expert tennis Subjective reports indicated that only in the
facilitating corticospinal unfamiliar skills players tennis imagery condition did experts differ
representations of actions from novices in the ability to form
proprioceptive images
To investigate multi-modal fMRI during Davidson-Kelly et al., 42-year-old expert Pattern of activation for performed and
musical imagery performed by imagery and 2011 pianist imagined piano music was similar, with the
expert pianist simulated motor motor system of the brain showing similar
performance of a activation during both conditions (except
memorized extract for M1)
Study of dynamic fMRI of golf putts in Bezzola et al., 2012 11 novice golfers and Training induces functional neuroplasticity
neurofunctional changes longitudinal study age-matched controls and skill improvement is associated with a
induced by a physical training over 40 h training modified activation pattern
utilized to evaluate experts’ imagery use over time (e.g., profes- extended to include neuroscience methods, emotion, and action
sional dancers). This approach offers researchers a naturalistic (Smith and Kosslyn, 2007). Furthermore, Martha Farah has stated
laboratory for investigating imagery and action (Moran, 2009). that the present paradigm of cognitive neuroscience “is far from
having outlived its usefulness . . . and I’d like to see it continue to
NEW APPROACHES TO OLD QUESTIONS move towards the edge of our understanding” (2000b, p. 362).
Borrowing from Boring (1957), it may be argued that imagery The evidence presented heretofore, on the growth of the
research has a long past but only a short history and that some “strength-based” approach, conveys how it has augmented the
enduring questions in this field remain unanswered. As we have study of abilities within the cognitive neuroscience triangle
noted, perennial issues around imagery measurement, theory, (Kosslyn, 1994). The expansion of the “strength-based” approach
and function have been central to enquiry for over a century within mental imagery research opens up new modes of enquiry
(Roekelein, 2004). For instance, Jeannerod (2006b) states that the for mental imagery and perception (e.g., Tartaglia et al., 2009)
“mental conception of action” or the “motor idea,” “to account for and specifically in our understanding of action processes. Motor
the role of memory images or remote impressions in shaping an cognition research has the potential to shed light on imagery
action” has been with us since the time of William James (p. 360). processes, the representation of action, and the role of imagery
The key issue is how the prevailing paradigm of cognitive neuro- processes in experts.
science has shifted in recent decades. We have seen how the field
of cognitive psychology has been reconstituted since the 1990s MOTOR COGNITION
with an increasing emphasis on neural implementation (Smith Three key questions within the field of motor cognition can be
and Kosslyn, 2007; Anderson, 2010). Moreover, the key topics that understood by applying the “strength-based” approach. One issue
delineated cognitive psychology in the early textbooks have been surrounds the role of action coupled with motor imagery, what

has been termed dynamic imagery (MacIntyre and Moran, 2010). only imagine the pertinent senses (Holmes and Collins, 2001;
Motor imagery, by definition, occurs in the absence of any overt Weinberg, 2008; MacIntyre and Moran, 2010). Initial evidence,
movement or motor output (Guillot and Collet, 2010). However, from fMRI studies (Ross et al., 2003), dual-task studies (Smyth
on the basis that athletes often engage in movements while and Waller, 1998), and qualitative accounts of how elite perform-
engaging in imagery, sport psychologists have recommended that ers employ imagery (Munroe et al., 2000; MacIntyre and Moran,
performers apply this in their imagery practice (Holmes and 2007a,b) suggests that the simplistic multi-sensory application
Collins, 2001) and moreover, they have amended their definition of imagery merits further investigation. Interestingly, this topic
of mental imagery to include possible motor output (Morris et al., would be of interest across the domains of music, sport, and
2005). Researchers had noted that athletes engaged in either syn- dance that have been referred to in this paper. From another
chronous movements (e.g., simulating the task) or asynchronous perspective, the importance of different senses underlying the
movements (e.g., other movements) during imagery (MacIntyre reported imagery effects has yet to be fully ascertained and both
and Moran, 2010). The role of these quasi-movements (Nikulin inhibitory processes during imagery and consolidation effects are
et al., 2008) has yet to be rigorously evaluated. Preliminary evi- only beginning to be explored (e.g., Guillot et al., 2012a,b).
dence suggests that athletes find this beneficial (MacIntyre and Fourthly, as discussed in the introduction, measuring imagery
Moran, 2010) but to date research has not investigated the com- ability has been an issue of controversy for the field since Galton’s
plex issue of coupling action and motor imagery by athletes (for early attempts at quantification (Galton, 1883). The question
an exception see Guillot et al., 2013). Evidence from other sam- “why do people differ so much in their imagery abilities” (Kosslyn
ples suggests that this is a topic worthy of further research. For et al., 2001a, p. 641) is still pertinent today. The influence of
example, Ionta et al. (2012) reported that variations in the hand implicit or explicit expertise on mental rotations findings, in
position of participants’ during mental rotations tasks influenced mental travel research, and in mental practice studies has been
the latencies for congruent stimuli. They concluded that sensory- established (Guillot and Collet, 2005). And moreover, a trend in
motor and postural information coming from the body might imagery research has been the expansion of the imagery ability
influence mental rotation of body parts according to specific, from one unitary construct to a number of distinct abilities which
somatotopic rules. These preliminary findings were congruent reflect different neural processes (e.g., dorsal vs. ventral stream;
with the body-specificity hypothesis that claims that body-specific Blajenkova et al., 2006). What is less clear is how imagery abilities
patterns of motor experience shape the way we think (Casasanto, are developed and what is the precise role of these imagery abil-
2011). Furthermore, future findings from this line of enquiry may ities in moderating imagery effects (Madan and Singhal, 2012).
have ramifications for the recent accounts of embodied cogni- One alternative to the plethora of pencil and paper imagery tests
tion (Borghi and Cimatti, 2010; Gallese and Sinigaglia, 2012). employed in sport (e.g., Williams and Cumming, 2011) or neu-
One confound that has been noted in research on the action- roscience settings (e.g., McAvinue and Robertson, 2007) and is to
motor imagery coupling is the imagery perspective (the viewpoint employ a compound measure, the motor imagery index, which
adopted during visual imagery). This has resulted in debate combines psychometric, behavioral (e.g., mental travel), and psy-
in both sport psychology and cognitive neuroscience. Moran chophysiological measures (Collet et al., 2011). It is vital that we
(2012) noted that the complexity of agency, visual perspective, are able to quantify imagery abilities if we are to match partic-
and confounds with kinesthetic or motor imagery were reflected ipants for competence or if we wish to evaluate the trainability
in the findings emanating from sport psychology. Researchers of imagery abilities. Recent research has explored the trainability
had developed sophisticated methodologies to attempt to control of imagery vividness using robust measures and interestingly, the
and measure the visual perspective adopted during testing. The only reported changes were in the metacognitive understanding
“strength-based” approach is one possible route to understanding of their imagery (Rademaker and Pearson, 2012).
this topic further. A recent special issue of the Journal of Mental
Imagery on whether the internal viewpoint is a default hypothesis META-IMAGERY
is testament to the continuing interest in this topic (Morris and Theoretically, a potentially valuable new route for imagery
Spittle, 2012). The issue of visual perspective in mental imagery, researchers in cognitive neuroscience concerns the investigation
because of the conflation with both visual imagery and motor of the neglected topic of “meta-imagery processes”—“their beliefs
imagery in the past, has been noted as a topic that necessitates about the nature and regulation of their own imagery skills”
further research (Madan and Singhal, 2012). (Moran, 2002, p. 415). Research in the expertise literature sug-
A second question that relates to motor cognition is the process gests that meta-cognition, people’s insight into, and control over
of multi-sensory integration, which is an area of current debate their own mental processes, is a factor that differentiates novices
(Foxe and Molholm, 2009). More specifically, the relative contri- from experts (Moran, 2012). Interest in this topic surprisingly
bution of different senses in the simulation of action is of direct emerged from a survey by Denis and Carfantan (1985) who sur-
concern to neuroscientists (Lacey and Lawson, 2013). Recently, veyed undergraduate students on their knowledge about imagery
this topic has received increased attention because of concerns research findings in psychology. The motive for their study was
with the traditional approach in the understanding and applica- to quantify the participants’ tacit knowledge of imagery effects
tion of multi-sensory imagery. For example, within sport psychol- (Denis, 2012). In order to assess the level of tacit knowledge among
ogy, it was traditionally assumed that a multi-modal approach experimental subjects they were asked to predict the outcomes
was optimal (Morris et al., 2005). However, this has recently of various imagery experiments that were described but not for-
been questioned on the basis that it may be more important to mally named (e.g., is memory for pictures better than memory

for words?). The findings indicated that although the majority this latter approach has led to advances in our understanding of
of participants predicted correctly that imagery would have ben- the overlap between visual perception and imagery in scanning
eficial effects on learning and reasoning, few subjects were able tasks (Borst and Kosslyn, 2008). Based on such progress, there is
to predict accurately the results of mental rotation experiments an imperative for researchers to maintain the traditional approach
(whereby more time is required to accomplish greater amounts in order to answer key questions. For example, the use of ran-
of rotation of images) or mental scanning studies (whereby domized controlled trials to explore the role of motor imagery
longer distances between points in an image take longer to scan in patients with sub-acute neglect (Welfringer et al., 2011) and
than shorter distances). Furthermore, a majority of participants with stroke victims (Ietswaart et al., 2011) are essential for the
rejected the idea that mental imagery could enhance the per- validation of imagery interventions. Similarly, the role of men-
formance of motor skills (the mental practice effect). This latter tal imagery among clinical patient samples (Pearson et al., 2012)
finding led Denis and Carfantan (1985) to conclude “how coun- and stroke victims (Confalonieri et al., 2012) requires contin-
terintuitive the idea is that motor skills may be affected by purely ued investigation. Unsurprisingly, studies of patients will, in all
mental practice” (p. 56). The naïve responses of the participants likelihood, continue to inform our appreciation of the deficits,
in this study are in stark contrast to the evidence that has recently challenges, and recovery strategies of those with specific with rare
emerged from sporting samples. Researchers have asked athletes disorders (ter Horst et al., 2012). This again raises aforementioned
and dancers to indicate why, where, how, what, and when they paradox of expertise among patient samples. A recent study with
use mental imagery processes (e.g., Nordin and Cumming, 2005; a Paralympic athlete led to the conclusion that only tasks that we
MacIntyre and Moran, 2007a,b). Athletes’ responses from both have physical experience of recruit the motor system (Olson and
interviews and surveys demonstrated a comprehensive knowl- Nyberg, 2011).
edge of the multi-model potential of imagery, their awareness Despite the success of traditional approaches to imagery, how-
of both mental practice and mental travel effect, and high- ever, a strength-based approach may contribute in the devel-
lighted the sophisticated nature of their understanding of imagery opment and refinement of imagery inventories, as implicitly
(MacIntyre and Moran, 2010). employed in the case of the Object-Spatial Imagery Questionnaire
In 2002, Moran suggested that it would be interesting to dis- (Blajenkova et al., 2006). Similarly, investigations of the neu-
cover if top athletes have greater insight into and control over ral processes underlying imagery effects in expert samples will
their use of imagery compared with their less successful counter- help to elucidate both applied and theoretical aspects of men-
parts. To date the preponderance of the evidence favors an exper- tal imagery. However, the new approach advocated in this paper
tise effect for meta-imagery. In fact, a model of meta-imagery has its limits too. For example, analyzing the imagery skills of
in action suggests that there are three components-knowledge, motor experts in an atheoretical manner will not advance concep-
monitoring and control, which opens up possibilities of devel- tual understanding and may provide spurious findings. Instead,
oping a test of meta-imagery (MacIntyre and Moran, 2010). a rigorous, theory-driven approach with converging methods
Furthermore, contemporary evidence from cognitive psychology is required for strength-based approaches to yield benefits to
supports the role of meta-cognitive knowledge of imagery abil- imagery researchers. As anticipated by Kosslyn et al. (2002) over a
ity and relates it to our ability to judge individual episodes of decade ago, “individual differences can actually help to reveal the
imagery (Pearson et al., 2011). The voluntary nature of imagery nature of underlying mechanisms (p. 342).”
and the role of conscious awareness during imagery tasks make
it amenable to introspection (Pearson et al., 2008), ironically CONCLUSIONS
the method that was central to the demise of the scientific Cognitive neuroscience has made impressive progress in the illu-
study of imagery, a century ago (see Block, 1983). While in mination of the nature, function, and neural basis of mental
the past the study of metacognition has targeted intellectual imagery (Kosslyn, 2010). Nevertheless, certain aspects of this con-
skills “if intellectual and perceptual-motor skills rely on simi- struct (e.g., its relationship to skilled performance) have been
lar mechanisms, one would expect metacognition to apply to relatively neglected by mainstream imagery researchers. In this
the guidance of perceptual-motor skills, just as it does to the article, certain significant trends—antecedents of a strength-
guidance of intellectual skills” (Augustyn and Rosenbaum, 2005, based approach to imagery—that may be detected over more
p. 911). Consequently, armed with a comprehensive roadmap of than a century of imagery research have been highlighted. These
imagery processes and an increased understanding of action, the trends provide signposts for at least four potentially fruitful
study of meta-imagery could provide a back door into the typ- new avenues of inquiry. Firstly, imagery research continues to
ically impenetrable realm of sensory perception (Pearson et al., evolve from an excessively narrow focus on the visual sense (e.g.,
2011). Galton, 1883; Perky, 1910) to a modern concern with inves-
tigation of the true multi-sensory character of this construct.
SOME OBSERVATIONS ON THE STRENGTH-BASED Next, theoretical understanding of mental imagery has deep-
APPROACH WITHIN COGNITIVE NEUROSCIENCE ened with increasing awareness of the multi-dimensional nature
In this paper, we have argued that a strength-based approach of this construct. Thirdly, major methodological advances, both
to mental imagery can augment rather than replace the tradi- in technology and sampling, have illuminated both fundamen-
tional approach to this construct. After all, this latter approach has tal and applied questions. Finally, the preliminary application
been highly successful in, for example, illuminating both visual of the “strength-based” approach has been shown to be useful
imagery and visual cognition (Kosslyn et al., 2006). More recently, in enriching our understanding of the neural basis of expert

performance (Ross et al., 2003; Milton et al., 2007; Olsson with the “strength-based” approach will be most obvious in the
et al., 2008) and the “clever application” of imagery in sport domain of motor cognition where issues of dynamic imagery,
(Moran et al., 2012), dance (Bläsing et al., 2012), and music visual imagery perspective, and multi-modal integration can be
contexts (Meister et al., 2004). It is noteworthy that the inter- explored. Also, the area of metacognition research is a rapidly
est in exploring the naturalistic expertise in imagery (e.g., dance) growing field and meta-imagery as a topic is evolving. Meta-
with neuroscientific methods stems is bi-directional. For exam- imagery research is a topic than can benefit from the “strength-
ple, “neuroscientists have turned to dancers as a valuable human based” approach. In summary, through the lens of motor cog-
resource in possession of a rich skill set . . . to address issues nition, further exploration of the construct of mental imagery
of how the brain coordinates perception with action” (Cross, will ensure that the topic of will remain part of the cognitive
2010, p. 197). The implications of augmenting existing paradigms neuroscience lexicon for many decades to come.
REFERENCES Block, N. (1983). Mental pictures and Bläsing, M. Puttke, and T. Schack of expert performance. Psychol. Rev.
Anderson, J. R. (2010). Cognitive cognitive science. Philos. Rev. 92, (Hove, East Sussex: Psychology 100, 363–406.
Psychology and its Implications. New 499–541. Press), 177–202. Ericsson, K. A., and Smith, J. (1991).
York, NY: Worth Publishers. Borghi, A. M., and Cimatti, F. (2010). Davidson-Kelly, K., Hong, S., “Prospects and limits in the empir-
Arora, S., Aggarwal, R., Sirimanna, Embodied cognition and beyond: Dhinakaran, J., Joseph, M., and ical study of expertise: an introduc-
P., Moran, A., Grantcharov, T., acting and sensing the body. Gray, C. (2011). “An fMRI study tion,” in Toward a General Theory
Kneebone, R., et al. (2011). Mental Neuropsychologia 48, 763–773. of expert musical imagery: to of Expertise: Prospects and Limits,
practice enhances surgical techni- Boring, E. G. (1957). A History of what extent do imagined and eds K. A. Ericsson and J. Smith
cal skills: a randomized controlled Experimental Psychology. New York, executed performance share (Cambridge: Cambridge University
study. Ann. Surg. 253, 265–270. NY: Prenctice Hall. the same neural substrate,” in Press), 1–38.
Augustyn, J. S., and Rosenbaum, D. Borst, G., and Kosslyn, S. M. (2008). Proceedings of the International Farah, M. J. (2000a). “The neural
A. (2005). Metacognitive control Visual mental imagery and visual Symposium on Performance basis of mental imagery,” in The
of action: preparation for aiming perception: structural equivalence Science 2011, eds A. Williamon, Cognitive Neurosciences, ed M. S.
reflects knowledge of Fitts’ Law. revealed by scanning processes. D. Edwards, and L. Bartel Gazzaniga (Cambridge, MA: MIT
Psychon. Bull. Rev. 12, 911–916. Mem. Cognit. 36, 849–862. (Utrecht: European Association Press), 965–974.
Behrmann, M. (2000). The mind’s eye Brown, S., Martinez, M. J., and Parsons, of Conservatoires (AEC), Toronto), Farah, M. J. (2000b). Interview with
mapped onto the brain’s matter. L. M. (2005). The neural basis of 613–617. Martha Farah. J. Cogn. Neurosci. 12,
Curr. Dir. Psychol. Sci. 9, 50–54. human dance. Cereb. Cortex 16, Decety, J., Jeannerod, M., and Prablanc, 360–363.
Behrmann, M., Winocur, G., 1157–1167. C. (1989). The timing of mentally Fourkas, A. D., Bonavolonta, V.,
and Moscovitch, M. (1992). Casasanto, D. (2011). Different represented actions. Behav. Brain Avenanta, A., and Aglioti, S. M.
Dissociation between mental bodies, different minds: the Res. 34, 35–42. (2008). Kinesthetic imagery and
imagery and object recognition in a body specificity of language and Denis, M. (2012). “Decades of imagery. tool-specific modulation of corti-
brain-damaged patient. Nature 359, thought. Curr. Dir. Psychol. Sci. 20, Reminisences of a shared journey,” cospinal representations in expert
636–637. 378–383. in Essays in honour of Michel Denis: tennis players. Cereb. Cortex 18,
Bensafi, M., Porter, J., Pouliot, S., Casby, A., and Moran, A. (1998). From Mental Imagery to Spatial 2382–2390.
Mainland, J., Johnson, B., Zelano, Exploring mental imagery in swim- Cognition and Language, eds V. Foxe, J. J., and Molholm, S. (2009). Ten
C., et al. (2003). Olfactory activity mers: a single-case study design. Ir. Gyselinck and F. Pazzaglia (Hove, years at the Multisensory Forum:
during imagery mimics that dur- J. Psychol. 19, 525–531. East Sussex: Psychology Press), musings on the evolution of a field.
ing perception. Nat. Neurosci. 6, Collet, C., Guillot, A., Lebon, F., 203–254. Brain Topogr. 21, 149–154.
1142–1144. MacIntyre, T., and Moran, Denis, M., and Carfantan, M. (1985). Gabbard, P., Cacola, P., and Bobbio, T.
Bezzola, L., Merillat, S., and Jancke, L. A. (2011). Measuring motor People’s knowledge about images. (2012). The ability to mentally rep-
(2012). The effect of leisure activ- imagery: combining psychometric, Cognition 20, 49–60. resent action is associated with low
ity golf practice on motor imagery: qualitative, chronometric, and Djordjevic, J., Zatorre, R. J., Petrides, motor imagery ability in children:
an fMRI study in middle adult- psychophysiological techniques. M., Boyle, J. A., and Jones-Gotman, a preliminary investigation. Child
hood. Front. Hum. Neurosci. 6:67. Exerc. Sport Sci. Rev. 39, 89–92. M. (2005). Functional neuroimag- Care Health Dev. 38, 390–393.
doi: 10.3389/fnhum.2012.00067 Confalonieri, L., Pagnoni, G., Barsalou, ing of odor imagery. Neuroimage 24, Gallese, V., and Sinigaglia, C. (2012).
Blajenkova, O., Kozhevnikov, M., and L. W., Rajendra, J., Eickhoff, S. 791–801. What is so special about embodied
Motes, M. A. (2006). Object-spatial B., and Butler, A. J. (2012). Brain Driskell, J., Copper, C., and Moran, simulation? Trends Cogn. Neurosci.
imagery: a new self-report imagery activation in primary motor and A. (1994). Does mental prac- 15, 512–519.
questionnaire. Appl. Cogn. Psychol. somatosensory cortices during tice enhance performance? A Galton, F. (1883). Enquiries into Human
20, 239–263. motor imagery correlates with meta-analysis. J. Appl. Psychol. 79, Faculty. London: Dent.
Bläsing, B. E., Calvo-Merino, B., motor imagery ability in stroke 481–492. Guillot, A., and Collet, C. (2005).
Cross, E. S., Jola, C., Honisch, patients. ISRN Neurol. 2012, 1–17. Dror, I., Kosslyn, S. M., and Waag, Duration of mentally simulated
J., and Stevens, C. J. (2012). Cornoldi, C., and De Beni, R. (2012). W. (1993). Visual-spatial abili- movement: a review. J. Mot. Behav.
Neurocognitive control in dance “Foreword,” in Essays in Honour of ties of pilots. J. Appl. Psychol. 78, 37, 10–20.
perception and performance. Acta Michel Denis: From Mental Imagery 763–773. Guillot, A., and Collet, C. (2008).
Psychol. 139, 300–308. to Spatial Cognition and Language, Ericsson, K. A. (2009). Development Construction of the motor imagery
Bläsing, B., Puttke, M., and Schack, eds V. Gyselinck and F. Pazzaglia of Professional Expertise: Toward integrative model in sport: a review
T. (2010). “The neurocognition (Hove, East Sussex: Psychology Measurement of Expert Performance and theoretical investigation of
of dance. Introduction,” in The Press), ix–x. and Design of Optimal Learning motor imagery use. Int. Rev. Sport
Neurocognition of Dance: Mind, Cross, E. S. (2010). “Building a dance Environments. Cambridge, MA: Exerc. Psychol. 1, 31–44.
Movement and Motor Skills, eds B. in the human brain: Insights from Cambridge University Press. Guillot, A., and Collet, C. (2010). The
Bläsing, M. Puttke, and T. Schack expert and novice dancers,” in The Ericsson, K. A., Krampe, R. T., and Neurophysiological Foundations of
(Hove, East Sussex: Psychology Neurocognition of Dance: Mind, Tesch-Römer, C. (1993). The role of Mental and Motor Imagery. Oxford:
Press), 1–8. Movement and Motor Skills eds B. deliberate practice in the acquisition Oxford University Press.

Guillot, A., Di Rienzo, F., MacIntyre, Kosslyn, S. M. (2010). “Multimodal Processing in Navigation, Imagery, post-stroke: a critical review. Brain
T., Moran, A., and Christian, C. images in the brain,” in The and Perception, eds F. Mast and L. Injury 23, 263–277.
(2012a). Imagining is not doing Neurophysiological Foundations of Jancke (New York, NY: Springer), Meister, I. G., Krings, T., Foltys, H.,
but involves specific motor com- Mental and Motor Imagery, eds 1–16. Boroojerdi, B., Muller, M., Topper,
mands: a review of experimental A. Guillot and C. Collet (New Kosslyn, S. M., Thompson, W. L., R., et al. (2004). Playing piano
data related to motor inhibition. York, NY: Oxford University Press), and Ganis, G. (2006). The Case in the mind: an fMRI study on
Front. Hum. Neurosci. 6:247. doi: 3–16. for Mental Imagery. Oxford: Oxford music imagery and performance in
10.3389/fnhum.2012.00247 Kosslyn, S. M., Ball, T. M., and Reiser, University Press. pianists. Brain Res. Cogn. Brain Res.
Guillot, A., Hoyek, N., Louis, M., and B. J. (1978). Visual images pre- Kozhevnikov, M., and Blajenkova, 19, 219–228.
Collet, C. (2012b). Learning by serve metric spatial information: O. (2013). “Individual differences Milton, J., Solodkin, A., Hlustik, P.,
thinking: review and future direc- evidence from studies of image in object versus spatial imagery: and Small, S. L. (2007). The mind
tions for the timing of imagined scanning. J. Exp. Psychol. Hum. 4, from neural correlates to real-world of expert performance is cool and
movements. Int. Rev. Sport Exerc. 47–60. applications,” in Theoretical and focused. Neuroimage 35, 804–813.
Psychol. 5, 3–22. Kosslyn, S. M., Behrmann, M., and Applied Perspectives on Multisensory Moran, A. (2002). In the mind’s eye.
Guillot, A., Moschberger, K., and Jeannerod, M. (1995). The cogni- Imagery, eds S. Lacey and R. Psychologist 15, 414–415.
Collet, C. (2013). Coupling move- tive neuroscience of mental imagery. Lawson (New York, NY: Springer), Moran, A. (2009). Cognitive psy-
ment with imagery as a new Neuropsychologia 33, 1335–1344. 299–318. chology in sport: progress and
perspective for motor imagery Kosslyn, S. M., Cacioppo, J. T., Lacey, S., and Lawson, R. (2013). prospects. Psychol. Sport Exerc. 10,
practice. Behav. Brain Funct. 9, Davidson, R. J., Hugdahl, K., Theoretical and Applied Perspectives 420–426.
1–17. Lovallo, W. R., Spiegel, D., et al. on Multisensory Imagery. New York, Moran, A., Guillot, A., MacIntyre, T.,
Holmes, P. S., and Collins, D. J. (2002). Bridging psychology and NY: Springer. and Collet, C. (2012). Re-imagining
(2001). The PETTLEP approach to biology: the analysis of individ- Leone, G., Taine, M. C., and Droulez, J. motor imagery: building bridges
motor imagery: a functionalequiva- uals in groups. Am. Psychol. 57, (1993). The influence of long-term between cognitive neuroscience and
lence model for sport psychologists. 341–351. practice on mental rotation strate- sport psychology. Br. J. Psychol. 103,
J. Appl. Sport Psychol. 13, 60–83. Kosslyn, S. M., Digirolamo, G. J., gies of 3-D objects. Brain Res. Cogn. 224–247.
Hubbard, T. L. (2010). Auditory Thompson, W. L., and Alpert, Brain Res. 1, 241–255. Moran, A. P. (2012). Sport and Exercise
imagery: empirical findings. N. M. (1998). Mental rotation of MacIntyre, T., and Moran, A. (2007a). Psychology: A Critical Introduction.
Psychol. Bull. 136, 302–329. objects versus hands: neural mech- A qualitative investigation of London: Routledge.
Ietswaart, M., Johnston, M., anisms revealed by positron emis- imagery use and meta-imagery Moran, A. P., and MacIntyre, T. (1998).
Dijkerman, H. C., Joice, S., Scott, sion tomography. Psychophysiology processes among elite canoe-slalom “There’s more to an image than
C. L., MacWalter, R. S. et al. (2011). 35, 151–161. competitors. J. Imagery Res. Sport meets the eye”: a qualitative study
Mental practice with motor imagery Kosslyn, S. M., Ganis, G., and Phys. Act. 2, 1–23. of kinaesthetic imagery among elite
in stroke recovery: randomized con- Thompson, W. L. (2001a). Neural MacIntyre, T., and Moran, A. (2007b). canoe-slalomists. Ir. J. Psychol. 19,
trolled trial of efficacy. Brain 134, foundations of imagery. Nat. Rev. A qualitative investigation of meta- 406–423.
1373–1386. Neurosci. 2, 635–642. imagery processes and imagery Morris, A., and Spittle, A. (2012).
Ionta, S., Perruchoud, D., Draganski, Kosslyn, S. M., Thompson, W. L., direction among elite athletes. Target article: a default hypothe-
B., and Blanke, O. (2012). Wraga, M., and Alpert, N. M. J. Imagery Res. Sport Phys. Act. 2, sis of the development of internal
Body context and posture (2001b). Imagining rotation by 1–20. and external imagery perspectives. J.
affect mental imagery of hands. endogenous and exogenous forces: MacIntyre, T., and Moran, A. (2010). Mental Imagery 36, 1–30.
PLoS ONE 7:e34382. doi: Distinct neural mechanisms for “Meta-imagery processes among Morris, T., Spittle, M., and Watt,
10.1371/journal.pone.0034382 different strategies. Neuroreport 12, elite sports performers,” in The A. P. (2005). Imagery in Sport.
Isaac, A. R., and Marks, D. F. (1994). 2519–2525. Neurophysiological Foundations of Champaign, IL: Human Kinetics.
Individual differences in mental Kosslyn, S. M., and Koenig, O. (1992). Mental and Motor Imagery, eds Munroe, K., Giacobbi, P., Hall, C.,
imagery experience: developmen- Wet Mind. The New Cognitive A. Guillot and C. Collet (New and Weinberg, R. (2000). The four
tal changes and specialisation. Br. Neuroscience. New York/Toronto: York, NY: Oxford University Press), W’s of imagery use: where, when,
J. Psychol. 85, 479–500. The Free Press. 227–244. why, and what. Sport Psychol. 14,
James, W. (1890). Principles of Kosslyn, S. M., and Moulton, S. T. Madan, C. R., and Singhal, A. (2012). 119–137.
Psychology. New York, NY: Holt, (2009). “Mental imagery and Motor imagery and higher level cog- Munzert, J., Lorey, J., and Zentgraf,
Rinehart and Winston. implicit memory,” in Handbook of nition: four hurdles before research J. (2009). Cognitive motor pro-
Jeannerod, M. (1994). The represent- Imagination and Mental Simulation, can sprint forward. Cogn. Process. cesses: the role of motor imagery
ing brain: neural correlates of motor eds K. D. Markman, W. M. P. Klein, 13, 211–229. in the study of motor repre-
intention and imagery. Behav. Brain and J. A. Suhr (New York, NY: Malouin, F., and Richards, C. L. sentations. Brain Res. Rev. 60,
Sci. 17, 187–202. Psychology Press), 35–52. (2013). “Clinical applications of 306–326.
Jeannerod, M. (2001). Neural sim- Kosslyn, S. M., Pascual-Leone, A., motor imagery in rehabilitation,” in Munzert, J., and Zentgraf, J. (2009).
ulation of action: a unifying Felician, O., Camposano, S., Theoretical and Applied Perspectives Motor imagery and its implications
mechanism for motor cognition. Keenan, J. P., Thompson, W. L., on Multisensory Imagery, eds S. for understanding the motor sys-
Neuroimage 14, S103–S109. et al. (1999). The role of area 17 in Lacey and R. Lawson (New York, tem. Prog. Brain Res. 174, 219–229.
Jeannerod, M. (2006a). Motor visual imagery: convergent evidence NY: Springer), 397–420. Neisser, U. (1976). Cognition and
Cognition. New York, NY: Oxford from PET and rTMS. Science 284, Marks, D. F. (1973). Visual imagery dif- Reality: Principles and Implications
University Press. 167–170. ferences in the recall of pictures. Br. of Cognitive Psychology. New York,
Jeannerod, M. (2006b). The origin of Kosslyn, S. M., Seger, C., Pani, J., J. Psychol. 64, 17–24. NY: WH Freeman.
voluntary action. History of a phys- and Hillger, L. A. (1990). When is McAvinue, L. P., and Robertson, I. H. Neisser, U. (1978). “Memory: What
iological concept. C. R. Biol. 329, imagery used in everyday life? A (2007). Measuring motor imagery are the important questions?” in
354–362. diary study. J. Ment. Imagery 14, ability: a review. Eur. J. Cogn. Practical Aspects of Memory, eds M.
Kosslyn, S. M. (1980). Image and 131–152. Psychol. 20, 232–251. M. Gruneburg, P. E. Morris, and R.
Mind. Cambridge, MA: Harvard Kosslyn, S. M., Shephard, J. M., and McEwan, S. E., Hujbregts, M. P. J., N. Sykes (London: Academic Press),
University Press. Thompson, W. L. (2007). “Spatial Ryan, J. D., and Polatajko, H. 3–24.
Kosslyn, S. M. (1994). Image and Brain. processing during mental imagery: J. (2009). Cognitive strategy use Nielsen, B. J., and Cohen, L. G.
Cambridge, MA: MIT Press. a neurofunctional theory,” in Spatial to enhance motor skill acquisition (2008). The Olympic brain. Does

corticospinal plasticity play a role in the metacognition of visual imagery. Shepard, R. N., and Metzler, J. (1971). model of motor imagery. Int. Rev.
the acquisition of skills required for Psychol. Sci. 22, 1535–1542. Mental rotation of three dimen- Sport Exerc. Psychol. 6, 105–121.
high performance sports? J. Physiol. Perky, C. W. (1910). An experimental sional objects. Science 171, 701–703. Walsh, V., and Cowey, A. (2000). TMS
586, 65–70. study of imagination. Am. J. Psychol. Slotnick, S. D., Thompson, W. L., and and cognitive neuroscience. Nat.
Nikulin, V. V., Hohlefeld, F. U., 21, 422–452. Kosslyn, S. M. (2005). Visual men- Rev. Neurosci. 1, 73–79.
Jacobs, A. M., and Curio, G. Pylyshyn, Z. (2003). Explaining mental imagery induces retinotopically Weinberg, R. S. (2008). Does imagery
(2008). Quasi-movements: a novel tal imagery: now you see it, now organized activation of early visual work? Effects on performance and
motor-cognitive phenomena. you don’t. Trends Cogn. Sci. 7, areas. Cereb. Cortex 15, 1570–1583. mental skills. J. Imagery Res. Sport
Neuropsychologia 46, 727–742. 111–112. Smith, E. E., and Kosslyn, S. M. (2007). Phys. Act. 3, 1–21.
Nitschke, J. B., Dixon, G. E., Rademaker, R. L., and Pearson, J. “Motor cognition and mental sim- Welfringer, A., Leifert-Fiebach, G.,
Sarinopoulos, I., Short, S. J., (2012). Training visual imagery: ulation,” in Cognitive Psychology: Babinsky, R., and Brandt, T. (2011).
Cohen, J. D., Smith, E. E., et al. improvements of metacogni- Mind and Brain, eds E. E. Smith and Visuomotor imagery as a new tool
(2006). Altering expectancy damp- tion, but not imagery strength. S. M. Kosslyn (Upper Saddle River, in the rehabilitation of neglect: a
ens neural response to aversive Front. Psychol. 3:224. doi: NJ: Prentice Hall), 451–481. randomised controlled study of
taste in primary taste cortex. Nat. 10.3389/fpsyg.2012.00224 Smyth, M. M., and Waller, A. (1998). feasibility and efficacy. Disabil.
Neurosci. 9, 435–442. Richardson, A. (1967a). Mental prac- Movement imagery in rock climb- Rehabil. 33, 2033–2043.
Nordin, S. M., and Cumming, J. tice: a review and discussion, Part 1. ing: patterns of interference from Williams, S. E., and Cumming, J.
(2005). Professional dances describe Res. Q. 38, 95–107. visual, spatial and kinaesthetic sec- (2011). Measuring athlete imagery
their imagery: where when, what, Richardson, A. (1967b). Mental prac- ondary tasks. Appl. Cogn. Psychol. ability: the sport imagery abil-
why, and how. Sport Psychol. 19, tice: a review and discussion, Part II. 12, 145–157. ity questionnaire. J. Sport Exerc.
395–416. Res. Q. 38, 263–273. Steiger, J. H., and Yuille, J. C. (1983). Psychol. 33, 416–440.
Olson, C. J., and Nyberg, L. (2011). Roekelein, J. E. (2004). Imagery and Long term memory and men- Wraga, M., and Kosslyn, S. M. (2002).
Brain simulation of action may be Psychology: A Reference Guide. tal rotation. Can. J. Psychol. 37, “Imagery,” in Encyclopedia of
grounded in physical experience. Westport, CT: Praeger. 367–389. Cognitive Science, Vol. 2 (London:
Neurocase 17, 501–505. Rosenbaum, D. A. (2005). The Stewart, L., and Walsh, V. (2006). Nature Publishing Group),
Olsson, C. J., Johnsson, B., Larsson, Cinderella of psychology: the “Transcranial magnetic stimulation 466–470.
A., and Nyberg, L. (2008). Motor neglect of motor control in the in cognition,” in Methods in Mind, Yarrow, K., Brown, P., and Krakauer,
representations and practice science of mental life and behavior. eds C. Senior, T. Russell, and M. S. J. W. (2009). Inside the brain of an
affect brain systems underlying Am. Psychol. 60, 308–317. Gazzaniga, (Cambridge, MA: MIT elite athlete: the neural processes
imagery: an fMRI study of internal Ross, J. S., Thach, J., Ruggieri, P. M., Press), 1–26. that support high achievement
imagery in novices and active high Lieber, M., and Lapresto, E. (2003). Tartaglia, E. M., Bamert, L., Mast, F. W., in sports. Nat. Rev. Neurosci. 10,
jumpers. Open Neuroimag. J. 2, The mind’s eye: functional MR and Herzog, M. H. (2009). Human 585–596.
5–13. imaging evaluation of golf motor perceptual learning by mental
Paivio, A., Yuille, J. C., and Madigan, imagery. Am. J. Neuroradiol. 24, imagery. Curr. Biol. 19, 2081–2085. Conflict of Interest Statement: The
S. A. (1968). Concreteness, imagery, 1036–1044. ter Horst, A. C., Cole, J., van Lier, authors declare that the research
and meaningfulness values for 925 Saarliluoma, P., Karlsson, H., Lyytinen, R., and Steenbergen, B. (2012). The was conducted in the absence of any
nouns. J. Exp. Psychol. 76, 1–15. H., Teras, M., and Geisler, F. (2004). effect of chronic deafferentation on commercial or financial relationships
Pascual-Leone, A., Nguyet, D., Visuospatial representations used mental imagery: a case study. PLoS that could be construed as a potential
Cohen, L. G., Brasil-Neto, J. P., by chess experts: a preliminary ONE 7:e42742. doi: 10.1371/jour- conflict of interest.
Cammarota, A., and Hallett, M. study. Eur. J. Cogn. Psychol. 16, nal.pone.0042742
(1995). Modulation of muscle 753–766. Van den Auweele, Y., Depreeuw, E., Received: 01 May 2012; accepted: 11
responses evoked by transcranial Sarter, M., Berntson, G. G., and Rzewnicki, R., and Ballon, F. (1993). March 2013; published online: 01 April
magnetic stimulation during the Cacioppo, J. T. (1996). Brain imag- “Elite performance and personal- 2013.
acquisition of new fine motor skills. ing and cognitive neuroscience: ity: from description and predic- Citation: MacIntyre TE, Moran AP,
J. Neurophysiol. 74, 1037–1045. toward strong inference in attribut- tion to diagnosis and intervention,” Collet C and Guillot A (2013) An emerg-
Pearson, D. G., Deeprose, C., Wallace- ing function to structure. Am. in Handbook of Research on Sport ing paradigm: a strength-based approach
Hadrill, S. M. A., Heyes, S. B., Psychol. 51, 13–21. Psychology, eds R. N. Singer, M. to exploring mental imagery. Front.
and Holmes, E. A. (2012). Assessing Schmuckler, M. A. (2001). What is eco- Murphey, and L. K. Tennant (New Hum. Neurosci. 7:104. doi: 10.3389/
mental imagery in clinical psychol- logical validity? A dimensional anal- York, NY: Macmillan), 257–289. fnhum.2013.00104
ogy: a review of imagery measures ysis. Infancy 2, 419–436. Vandell, R. A., Davis, R. A., and Copyright © 2013 MacIntyre, Moran,
and a guiding framework. Clin. Senior, C. T., Russell, T., and Gazzaniga, Clugston, H. A. (1943). The func- Collet and Guillot. This is an open-
Psychol. Rev. 33, 1–23. M. S. (2006). Methods in Mind. tion of mental practice in the acqui- access article distributed under the terms
Pearson, J., Clifford, C. W. G., and Cambridge, MA: MIT Press. sition of motor skills. J. Gen. Psychol. of the Creative Commons Attribution
Tong, F. (2008). The functional Sevdalis, N., Moran, A., and Arora, S. 29, 243–250. License, which permits use, distribution
impact of mental imagery on con- (2013). “Mental imagery and men- Wakefield, C. J., Smith, D., Moran, A., and reproduction in other forums, pro-
scious perception. Curr. Biol. 18, tal practice applications in surgery,” and Holmes, P. (2013). Functional vided the original authors and source
982–986. in Multisensory Imagery, eds S. Lacey equivalence or behavioural match- are credited and subject to any copy-
Pearson, J., Rademaker, R., and Tong, and R. Lawson (New York, NY: ing? A critical reflection on 15 years right notices concerning any third-party
F. (2011). Evaluating the mind’s eye: Springer), 343–363. of research using the PETTLEP graphics etc.

published: 04 February 2013
doi: 10.3389/fpsyg.2013.00001
Vividness of visual imagery and incidental recall of verbal

cues, when phenomenological availability reflects
long-term memory accessibility
Amedeo D’Angiulli *, Matthew Runge, Andrew Faulkner , Jila Zakizadeh, Aldrich Chan and Selvana Morcos
Neuroscience of Imagery, Cognition, and Emotion Research Lab, Carleton University, Ottawa, ON, Canada
Edited by: The relationship between vivid visual mental images and unexpected recall (incidental
recall ) was replicated, refined, and extended. In Experiment 1, participants were asked
to generate mental images from imagery-evoking verbal cues (controlled on several verbal
Reviewed by:
Joel Pearson, The University of New properties) and then, on a trial-by-trial basis, rate the vividness of their images; 30 min later,
South Wales, Australia participants were surprised with a task requiring free recall of the cues. Higher vividness rat-
Giorgio Ganis, Plymouth University, ings predicted better incidental recall of the cues than individual differences (whose effect
UK
was modest). Distributional analysis of image latencies through ex-Gaussian modeling
*Correspondence:
showed an inverse relation between vividness and latency. However, recall was unrelated
Amedeo D’Angiulli , Neuroscience of
Imagery, Cognition, and Emotion to image latency. The follow-up Experiment 2 showed that the processes underlying trial-
Research Lab, Carleton University, by-trial vividness ratings are unrelated to the Vividness of Visual Imagery Questionnaire
1125 Colonel By Drive, Room 1316 (VVIQ), as further supported by a meta-analysis of a randomly selected sample of relevant
Dunton Tower, Ottawa, ON K1S 5B6,
literature. The present findings suggest that vividness may act as an index of availability
Canada.
e-mail: amedeo@connect.carleton.ca of long-term sensory traces, playing a non-epiphenomenal role in facilitating the access of
those memories.
Keywords: episodic memory, incidental recall, multi-trace theory, visual imagery, vividness, VVIQ
INTRODUCTION with other variables, or did not appropriately interpret the valid-
People often report they experience vivid spontaneous visual men- ity criteria by anchoring the vividness construct to models of
tal images in situations in which they have to recall something they memory and verbal report underlying processes. This is a situ-
did not expect to recall (incidental recall). Early imagery studies ation analogous to the one denounced years ago by Ericsson and
revealed that the spontaneous and involuntary appearance of a Simon (1980) in the context of models of verbal reports, instru-
vivid visual mental image consistently occurred in response to cer- ments such as vividness ratings/scale/questionnaires seem to be
tain memory conditions and tasks involving incidental recall. For used in a brute empirical fashion, without considering a satisfac-
example, upon asking subjects to remember the type of breakfast tory a priori theory of the processes involved in the measurement
one had in the morning (Galton, 1880), the number of windows instruments themselves. For the latter reason, it has been argued
in one’s house (Shepard, 1966) or to verify a property of an expe- that there has also been confusion between issues of validity (e.g.,
rienced event with no aid of a current percept (Goldenberg et al., discriminant or construct) and issues of reliability (e.g., specificity
1992) individuals often report vivid images. In such context, vivid- and precision). In the context of these challenges, the measure-
ness is traditionally defined as a construct expressing the self-rated ment of vividness has been hotly debated. As Pearson (1995)
degree of richness, amount of detail (resolution), and clarity of points out, vividness is usually measured using the Vividness of
a mental image, as compared to the experience of actual seeing Visual Imagery Questionnaire (VVIQ) or its updated version, the
(D’Angiulli and Reeves, 2007). Although vividness correlates with VVIQ2 (Marks, 1995). However, these are not ideal measures for
performance on certain memory tasks (Baddeley and Andrade, the experimental study of vividness per se, as they only measure the
2000), with arousal level (Barrowcliff et al., 2004; Bywaters et al., overall individual’s ability to generate vivid mental images (“trait
2004), with positive emotional valence toward a stimulus (Alter vividness”), not differences between single experiences of men-
and Balcetis, 2010), and with increased visual cortex activity (Farah tal imagery (“state vividness”). To study specific processes behind
and Peronnet, 1989; Farah et al., 1989; Sparing et al., 2002; Cui the phenomenon of vividness itself, it is more appropriate to use
et al., 2007; Cattaneo et al., 2011, 2012), any attempt to clarify its trial-by-trial self-reports in which the vividness of each individual
function and its relationship to underlying processes still presents mental image is rated immediately after its generation by the sub-
numerous challenges. ject (Begg, 1988; Hertzog and Dunlosky, 2006; D’Angiulli, 2009;
Manipulating vividness directly is difficult, and the lack of Pearson et al., 2011). The self reports were successfully employed in
converging analyses has generally led to the use of correlational several previous studies, where the findings were consistent with
approaches that examine vividness predominantly as an index of both VVIQ research and new results outside the VVIQ’s realm
individual differences in the ability to generate mental images. of individual differences, which demonstrates that it is a reason-
Furthermore, many preceding studies either confounded vividness ably robust measure (D’Angiulli, 2002, 2008, 2009; D’Angiulli and
www.frontiersin.org February 2013 | Volume 4 | Article 1 | 51

D’Angiulli et al. Image vividness and unexpected recall
Reeves, 2002, 2007; Alter and Balcetis, 2010; Rabin et al., 2010; Pear- whether the observed effects were discriminatively and specifi-
son et al., 2011). Despite these successes, so far there has been no cally linked to recall processes (refer to Sheehan, 1973, for one
clear empirical evidence showing exactly why trial-by-trial vivid- exception). Furthermore, individual differences were often glob-
ness reports should be considered more informative and reliable ally defined by the VVIQ, such that “good” versus “poor” imagers
than the VVIQ. Do these sets of verbal reports reflect different or determined “high” versus “low” vividness, respectively. Finally, the
overlapping processes? lack of control for factors relating to the cued words themselves was
Many of the mentioned challenges could be mitigated by devel- a consistent problem in previous research. In the present research,
oping a model of processes underlying trial-by-trial vividness a direct imagery and incidental recall paradigm were used, and sev-
self-reports in visual mental image generation tasks, as opposed eral verbal properties were controlled for (age of word acquisition,
to just VVIQ measurement. One of the goals of the model should word familiarity/frequency, imageability, and concreteness).
be to clarify the non-epiphenomenal role of the subjective vivid- We compared two hypothetical cognitive components of men-
ness experience, a fundamental and difficult issue that continues tal image generation from verbal descriptions, which possibly
to elude research efforts. An opportunity to gain some upper hand could account for the outcomes of Experiment 1. If the relation-
may be offered by conditions in which vivid imagery influences ship between vividness and unexpected recall were contingent
incidental recall in example situations such as the one mentioned upon shared processing while encoding the cues in the study
earlier. The link between vividness and incidental recall was first phase (image generation), a possible relationship may be explained
suggested long ago (Richardson, 1969; Paivio, 1971) but the best by depth of elaboration (Craik and Lockhart, 1972; Eysenck and
evidence comes from studies showing that self-reported vivid- Eysenck, 1980). The more time spent elaborating the imagined
ness is related with incidental recall of imagery-evoking verbal material, the more subjectively vivid the material should be. Sub-
cues (Sheehan and Neisser, 1969; Sheehan, 1971, 1972b, 1973). In sequently, this should lead to better retention and recall in the test
a typical paradigm devised by Sheehan (1972a), “vivid imagers” phase (free incidental recall). The main predictions derived from
and “non-vivd imagers,” as defined by the VVIQ, were either this hypothesis were that: (1) a direct relationship between image
intentionally or incidentally instructed to recall concrete (high latency and incidental recall should exist, as should a relationship
imagery-evoking) and abstract (low imagery-evoking) words. between incidental recall and self-rated vividness; (2) however,
Results showed that vivid imagers recalled concrete words sig- the correlation between vividness and incidental recall should be
nificantly better in the incidental than in the intentional recall accounted for by image latency. Therefore, the correlation between
condition; whereas recall of abstract words was similarly poor in vividness and incidental recall should be non-significant and/or
both conditions. correspond to a small effect size when image latency would be
In another line of research, Neisser and Kerr used objec- controlled for.
tive methods of mnemonic effectiveness and response time to A possible alternative based on neurocognitive considerations is
study the spatial properties of visual imagery (Neisser and Kerr, that vividness ratings rely on an index of the availability of multiple
1973; Keenan and Moore, 1979; Kerr and Neisser, 1983). They sensory traces in long-term memory (Hintzman and Block, 1971).
asked the subjects to construct images in three different condi- Thus, because the strength of vividness would reflect the magni-
tions according to presented sentences describing two objects in tude of the networks of sensory traces consolidated from episodic
a given reciprocal spatial relation (concealed, next to/“pictorial,” memory (Morris and Hampson, 1983; Rabbitt and Winthorpe,
far from/“separate”) and measured incidental recall rates of target 1988), higher vividness ratings should be associated with better
verbal cues. Visual images acted as mnemonics in the concealed incidental recall performance (higher likelihood of accessing long-
condition as well as the “pictorial” condition. If the procedure term traces). This model would also predict that the relationship
changed subtly and intentional learning was used instead, the between vividness and incidental recall can be partly explained
objects in the concealed condition were recalled no better than by individual differences in participants’ ability to access long-
the separate condition. The data from these experiments also term memory sensory information based on the prior estimate of
showed that concealed images were less vivid than pictorial images, availability supported by vividness judgments. The latter aspect
and response time was longer for less vivid images. Although could be conceived as a “meta-imagery” contribution, where the
instruction for imagery/recall had an effect on imagery vividness, vividness judgment may reflect “a judgment of the richness of
incidental recall was invariably found to predict vividness even the current image combined with an estimate of the additional
in studies that attempted to falsify Neisser and Kerr’s findings sensory information that could be incorporated, should the task
(Keenan, 1983). requirements change.”(Baddeley and Andrade, 2000; p. 141). Con-
The association between vividness and incidental recall is a rel- sequently, individuals with greater metacognitive ability should
atively consistent finding across several different conditions and experience more vivid images, be more efficient and faster in gener-
manipulations, and suggests that incidental recall could be used ating images, and yield higher incidental recall than the individuals
as the benchmark variable against which alternative hypothe- who possess a reduced metacognitive ability. If greater vividness
ses on the nature of imagery vividness and its function could were related to greater incidental recall accuracy, and the rela-
be compared. Because older research had several shortcomings, tionship was not simply due to longer image latencies, then this
Experiment 1 was designed to replicate, generalize, and extend would support the hypothesis that vividness acts as an index of
said relationship. Most of those studies used global or delayed self- stored memory trace availability, and plays a non-epiphenomenal
report of vividness. In addition, image generation time was con- role in determining the likelihood of accessing such memories in
founded with vividness, and most paradigms did not clearly show long-term memory.
Frontiers in Psychology | Perception Science February 2013 | Volume 4 | Article 1 | 52

In all the following experiments, explicit instructions to gener- process, and it could not be rated if an image does not reach to
ate mental images was adopted as this manipulation has proven conscious awareness, all cases rated “no image” were eliminated
to be perhaps the most reliable and most direct way to ensure that from our analysis.
participants are actually generating mental images, as shown by
converging evidence from hundreds of studies showing that the PROCEDURE
report of having an image at request is associated with behav- The protocol for Experiment 1 was approved by the Carleton
ioral, neural, or clinical neuropsychological indices. In addition, University Research Ethics Board.
while direct interference of imagery on low-level perception is Image generation phase
an established phenomenon (Craver-Lemley and Reeves, 1987), Participants were seated facing a computer monitor and pressed
the opposite effect, direct interference of low-level perception on the right mouse button to begin each trial. Upon clicking the
imagery, is either weak and ubiquitous (see D’Angiulli, 2002) or is mouse, an alerting beep was sounded, followed 250 ms later by the
based again on introspective reports (as in Baddeley and Andrade, display of a noun-cue at the center of the screen. Participants were
2000). Therefore, the latter manipulations are no better or different instructed to read the cue silently and as quickly as possible. They
than the ones we used for verifying the employment of imagery. were immediately asked to generate an image that corresponded to
the noun-cue. Participants were required to press the right mouse
EXPERIMENT 1 button again when they considered their image to be complete,
METHOD AND MATERIALS and at its most vivid.
Participants Upon pressing the button, another alerting beep was sounded,
Participants were 26 first-year university students age range = 17– followed 250 ms later by a horizontal array of seven choices appear-
25; 14 female and 12 male). None had participated in an imagery ing near the bottom of the screen. From left to right, each button
study before (Campos et al., 2007). Participants signed up through was labeled with one of seven vividness level descriptions in a
a subject pool within 3 weeks of beginning introductory psychol- seven-point scale format [(1), “no image”; (2), “very vague/dim”;
ogy courses, with 2% credit toward their final grade used as incen- (3), “vague/dim”; (4), “not vivid”; (5), “moderately vivid”; (6),
tive. No significance was found for gender or age against any fac- “very vivid”; and (7), “perfectly vivid”], as in Marks (1995). Time
tors, so these variables were dropped from further consideration. was taken to familiarize participants with the rating system during
pre-test practice sessions. Participants used the mouse to click on
Stimuli one of these seven buttons, and were instructed to rate any failure
A body of 60 verbal description-cues from previous research to generate an image as a “no image.” There was no deadline for
(D’Angiulli and Reeves, 2002; available in D’Angiulli, 2001a) were their response.
matched with regards to noun or compound word frequency, Following the vividness response, the array of buttons disap-
imageability, concreteness, and reading time. These cues included peared and the display reverted back to a screen instructing the
single-noun and double-noun descriptions comprising both ani- participant to click the mouse when they were ready to begin the
mate (e.g., dog, cat) and inanimate objects (e.g., car, bottle). The next trial. In an effort to minimize imagery persistence between
present data showed no significant differences between the two trials, stimuli were presented in random order with a minimum
subsets of stimuli in terms of vividness or latency of elicited inter-trial interval of 5 s (Craver-Lemley and Reeves, 1987). Par-
imagery. Secondary analyses indicated that these descriptions were ticipants were not informed that latency times were covertly mea-
rated as emotionally neutral, with negligible inter-item variabil- sured. Button presses were justified as a means to signal a complete
ity along a simple emotional rating scale (D’Angiulli, 2001b). In image, which was ready to be rated, and prompt the appearance
addition, the 10 noun-cues were selected from earlier research of the vividness scale buttons.
(Paivio et al., 1968) to use as buffer items during the incidental
recall phase of the experiment (i.e., to filter out recency and pri- Free incidental recall phase
macy effects during recall). The 60 cues were presented in random After completing the image generation phase, participants took a
order, preceded by five buffer noun-cues and followed by five other 20 min break. Afterward, they were asked to return to the lab to
buffer noun-cues (which were presented in a fixed order). fill out additional paperwork, to receive course credit, and com-
Stimuli properties previously shown to intercorrelate were con- plete the debriefing process. Prior to the image generation phase,
trolled for. Verbal cues with higher concreteness levels were shown participants had not been informed that they would be required
to be recalled at significantly higher rates (Paivio, 1971), as were to recall any of the stimuli. Upon their return, precisely 30 min
high frequency words (e.g., Miller and Roodenrys, 2009). Image- from the end of the image generation phase, they were asked to
ability, which refers to how easily a mental image can be generated complete the incidental recall task, wherein they were required to
from a word, has been correlated with concreteness (Tse and Altar- recall and record as many of the previously read descriptions as
riba, 2007). Age of acquisition, which refers to the average age a possible.
word enters a subject’s lexicon was indirectly controlled for, as it Each phase of the experiment was exclusively conducted by
is highly correlated with both imageability (Ma et al., 2009) and one of two paid undergraduate research assistants. Both research
concreteness (Barry and Gerhand, 2003). The well-validated MRC assistants received training in their module, yet remained naïve
Psycholinguistic Database (Clark, 1997) was used to ensure the to the purposes and hypotheses of the study. Final debrief-
words used for cuing had approximately the same scores on these ing was conducted through an exit interview with the principal
factors. Because it was assumed that vividness is an image-specific investigator.

RESULTS AND DISCUSSION Table 1 | Results of the ex-Gaussian fit to empirical image latency
Preliminary analyses were conducted on the empirical distribu- distributions in unconstrained image generation phase of Experiment
tions of raw response times (RTs) for each level of vividness (except 1 (see text for details).
level 1 = “no image”). A total of 1490 valid observations were avail-
Vividness λ µ r2 MRT SDRT N
able after all cases with a rating of “no image” (5% of total trials)
were removed. Data were binned using the smallest increment that 2 18046.0 1500.0 0.68 19547.2 26055.7 17
did not make the histograms appear too irregular. From the initial 3 8807.0 2500.0 0.95 11307.2 14520.5 31
binning it became apparent that our RT data could be fitted by 4 18641.0 5000.0 0.95 23641.2 28188.5 64
an ex-Gaussian – that is, the convolution of an exponential with 5 12027.0 2500.0 0.96 14527.3 16280.8 174
a Gaussian. This ex-Gaussian model has been used successfully 6 11612.0 2500.0 0.96 14112.3 15416.4 328
in several experimental paradigms (for reviews, see Ratcliff, 1979, 7 8162.0 5000.0 0.99 12162.0 18938.2 449
1993; McNicol and Stewart, 1980; Luce, 1986) to fit explicit the-
oretical distribution functions and to give convenient summary All values reported in the table – except the ones corresponding to n’s – are in
of empirical RT distributions. The assumption of the ex-Gaussian ms; 500 < σ < 1000. Robust regression with Ramsay’s weighting function.
model is that RT is the sum of two other random variables, one
distributed as a Gaussian and one distributed as an exponential
(Luce, 1986). Previous work (D’Angiulli and Reeves, 2002) has
supported the hypothesis that the ex-Gaussian model reflects the
time to retrieve images from memory so that “image generation”
can be essentially reduced to “retrieving images from memory.”
Therefore, variations in each of ex-Gaussian parameters across
vividness levels could be assumed to describe the core underly-
ing generative processes common to both imagery and incidental
recall. The ex-Gaussian model was fitted using a robust regression
method due to Hoaglin et al. (1983).
To ensure the ex-Gaussian reflected the shape of the group
data, and the shape of the individual data, the model was first
vincentized for individual data, and then averaged over vivid-
ness levels. Histograms were constructed by pooling the raw RTs
from each vividness level over subjects, irrespective of the individ-
ual source of the RTs. This method has been used in situations
where there are too few trials for single subjects (see Ratcliff,
1979). We verified whether the related observations were seri-
ally independent and not autocorrelated for each subject, if so we
could assume independence of collective observations (see Neter
et al., 1996). In our case, the Durbin–Watson autocorrelation test
statistic D clearly exceeded the upper bound in the assessment
of each subject [du > 1.62; α = 0.05; n = 60; lag = 1] as well as
for each vividness level submitted to fitting, thereby showing no FIGURE 1 | Ex-Gaussian model-fit of RT distributions for images rated
autocorrelation. with vividness 2–4 and 5–7 in Experiment 1.
Table 1 shows the ex-Gaussian fit to the distribution histograms
of RTs obtained for each vividness level. For each distribution, the
ex-Gaussian fit explained at least 68% of the variance associated to point out the enormous variability in the response latencies,
with RTs. The general distribution of the vividness data showed and that the relationship between vividness could not be easily
the median rating was a value of 4 (“non-vivid”). Examination of guessed by naïve participants. Therefore, it is rather implausible
each vividness level regressed onto RTs showed both distributions that the observed pattern might be due to response-bias based on
were best summarized by piecewise linear regressions of oppo- an explicit or conscious criterion-shift, or set of decisions, since
site slope. These data supported a clear split between vivid (rating this would have required the participants to first tacitly simulate
values 5–7) and non-vivid (2–4) observations. the ex-Gaussian model, and then retrofit their responses coher-
The Gaussian of both vivid images (levels 5–7) and less vivid ently to the model to produce the observed pattern. Because this
images (levels < 4) are reported in Figure 1. Both distributions would have to be done uniformly by all participants, the variability
have comparable standard deviation, as evidenced by the left tail should have been much more contained than what we observed.
of the distributions. However, the distribution of less vivid images The key analysis examined the predictability of recall and
is delayed >500 ms, as evidenced by the shift on the time axis. RTs from vividness rating category (non-vivid versus vivid). In
Consistent with previous findings (D’Angiulli and Reeves, 2002), an effort to meet assumptions for parametric procedures and
more vivid images were typically associated with shorter Gaussian augment robustness to violations, the distribution of RTs was
latency components than were less vivid images. It is important normalized with a logarithmic transformation, after which no

FIGURE 2 | Mean proportion of incidental recall for verbal cues

corresponding to vivid (rating: 5–7) and non-vivid (rating: 2–4) images
in Experiment 1.
FIGURE 3 | Mean image generation time for verbal cues corresponding

to vivid (rating: 5–7) and non-vivid (rating: 2–4) images in Experiment 1.
multivariate outliers were detected. Figure 2 shows the within-
subject mean proportion of incidentally recalled imagery-evoking
verbal cues presented during the image generation phase against
the rated vividness level. Figure 3 shows the within-subjects mean statistically significant predictive model for recall (see Appen-
RTs of image generation against the rated vividness level (for pre- dix) that accounted for 28.3% of the variance in incidental
sentation, RT data are expressed as seconds, derived from antilog recall. No change was observed if the model was fit to pre-
transformation). The proportion of recalled cues corresponding dict recall when response time was added as a predictor [χ2 (65,
to vivid images was 0.77 (SE = 0.05), whereas the proportion 1490) = 389.437, p < 0.001]. RT did not exert an influence on the
of recalled cues corresponding to non-vivid images was 0.19 model (B = −0.002, p = 0.587), which further confirmed the null
(SE = 0.04). A paired samples test showed the difference to be sig- effect of RT on recall. Therefore, vividness could not account
nificant [t (25) = 6.69; p < 0.0001], explaining 74% of the variance. for recall accuracy simply because participants spent more time
In contrast, the mean RTs for vivid (14.8 s, SE = 2.71) and non- imagining the items corresponding to the verbal cues.
vivid (13.33, SE = 2.14) cues did not differ [t (25) < 1, p = 0.34; A linear mixed model was fit to the data to assess the contribu-
R 2 < 0.01]. tion of stimulus and RTs to linear change in vividness of imagery.
A linear regression analysis examining the effect of individual The variables in the model were evaluated by a Type III test. Since
differences on the total number of images recalled showed that the sample size was not large, Restricted Iterative Generalized Least
14% of the variance in incidental recall accuracy was explained by -squares (RIGLS) was used (Goldstein, 1986). Stimuli and RT had
participants’ average vividness rating [F (1, 25) = 4.05, MSe = 0.38, a significant effect on vividness [F (59, 1000) = 1.59, η2 = 0.086,
p = 0.05]. Therefore, the role of individual differences was modest p < 0.05], and F (1, 1103) = 5.17,η2 = 0.005, p < 0.05, respectively.
and its effect size (r) was significantly smaller than that of vividness Therefore, because the effects were small, RTs and stimuli influ-
described earlier (0.86 versus 0.37, z = 3.07, p = 0.002). enced vividness only minimally. There was no interaction between
A two-predictor model (stimulus and vividness) was fit- stimuli and RTs (F < 1).
ted to the data to test the hypothesis regarding relationship To determine if recall and vividness ratings were affected by the
between vividness and recall. Stimulus was plotted as a nom- verbal properties of the word stimuli that were not kept constant
inal factor, in which each category was a noun-cue. It was during stimulus selection, correlation analyses were conducted on
included as a predictor to ensure vividness effects were not age of acquisition, and familiarity versus recall. No significant rela-
due to the tendency for some words to produce more vivid tionship was found between the percentage of participants that
images than others. The resulting model [Predicted logit of recalled a cue, and either age of acquisition (r = 0.213, p = 0.317)
(Recall) = 0.664 + β1∗Vividness + β2∗Stimuli ] was statistically reli- or familiarity scores (r = 0.118, p = 0.445). In addition, effects of
able, χ2 (62, 1441) = 340.969, p < 0.001 (see Appendix A for analy- stimuli regressed onto vividness, recall and RTs all explained less
sis details). According to the model, greater vividness ratings than 0.5% of the variance.
for noun-cues predicted recall with an overall success rate of The results of Experiment 1 implicate vividness ratings as a
72.2%. The model correctly classified 83.7% of unrecalled cues predictor of incidental recall for imagery-evoking cues. The effect
and 54.7% of recalled cues. Stimulus and vividness generate a of individual differences in imaging ability on incidental recall was

much smaller than the effect of vividness. Because image latency static scenes, but should not relate to perceived imagery latency
was unrelated to incidental recall, and inversely related to vivid- when the effects of vividness are removed.
ness, these data were incompatible with the depth of elaboration Conversely, if the VVIQ2 accounts for most of the relation-
account. Because the effect of vividness on incidental recall for ship between trial-by-trial vividness ratings and perceived imagery
verbal cues was tested, the influence of expectancy and demand latency, then vividness judgments could be attributed to similar
characteristics were minimized. These results support the valid- individual metacognitive skill differences involved in the two types
ity of vividness as a measurable construct, and as an entity which of vividness measures (Baddeley and Andrade, 2000; Pearson
may represent real underlying memory processes. Vividness rat- et al., 2011). However, because dynamic mental imagery capac-
ings likely reflect a process which provides a natural mnemonic for itates working memory more than static mental imagery, fewer
unexpected retrieval of implicitly coded information (see Kosslyn resources are available for concurrent metacognitive processes.
et al., 2006). Then, under such circumstances one would expect less vivid
images for dynamic scenes than static ones.
EXPERIMENT 2
Although Experiment 1 did not include a measure of VVIQ, inci- METHODS AND MATERIALS
dental and intentional recall has traditionally shown a modest Participants
correlation with the VVIQ and VVIQ2, with average effect sizes Participants were 44 female undergraduate students (age range:
generally of about r = 0.13 (see McKelvie, 1995; Dean and Mor- 18–25). Participants signed up through a subject pool, with 2%
ris, 2003). More recent evidence suggests the relationship between credit toward their final grade used as incentive. All participants
VVIQ2 and trial-by-trial vividness ratings is weak to moderate had normal or corrected-to-normal vision, and no reported or
(r < 0.20) (D’Angiulli, 2001a; D’Angiulli and Reeves, 2007). Also, documented learning disabilities. Participation required the atten-
the patterns of results from Sheehan (1971, 1972b) suggest the dance of two appointments. The first appointment was a prelimi-
quality of imagery is contingent upon properties of the stimuli nary screening session, where participants filled out the VVIQ2
within the setting of each trial, and predicts incidental free recall and individual data. The second appointment was the experi-
and recognition performance. Lastly, other studies found the mod- mental session. Five potential participants were excluded from the
est correlation between trial-by-trial ratings and VVIQ holds only experiment, as they were unable to evoke the images as required.
for female participants (Sheehan, 1971, 1973).
In contrast with these findings, Pearson et al. (2011) reported Materials
large predictive effects of both trial-by-trial vividness ratings and An adaptation of 17 static and 17 dynamic scene descriptions were
VVIQ2 scores when related to bias in reporting a dominant pattern used (Baddeley and Andrade, 2000; Experiment 4, see Appendix A,
during a binocular rivalry task. The underlying assumption was p. 144). The scenes were adapted such that words including British
that similar metacognitive processes (i.e., knowing how and what content (e.g., Big Ben) were substituted with equally long words
the observer knows about his/her own processes of visual mental describing North American content (e.g., CNN Tower) which were
imagery) would be used in trial-by-trial vividness ratings and in validated through pilot experiments. During the screening phase,
VVIQ2. If this assumption is correct, the overlapping processes a question from the visual portion of the procedure for assessing
could shed some light on the results of our Experiment 1. One expectations on the vividness of imagery was asked (Baddeley and
interpretation of the results of Experiment 1 is that trial-by-trial Andrade, 2000; see Appendix C, Q2, Question 2, p. 145). After the
vividness ratings may be accounted for by the same metacogni- experimental phase, a tacit knowledge assessment procedure was
tive judgment processes involved in responding to the VVIQ2. administered.
Experiment 2 was designed to examine the putative relationship
between vividness ratings and VVIQ2. If the association between PROCEDURE
the VVIQ2 and vividness ratings were confirmed in Experiment The protocol for Experiment 2 was approved by the Carleton
2, then one may also explain the basis through which vividness University Research Ethics Board.
ratings could predict incidental recall in terms of the overlapping Participants were given instructions, and 10 min of practice
metacognitive processes involved in the VVIQ2. with five dynamic and five static imagery scenes. Between each
The design of Experiment 2 was a variation of the paradigm practice trial, participants were required to report how well they
used by Baddeley and Andrade (2000). Upon completing the could control each image. Only participant ratings with vividness
VVIQ2, female participants were asked to read a short description greater than “extremely slow” (1) for 80% of the practice trials
of a static or dynamic scene, and press a key upon generating com- qualified for the entire experiment. One participant was elimi-
plete visual mental image. Participants then rated the vividness and nated from the initial pool under such criteria. Upon completing
the subjectively perceived latency of the image on a trial-by-trial the practice session, participants verbally repeated the instructions
basis. If, as the results of Experiment 1 would suggest, vividness to the experimenter to ensure the instructions were understood.
ratings are based on an index of multiple sensory traces available in Participants were instructed to silently read a description of a
long-term memory, this account would predict: (1) higher trial- dynamic or static scene displayed on a computer screen, which
by-trial vividness ratings for dynamic scenes than static scenes, occurred 250 ms after an alerting beep. The experiment consisted
and (2) a negative (i.e., inverse) relationship between trial-by-trial of 17 dynamic, and 17 static descriptions. Participants were tested
vividness and perceived imagery latency. The VVIQ2 should cor- individually, and the procedure lasted approximately 40 min.
relate with trial-by-trial vividness ratings from both dynamic and Upon reading each description, participants were required to press

a key to indicate the description was understood. Participants were differed from Baddeley and Andrade’s findings (Experiment 4).
instructed to imagine the description with their eyes open, and as (They found dynamic imagery was significantly less vivid than
seen from the front. Outline drawings were shown as examples static imagery). Images produced for the VVIQ2 were signifi-
before the experiment began. Each description was presented in cantly more vivid (M vviq2 = 5.68, SDvviq2 = 0.62) than vividness
random order with an inter-trial interval of 5 s (Craver-Lemley for static images [t (38) = 3.88, p < 0.0001], and dynamic images
and Reeves, 1987). Upon forming a complete mental image, par- [t (38) = 2.83, p < 0.01].These data may be interpreted as evidence
ticipants were required to press a button on a mouse. Four seconds that participants were generally much more confident in their
after the button press, participants were shown buttons to rate per- imagery abilities than what they were capable of demonstrat-
ceived vividness, and perceived latency of the images. Participants ing during the experimental procedure. The discrepancy between
were asked to rate their image as “complete” or “finished” when the trial-by-trial vividness level and VVIQ2 imply a lack of agree-
image was maximally clear and detailed (see Cocude and Denis, ment between metacognitive judgment as measured through the
1988). Participants were required to rate their mental image as VVIQ2, and verbal reports specific to the actual imagery task.
they had experienced it at the time of the key press. There was no Table 2 shows correlations among all measures. VVIQ2 was sig-
deadline for the rating responses. nificantly correlated with vividness of static imagery, but was not
The presentation order of the scales was randomized, such related to vividness of dynamic imagery, nor perceived latency in
that vividness could follow or precede perceived imagery rating. both static and dynamic imagery conditions. A very strong inverse
The second rating task followed immediately after the first rat- relationship between trial-by-trial vividness ratings and perceived
ing response. The vividness scale consisted of a horizontal array imagery latency was observed in both static and dynamic imagery
of seven buttons appearing at the center of the screen. From left conditions, with strong to marginal evidence of the same trends
to right, each button was labeled with a short description cor- in crossed conditions.
responding to one of seven levels of the vividness scale used in Whereas vividness ratings correlated with perceived latency, the
Experiment 1. The imagery latency (speed) scale consisted of a hor- VVIQ2 did not. These data provide very weak evidence validating
izontal array of seven buttons appearing at the center of the screen. the VVIQ2, when the criterion is a self-report, subjective third
From left to right, each button was labeled with a short descrip- variable. Logically, one would not expect any predictive success
tion corresponding to one of the seven levels: from “extremely fast” of VVIQ2 in relation to a behavioral variable such as incidental
(7), to “extremely slow” (1). Valid trials were defined by vividness recall. The observed patterns were analyzed to determine if they
greater than 1. Subjects were instructed to give a “1” response could be predicted by expectations or tacit knowledge (Pylyshyn,
if they were unable to form a mental image. Upon completing 2003). There was no significant difference in the number of partic-
the experiment, participants underwent a post-experimental inter- ipants expecting vivid imagery to be less or more vivid than static
view, wherein they quickly described what they had imagined for imagery (χ2 < 1). Figure 4 describes participant responses con-
seven randomly probed descriptions from both dynamic and sta- cerning self-rated predictions about the type of relationship they
tic condition. Post-experimental interviews were concluded with expect to exist between perceived vividness and perceived imagery
the tacit knowledge assessment procedure (Baddeley and Andrade, latency, as documented during the preliminary screening session.
2000), and included the following question: Most participants predicted a positive relationship, or no rela-
“We are interested in knowing if you think that there was a tionship between vividness and imagery latency. One participant
relationship between how vivid your images were and other correctly predicted the inverse relationship. Upon removing the
factors. Please just tell us what you expect or think, please do data of this participant from the analysis, there were no significant
not use images to answer the question, we are just interested differences between results.
in what you predict or think about things that may be related In conclusion, the association between the VVIQ2 and vivid-
or may determine the vividness of your images.” ness ratings was not observed consistently in both the conditions
of Experiment 2, and if collapsed across conditions (static and
RESULTS AND DISCUSSION dynamic) the effect becomes modest and not significant. VVIQ2
To eliminate effects of discrepant scales, total scores for the
VVIQ2 were converted to mean vividness values through a
simple linear transformation. The transformation resulted in Table 2 | Correlation matrix among VVIQ2 and self-reported image
a seven-point scale; henceforth, referred to as mean vviq2. vividness ratings and perceived generation speed in dynamic and
As in Experiment 1, we considered only valid responses. The static imagery conditions of Experiment 2.
rate of excluded invalid trials was approximately 3% (level
Dyn. Stat. Dyn. Stat.
1 = “no image”), a proportion similar to Experiment 1. On
vividness vividness speed speed
average, images were reported as moderately vivid, and were
produced at a relatively fast perceived latency in both static VVIQ2 0.259 0.505** 0.044 −0.219
(M viv. = 5.31, SDviv. = 0.55; M speed = 5.40, SDspeed = 0.39) and Dyn. vividness – 0.679** −0.626** −0.652**
dynamic (M viv. = 5.29, SD = 0.77; M speed = 5.59, SDspeed = 0.62) Stat. vividness – – −0.282† −0.531**
conditions. Paired samples contrasts showed dynamic imagery was Dyn. speed – – – 0.622**
perceived as significantly faster than static imagery [t (38) = 2.52,
p < 0.025]. However, mean vividness ratings did not differ Dyn., dynamic imagery condition; Stat., static imagery condition. N = 39.
between the two conditions [t(38) < 1, p = 0.797]. The latter result †
p < 0.10, **p < 0.01.

trial-by-trial vividness ratings, they would not likely be the same

ones underlying VVIQ measures. Taken together the results of
Experiment 1 and Experiment 2 are consistent with those observed
in a meta-analysis we conducted, representing 5% of the litera-
ture pertaining to “vividness” and “VVIQ” (reported in Appendix
B). The proportion of significant and non-significant experimen-
tal outcomes for trial-by-trial vividness ratings and VVIQ factor
effects were calculated. For behavioral, cognitive, and neural mea-
sures, a greater number of significant experimental outcomes
accompanied trial-by-trial vividness ratings than the VVIQ. Fur-
thermore, the correlation between VVIQ scores and trial-by-trial
vividness ratings for 21 entries showed an average correlation of
0.15, and variability in these values ranged from r = −0.27, to
r = 0.64. Consistent with the results of experiment 2, these addi-
tional results support the contention that trial-by-trial vividness
self-reports and VVIQ scores share some descriptive properties
of visual imagery. However, trial-by-trial vividness ratings seem to
resolve the construct of mental imagery with much greater reliabil-
FIGURE 4 | Percentages of participants predicting what type of
ity. Although metacognitive processes may be occurring in single
relationship they tacitly think there should be between vividness
ratings and speed of imagery in Experiment 2.
trial judgment, it is perhaps more parsimonious to assume that
vividness ratings are mostly a form of Level 2 retrospective verbal
reports (Ericsson and Simon, 1993).
also failed to validate against a third self-report criterion variable Considered as retrospective verbal reports, vividness ratings
(perceived image latency). If the VVIQ2 assesses individual differ- may be based on a direct translation of residual top-down sen-
ences in metacognitive ability, it seems implausible that such abili- sory traces available in long-term memory (D’Angiulli and Reeves,
ties would predict incidental recall. Because trial-by-trial vividness 2002), wherein vividness intensity is proportional to the mag-
predicted incidental recall, the metacognitive aspects assumed to nitude of sensory traces available. This statement agrees with
be reflected by VVIQ2 do not appear to influence vividness and a number of neurocognitive considerations borne out of MMT
the mental imagery process to a significant degree. research. According to that theoretical framework, each sensory
trace is distributed across the cortex, such that various distributive
GENERAL DISCUSSION patterns are unique to a specific sensory input, and is distinct from
Despite controlling for imageability, concreteness, age of acquisi- all other distributive patterns (Hintzman, 1976). Sensory traces
tion, and verbal frequency/familiarity, the results from Experiment are thought to be indexed by the hippocampus (Ryan et al., 2001),
1 showed a positive relationship between vividness ratings and and integrated into a mental image by the cuneus, precuneus, and
incidental recall of imagery-evoking cues. These results are not occipital lobes (Svoboda et al., 2006; Svoboda and Levine, 2009;
consistent with depth of elaboration, as faster image generation Cabeza and St. Jacques, 2007). However, hippocampal indexing
latencies accompanied higher vividness ratings, a pattern opposite becomes less influential as each individual sensory trace is inte-
to what depth of elaboration would predict. Furthermore, because grated into cortical networks through successive (re)presentations
depth of elaboration predicts a positive correlation between inci- (Takashima et al., 2009). Mental images are consolidated neural
dental recall and image generation time, it again fails to account patterns that correspond to these “synthetic” sensory long-term
for the data from Experiment 1. traces, whose levels of interconnectedness are correlated to their
Our findings are compatible with an alternative model of perceived reportable vividness (Rabin et al., 2010).
vividness processes based on multi-trace memory theory (MMT; Our study also indicates that although the VVIQ or VVIQ2
Moscovitch et al., 2005). This model proposes that vividness rat- may very well measure an individual’s ability to generate vivid
ings are based on an index of the availability of multiple sensory mental images (“trait vividness”), it likely lacks the resolution
traces in long-term memory, the strength of vividness reflect- to measure an individual’s ability to experience vivid mental
ing the magnitude of the networks of sensory traces that have images in specific situational contexts (“state vividness”). To study
been consolidated from episodic memory. This is described by the specific processes behind the phenomenon of vividness itself
inverse relationship between vividness ratings and image latency (rather than “trait vividness”), it is perhaps more appropriate to
(the “vivid-is-fast” relation). Thus, higher vividness ratings are use trial-by-trial self-reports, wherein vividness is rated imme-
associated with higher likelihood of incidental recall, as shown by diately after its generation (Begg, 1988; Hertzog and Dunlosky,
the data of Experiment 1. 2006; D’Angiulli, 2009; Pearson et al., 2011). Such self-reports
The follow-up results observed in Experiment 2 showed that have met with compounding success progressing beyond the
individual differences, as measured by the VVIQ2, are not a viable VVIQ’s realm of individual differences, while remaining generally
account for the relationship in Experiment 1 between vividness consistent with it. Vividness ratings demonstrate the reasonably
and incidental recall. Most important, the results of Experiment robust nature of self-reports as a measure of “state” and “trait”
2 also suggest that if there were metacognitive aspects involved in vividness (D’Angiulli, 2002; D’Angiulli and Reeves, 2002, 2007;

D’Angiulli, 2009; Alter and Balcetis, 2010; Rabin et al., 2010; Pear- Therefore, we conclude, in certain conditions conscious phenom-
son et al., 2011). This particular issue is critical given the recent enological experience associated with imagery does not have a
resurgence of use of the VVIQ in cognitive neuroscience – espe- trivial role as it can have a critical influence on recall performance.
cially in the realm of neuroimaging (Amedi et al., 2005; Palmiero
et al., 2010). ACKNOWLEDGMENTS
In summary, we found that trial-by-trial vividness ratings pre- We thank Adam Reeves and the late Ulric Neisser for comments
dict incidental recall, and the relationship cannot be attributed on earlier ideas and bits of manuscripts before the complete draft
to depth of elaboration or metacognitive processes related to self- even existed. Many thanks also go to John M. Kennedy, Robert
appraisal of individual imagery ability, as measured by the VVIQ2. Kunzendorf, J. T. E. Richardson, Stuart McKelvie for comments
Our results suggest that vividness of imagery makes implicit infor- and pre-reviews of the ideas and data presented here, Judy Hall for
mation available to consciousness, and to some extent, is linked meta-analytic advice. Portions of this research were presented at
with the associative processes through which phenomenal avail- ICOM (International Congress on Memory), University of York,
ability translates into access of incidental episodic memories. UK, 2012.
REFERENCES with TMS reveals a functional over- D’Angiulli, A. (2008). Is the spotlight an imagery: event-related potentials
Alter, A. L., and Balcetis, E. (2010). lap between vision and imagery in obsolete metaphor of “seeing with during mental image generation. J.
Fondness makes the distance grow the early visual cortex. Neuroimage the mind’s eye”? A constructive nat- Cogn. Neurosci. 1, 302–316.
shorter: desired locations seem 59, 3015–3020. uralistic approach to the inspection Galton, F. (1880). Statistics of mental
closer because they seem more vivid. Cattaneo, Z., Pisoni, A., Papagno, C., of visual mental images. Imagin. imagery. Mind 5, 301–318.
J. Exp. Soc. Psychol. 47, 16–21. and Silvanto, J. (2011). Modula- Cogn. Pers. 28, 117–135. Goldenberg, G., Steiner, M., Podreka,
Amedi, A., Malach, R., and Pascual- tion of visual cortical excitabil- D’Angiulli, A. (2009). “Vividness and I., and Deecke, L. (1992). Regional
Leone, A. (2005). Negative BOLD ity by working memory: effect behavioral-specificity in imagery: cerebral blood flow patterns related
differentiates visual imagery and of luminance contrast of men- not what you’d expect,” in Proceed- to verification of high and low
perception. Neuron 48, 859–872. tal imagery. Front. Psychol. 2:29. ings of the 31st Annual Meeting of The imagery sentences. Neuropsychologia
Baddeley, A. D., and Andrade, J. (2000). doi:10.3389/fpsyg.2011.00029 Cognitive Science Society: Symmetry 30, 1081–1092.
Working memory and the vividness Clark, C. (1997). MRC Psycholin- & Vividness in Vision, eds N. Taatgen Goldstein, H. (1986). Multilevel mixed
of imagery. J. Exp. Psychol. Gen. 129, guistic Database. The University and H. van Rijn (Amsterdam, NL: linear model analysis using iterative
126–145. of Western Australia School of The Cognitive Science Society), 49. generalized least square. Biometrika
Barrowcliff, A. L., Gray, N. S., Free- Psychology. Available at: http: D’Angiulli, A., and Reeves, A. 73, 43–56.
man, T. C. A., and MacCulloch, //websites.psychology.uwa.edu.au/ (2002). Generating visual men- Hertzog, C., and Dunlosky, J. (2006).
M. J. (2004). Eye-movements reduce school/MRCDatabase/mrc2.html tal images: latency and vividness are “Using visual imagery as a
the vividness, emotional valence Cocude, M., and Denis, M. (1988). Mea- inversely related. Mem. Cognit. 30, mnemonic for verbal associa-
and electrodermal arousal associ- suring the temporal characteristics 1179–1188. tive learning,” in Imagery and Spatial
ated with negative autobiographi- of visual images. J. Ment. Imagery 12, D’Angiulli, A., and Reeves, A. (2007). Cognition: Methods, Models, and
cal memories. J. Forens. Psychiatry 89–101. The relationship between self- Cognitive Assessment, eds T. Vecchi
Psychol. 15, 325–345. Craik, F., and Lockhart, R. (1972). Lev- reported vividness and latency and G. Bottini (Philadelphia, PA:
Barry, C., and Gerhand, S. (2003). Both els of processing: a framework for during mental size scaling of John Benjamins), 259–278.
concreteness and age-of-acquisition memory research. J. Verb. Learn. everyday items: phenomenolog- Hintzman, D. (1976). “Repetition and
affect reading accuracy but only con- Verb. Behav. 11, 671–684. ical evidence of different types memory,” in The Psychology of
creteness affects comprehension in a Craver-Lemley, C., and Reeves, A. of imagery. Am. J. Psychol. 120, Learning and Motivation, ed. G.
deep dyslexic patient. Brain Lang. 84, (1987). Visual imagery selectively 521–551. Bower (New York, NY: Academic
84–104. reduces vernier acuity. Perception 16, Dean, G. M., and Morris, P. E. Press), 47–91.
Begg, I. (1988). What does the vividness 599–614. (2003). The relationship between Hintzman, D. L., and Block, R. A.
of an image tell us about the value Cui, X., Jeter, C. B., Yang, D., Mon- self-reports of imagery and spatial (1971). Repetition and memory: evi-
of imagery? J. Ment. Imagery 12, tague, P. R., and Eagleman, D. M. ability. Br. J. Psychol. 94, 245–273. dence for a multi-trace hypothesis. J.
45–56. (2007). Vividness of mental imagery: Ericsson, K. A., and Simon, H. A. (1980). Exp. Psychol. 88, 297–306.
Bywaters, M., Andrade, J., and Turpin, individual variability can be mea- Verbal reports as data. Psychol. Rev. Hoaglin, D. C., Mosteller, F., and Tukey,
G. (2004). Determinants of the sured objectively. Vision Res. 47, 87, 215–251. J. W. (1983). Understanding Robust
vividness of visual imagery: the 474–478. Ericsson, K. A., and Simon, H. A. (1993). and Exploratory Data Analysis. New
effects of delayed recall, stimulus D’Angiulli,A. (2001a). Phenomenal and Protocol Analysis: Verbal Reports as York: Wiley.
affect and individual differences. temporal aspects of visual men- Data. Cambridge, MA: The MIT Keenan, J. M. (1983). Qualifications and
Memory 12, 479–488. tal image generation: validating ver- Press. clarifications of images of concealed
Cabeza, R., and St. Jacques, P. bal reports on vividness through Eysenck, M. W., and Eysenck, M. C. objects: a reply to Kerr and Neisser.
(2007). Functional neuroimag- latency analysis. Diss. Abstr. Int. 61, (1980). Effects of processing depth, J. Exp. Psychol. Learn. Mem. Cogn. 9,
ing of autobiographical memory. 5015. distinctiveness, and word frequency 222–230.
Trends Cogn. Sci. (Regul. Ed.) 11, D’Angiulli, A. (2001b). Vividness vs. on retention. Br. J. Psychol. 71, Keenan, J. M., and Moore, R. E. (1979).
219–227. imageability: effects on word fre- 263–274. Memory for images of concealed
Campos, A., Gómez-Juncal, R., and quency and image latency. Northwest Farah, M. J., and Peronnet, F. (1989). objects: a reexamination of Neisser
Pérez-Fabello, M. J. (2007). Expe- Cognition and Memory, 3rd Annual Event-related potentials in the study and Kerr. J. Exp. Psychol. Learn.
rience in imagery and imagery Meeting, May 26, Vancouver, BC. of mental imagery. J. Psychophysiol. Mem. Cogn. 5, 374–385.
vividness. Imagin. Cogn. Pers. 27, D’Angiulli, A. (2002). Mental image 3, 99–109. Kerr, N. H., and Neisser, U. (1983).
337–348. generation and the contrast Farah, M. J., Weisberg, L. L., Mon- Mental images of concealed objects-
Cattaneo, Z., Bona, S., and Silvanto, J. sensitivity function. Cognition 85, heit, M. A., and Peronnet, F. (1989). new evidence. J. Exp. Psychol. Learn.
(2012). Cross-adaptation combined B11–B19. Brain activity underlying mental Mem. Cogn. 9. 212–221.

Kosslyn, S. M., Thompson, W., and Applied Linear Statistical Models, 4th Richardson, A. (1969). Mental Imagery. Svoboda, E., McKinnon, M., and
Ganis, G. (2006). The Case for Men- Edn. New York: McGraw-Hill/Irwin New York: Spring Publishing Com- Levine, B. (2006). The functional
tal Imagery. New York, NY: Oxford Series: Operations and Decision Sci- pany. neuroanatomy of autobiographical
University Press. ences. Rosenthal, R. (1991). Meta-Analytic memory: a meta-analysis. Neuropsy-
Luce, R. D. (1986). Response Times: Paivio, A. (1971). Imagery and Verbal Procedures for Social Research (rev. chologia 44, 2189–2208.
Their Role in Inferring Elementary Processes. New York: Holt, Rinehart ed.). Newbury Park, CA: Sage. Takashima, A., Nieuwenhuis, I. L.
Mental Organization. New York: and Winston. Ryan, L., Nadel, L., Keil, K., Put- C., Jensen, O., Talamini, L. M.,
Oxford University Press. Paivio, A., Yuille, J. C., and Madigan, nam, K., Schnyer, D., Trouard, T., Rijpkema, M., and Fernández, G.
Ma, W., Golinkoff, R., Hirsh-Pasek, S. (1968). Concreteness, imagery, et al. (2001). Hippocampal com- (2009). Shift from hippocampal to
K., McDonough, C., and Tardif, T. and meaningfulness values for plex and retrieval of recent and very neocortical centered retrieval net-
(2009). Imageability predicts the age 925 nouns. J. Exp. Psychol. 76, remote autobiographical memories: work with consolidation. J. Neurosci.
of acquisition of verbs in Chinese 1–25. evidence from functional magnetic 29, 10087–10093.
children. J. Child Lang. 36, 405–423. Palmiero, M., Nakatani, C., Raver, D., resonance imaging in neurologi- Tse, C., and Altarriba, J. (2007). Test-
Marks, D. F. (1995). New directions for Belardinelli, M., and Leeuwen, C. cally intact people. Hippocampus 11, ing the associative-link hypothesis
mental imagery research. J. Ment. (2010). Abilities within and across 707–714. in immediate serial recall: evidence
Imagery 19, 153–167. visual and verbal domains: how spe- Sheehan, P. W. (1971). The role of from word frequency and word
McKelvie, S. J. (1995). Vividness cific is their influence on creativity? imagery in incidental learning. Br. J. image ability effects. Memory 15,
of Visual Imagery: Measurement, Creat. Res. J. 22, 369–377. Psychol. 62, 235–243. 675–690.
Nature, Function & Dynamics. New Pearson, D. G. (1995). The VVIQ and Sheehan, P. W. (1972a). The Function
York, NY: Brandon House. cognitive models of imagery: future and Nature of Imagery. Oxford: Aca- Conflict of Interest Statement: The
McNicol, D., and Stewart, G. W. (1980). directions for research. J. Ment. demic Press. authors declare that the research was
“Reaction time and the study of Imagery 19, 167–170. Sheehan, P. W. (1972b). Role of imagery conducted in the absence of any
memory,” in Reaction Times, ed. A. T. Pearson, J., Rademaker, R. L., and Tong, in incidental learning: replication commercial or financial relationships
Welford (London: Academic Press), F. (2011). Evaluating the mind’s eye: and extension of an effect. J. Exp. that could be construed as a potential
253–307. the metacognition of visual imagery. Psychol. 95, 226–228. conflict of interest.
Miller, L., and Roodenrys, S. (2009). Psychol. Sci. 22, 1535–1542. Sheehan, P. W. (1973). Stimulus imagery
The interaction of word frequency Pylyshyn, Z. W. (2003). Seeing and Visu- effect and the role of imagery in inci- Received: 02 April 2012; accepted: 02 Jan-
and concreteness in immediate serial alizing: It’s not What you Think. dental learning. Aust. J. Psychol. 25, uary 2013; published online: 04 February
recall. Mem. Cognit. 37, 850–865. Cambridge, MA: MIT Press. 93–102. 2013.
Morris, P. E., and Hampson, P. J. (1983). Rabbitt, P., and Winthorpe, C. (1988). Sheehan, P. W., and Neisser, U. (1969). Citation: D’Angiulli A, Runge M,
Imagery and Consciousness. London: “What do old people remember? The Some variables affecting the vivid- Faulkner A, Zakizadeh J, Chan A
Academic Press. Galton paradigm reconsidered,” in ness of imagery in recall. Br. J. Psy- and Morcos S (2013) Vividness of
Moscovitch, M., Rosenbaum, R. S., Practical Aspects of Memory: Cur- chol. 60, 71–80. visual imagery and incidental recall
Gilboa, A., Addis, D. R., Westma- rent Research and Issues, eds M. Shepard, R. N. (1966). Learning and of verbal cues, when phenomenological
cott, R., Grady, C., et al. (2005). M. Gruneberg, P. E. Morris, and recall as organization and search. J. availability reflects long-term memory
Functional neuroanatomy of remote R. N. Sykes (New York: Wiley), Verb. Learn. Verb. Behav. 5, 201–204. accessibility. Front. Psychology 4:1. doi:
episodic, semantic and spatial mem- 301–307. Sparing, R., Mottaghy, F. M., Ganis, G., 10.3389/fpsyg.2013.00001
ory: a unified account based on Rabin, J. S., Gilboa, A., Stuss, D. T., Thompson, W. L., Topper, R., Koss- This article was submitted to Frontiers in
multiple trace theory. J. Anat. 207, Mar, R. A., and Rosenbaum, R. S. lyn, S. M., et al. (2002). Visual cortex Perception Science, a specialty of Frontiers
35–66. (2010). Common and unique neural excitability increases during visual in Psychology.
Mosteller, F., and Bush, R. R. (1954). correlates of autobiographical mem- mental imagery – a TMS study in Copyright © 2013 D’Angiulli, Runge,
“Selected quantitative techniques,” ory and theory of mind. J. Cogn. healthy human subjects. Brain Res. Faulkner, Zakizadeh, Chan and Morcos.
in Handbook of Social Psychology, Neurosci. 22, 1095–1111. 938, 92–97. This is an open-access article distributed
ed. G. Lindzey (Cambridge, MA: Ratcliff, R. (1979). Group reaction time Svoboda, E., and Levine, B. (2009). The under the terms of the Creative Com-
Addison-Wesley), 289–334. distributions and an analysis of dis- effects of rehearsal on the functional mons Attribution License, which per-
Neisser, U., and Kerr, N. (1973). Spatial tribution statistics. Psychol. Bull. 86, neuroanatomy of episodic autobio- mits use, distribution and reproduction
and mnemonic properties of visual 446–461. graphical and semantic remember- in other forums, provided the original
images. Cogn. Psychol. 5, 138–150. Ratcliff, R. (1993). Methods for dealing ing: a functional magnetic resonance authors and source are credited and sub-
Neter, J., Kutner, M. H., Nachtsheim, with reaction time outliers. Psychol. imaging study. J. Neurosci. 29, 3073– ject to any copyright notices concerning
C. J., and Wasserman, W. (1996). Bull. 114, 510–532. 3082. any third-party graphics etc.

Table A2 | Basic repeated measure logistic regression model

APPENDICES
information.
APPENDIX A
A linear mixed model was fit to the data to assess the contribution Dependent variable Recall
of the variables to linear change in vividness of imagery. The analy- Probability distribution Binomial
sis was carried out using SPSS17. The variables in the model were Link function Logit
evaluated by a Type III test. Since the sample size was not large, Observation used 1490
Restricted Iterative Generalized Least Square (RIGLS) estimation
method was used. Table A1 shows the type III tests of fixed effects.
Table A3 | Evaluation result for logistic regression predictive model for
(Cases with rating of “no image” (vividness rating value 1) were
incidental recall using stimuli and vividness as predictor.
excluded from all analysis).
To test how well the factors in the dataset predict recall, a Predicted
logistic regression analysis was performed. The response for recall
was recorded as 1 for recalled verbal descriptions and 0 for not Cues of imagined objects Percentage
recalled verbal descriptions. Explanatory variables included vivid- correct
ness, stimuli, and reaction time (RT). Table A2 displays model Observed Non-recalled Recalled
specifications, including the specified distribution and link func-
tion. Table A3 summarizes the results. Table A4 shows the full Non-recalled 779 142 83.7
Cues
model results by predictor. Recalled 259 310 54.7
All 1490 valid observations were entered in the logistic regres- Overall 72.2
sion model as the preliminary linear mixed modeling fitting percentage
analysis indicated that residual errors were only modestly cor-
related within each subject and were independent across subjects.
Robustness to the violation of the assumption of independence based on the probability of the largest value of response variable,
was demonstrated by replicating the results with the following which was 1. Two stimuli, which caused singularity of Hessian
confirmatory repeated measure logistic regression model. matrix, were removed from the dataset, resulting in 1441 obser-
The dichotomous outcome for recall was further modeled with vations (and no difference in the results). Models specifications,
a repeated measure logistic regression analysis. The model was including the specified distribution and link function were same
as the initial logistic regression model.
Type III test evaluated the effect of explanatory variables
on recall accuracy in the purposed model. The test result
Table A1 | Type III tests of fixed effects in linear mixed model analysis
showed that Vividness and stimuli were significant predic-
testing the influences of stimuli and image generation time (RTs) on
tors for recall, χ2 = 14.77 and χ2 = 4276, p < 0.05 respectively
vividness ratings.
(see Table A5). No other significance was found. Table A6
Source df numerator df denominator F P shows estimation for parameters in the model. Table A7 shows
validity of predicted probabilities. The prediction for descrip-
Intercept 1 632.643 3002.619 0.000 tions which were not recalled was more accurate than that
Stimuli 59 1000.202 1.588 0.004 for the verbal descriptions which were, 50.5% of non-recalled
RTs 1 1103.724 5.172 0.023 descriptions and 22% of recalled descriptions were correctly
Stimuli × RTs 59 1046.513 0.956 0.572 predicted. This confirmed that the model had overall 72.5%
accuracy.
Dependent variable: vividness.

Table A4 | Logistic regression analysis of vividness and incidental recall data.
Predictor β SE Wald df p Exp (β)
Vividness 17.426 5 0.004

Vividness (1) −0.091 0.374 0.059 1 0.808 0.913
Vividness (2) 0.315 0.362 0.760 1 0.383 1.371
Vividness (3) 0.684 0.317 4.647 1 0.031 1.982
Vividness (4) 0.537 0.305 3.100 1 0.078 1.712
Vividness (5) 0.850 0.307 7.684 1 0.006 2.341
Stimuli 246.836 57 0.000
Stimuli (1) −2.776 0.751 13.658 1 0.000 0.062
Stimuli (2) −2.511 0.690 13.256 1 0.000 0.081
Stimuli (3) −2.485 0.690 12.955 1 0.000 0.083
Stimuli (4) −0.873 0.659 1.755 1 0.185 0.418
Stimuli (5) 1.055 0.894 1.393 1 0.238 2.873
Stimuli (6) −2.783 0.710 15.374 1 0.000 0.062
Stimuli (7) −3.751 0.895 17.561 1 0.000 0.023
Stimuli (8) −1.146 0.658 3.035 1 0.082 0.318
Stimuli (9) −2.282 0.696 10.739 1 0.001 0.102
Stimuli (10) −3.082 0.743 17.185 1 0.000 0.046
Stimuli (11) −0.587 0.662 0.788 1 0.375 0.556
Stimuli (12) −3.852 0.893 18.601 1 0.000 0.021
Stimuli (13) −0.987 0.647 2.330 1 0.127 0.373
Stimuli (14) −0.684 0.663 1.064 1 0.302 0.505
Stimuli (15) −0.008 0.696 0.000 1 0.991 0.992
Stimuli (16) −2.110 0.680 9.636 1 0.002 0.121
Stimuli (17) −2.882 0.747 14.874 1 0.000 0.056
Stimuli (18) −2.528 0.689 13.458 1 0.000 0.080
Stimuli (19) 0.254 0.744 0.117 1 0.733 1.289
Stimuli (20) −2.955 0.745 15.724 1 0.000 0.052
Stimuli (21) −3.342 0.797 17.586 1 0.000 0.035
Stimuli (22) −1.610 0.647 6.186 1 0.013 0.200
Stimuli (23) −2.172 0.679 10.230 1 0.001 0.114
stimuli(24) −0.780 0.661 1.391 1 0.238 0.458
Stimuli (25) −2.358 0.695 11.524 1 0.001 0.095
Stimuli (26) −1.756 0.658 7.112 1 0.008 0.173
Stimuli (27) −2.515 0.689 13.305 1 0.000 0.081
Stimuli (28) −3.760 0.895 17.656 1 0.000 0.023
Stimuli (29) −1.680 0.650 6.682 1 0.010 0.186
Stimuli (30) −2.244 0.676 11.026 1 0.001 0.106
Stimuli (31) −2.098 0.664 10.001 1 0.002 0.123
Stimuli (32) −2.052 0.666 9.500 1 0.002 0.128
Stimuli (33) −0.850 0.655 1.685 1 0.194 0.427
Stimuli (34) −1.538 0.651 5.577 1 0.018 0.215
Stimuli (35) −3.255 0.798 16.655 1 0.000 0.039
Stimuli (36) −0.493 0.664 0.551 1 0.458 0.611
Stimuli (37) −1.570 0.656 5.724 1 0.017 0.208
Stimuli (38) −0.527 0.658 0.641 1 0.423 0.590
Stimuli (39) −1.746 0.649 7.245 1 0.007 0.174
Stimuli (40) −1.934 0.672 8.296 1 0.004 0.145
Stimuli (41) −0.115 0.709 0.026 1 0.871 0.891
Stimuli (42) 0.262 0.723 0.132 1 0.716 1.300
Stimuli (43) −0.490 0.667 0.539 1 0.463 0.613
Stimuli (44) −0.764 0.651 1.376 1 0.241 0.466
Stimuli (45) −2.352 0.674 12.196 1 0.000 0.095
(Continued)
Frontiers in Psychology | Perception Science February 2013 | Volume 4 | Article 1| 62

Table A4 | Continued
Predictor β SE Wald df p Exp (β)
Stimuli (46) −1.479 0.645 5.259 1 0.022 0.228

Stimuli (47) −3.562 0.901 15.633 1 0.000 0.028
Stimuli (48) −2.927 0.747 15.370 1 0.000 0.054
Stimuli (49) −1.717 0.650 6.989 1 0.008 0.180
Stimuli (50) −1.421 0.643 4.883 1 0.027 0.241
Stimuli (51) −2.051 0.665 9.501 1 0.002 0.129
Stimuli (52) −1.712 0.682 6.299 1 0.012 0.181
Stimuli (53) −0.028 0.691 0.002 1 0.967 0.972
Stimuli (54) −2.172 0.679 10.238 1 0.001 0.114
Stimuli (55) −2.407 0.691 12.132 1 0.000 0.090
Stimuli (56) −1.138 0.735 2.395 1 0.122 0.321
Stimuli (57) −2.455 0.691 12.612 1 0.000 0.086
Constant 0.664 0.576 1.329 1 0.249 1.943
Variable(s) entered: vividness, stimuli.
Table A5 | Type III test of model effects for repeated measure logistic regression model.
Source Wald χ2 df p
(Intercept) 22.744 1 0.000

Vividness 14.766 5 0.011
RT 0.050 1 0.824
Vividness × RT 6.972 5 0.223
Stimuli 4276.081 25 0.000
Dependent variable: recall.

Table A6 | Parameter estimate for repeated measure logistic regression.
95% Wald confidence interval Hypothesis test
Parameter β SD Lower Upper Wald df P
(Intercept) 0.905 0.4538 – 1.794 3.976 1 0.046

[vividness = 2.00] 1.136 0.3524 0.445 1.826 10.383 1 0.001
[vividness = 3.00] 0.748 0.3818 0.000 1.496 3.837 1 0.050
[vividness = 4.00] 0.608 0.4688 −0.311 1.526 1.680 1 0.195
[vividness = 5.00] 0.162 0.2989 −0.424 0.748 0.295 1 0.587
[vividness = 6.00] 0.356 0.2540 −0.142 0.854 1.965 1 0.161
[vividness = 7.00] 0a – – – – – –
RT 0.004 0.0054 −0.007 0.015 0.540 1 0.462
[vividness = 2.00] × s −0.025 0.0188 −0.062 0.012 1.723 1 0.189
[vividness = 3.00] × s 0.018 0.0130 −0.008 0.043 1.892 1 0.169
[vividness = 4.00] × s −0.005 0.0132 −0.031 0.021 0.147 1 0.701
[vividness = 5.00] × s 0.000 0.0079 −0.016 0.015 0.005 1 0.942
[vividness = 6.00] × s −0.004 0.0100 −0.023 0.016 0.146 1 0.702
[vividness = 7.00] × s 0a – – – – – –
Stimuli
[stimuli = 1.00] −2.481 0.6790 −3.811 −1.150 13.345 1 0.000
[stimuli = 2.00] 0.293 0.8285 −1.331 1.917 0.125 1 0.723
[stimuli = 3.00] 0.036 0.7209 −1.377 1.449 0.002 1 0.960
[stimuli = 4.00] 0.023 0.5302 −1.016 1.062 0.002 1 0.965
[stimuli = 5.00] −1.597 0.4804 −2.538 −0.655 11.046 1 0.001
[stimuli = 6.00] −3.531 0.7974 −5.094 −1.968 19.609 1 0.000
[stimuli = 7.00] 0.318 0.6115 −0.880 1.517 0.271 1 0.603
[stimuli = 8.00] 1.291 0.9455 −0.563 3.144 1.863 1 0.172
[stimuli = 9.00] −1.313 0.6087 −2.506 −0.120 4.650 1 0.031
[stimuli = 10.00] −0.152 0.6943 −1.513 1.209 0.048 1 0.827
[stimuli = 11.00] 0.624 0.6256 −0.603 1.850 0.993 1 0.319
[stimuli = 12.00] −1.936 0.5493 −3.013 −0.860 12.428 1 0.000
[stimuli = 13.00] 1.380 0.6474 0.111 2.649 4.542 1 0.033
[stimuli = 14.00] −1.511 0.5988 −2.685 −0.337 6.368 1 0.012
[stimuli = 15.00] −1.784 0.5815 −2.924 −0.644 9.410 1 0.002
[stimuli = 16.00] −2.427 0.5536 −3.512 −1.342 19.224 1 0.000
[stimuli = 17.00] −0.305 0.5736 −1.429 0.819 0.282 1 0.595
[stimuli = 18.00] 0.417 0.7336 −1.020 1.855 0.324 1 0.569
[stimuli = 19.00] 0.066 0.7188 −1.343 1.475 0.008 1 0.927
[stimuli = 20.00] −2.730 0.7174 −4.136 −1.324 14.485 1 0.000
[stimuli = 21.00] 0.466 0.5001 −0.514 1.446 0.869 1 0.351
[stimuli = 22.00] 0.873 0.6202 −0.342 2.089 1.983 1 0.159
[stimuli = 23.00] −0.877 0.6083 −2.069 0.315 2.079 1 0.149
[stimuli = 24.00] −0.297 0.7287 −1.725 1.132 0.166 1 0.684
[stimuli = 25.00] −1.656 0.6106 −2.853 −0.460 7.359 1 0.007
[stimuli = 26.00] −0.097 0.7548 −1.576 1.382 0.016 1 0.898
[stimuli = 27.00] −0.711 0.6595 −2.004 0.581 1.164 1 0.281
[stimuli = 28.00] 0.054 0.6819 −1.282 1.391 0.006 1 0.937
[stimuli = 29.00] 1.313 0.9485 −0.546 3.172 1.915 1 0.166
[stimuli = 30.00] −0.786 0.6830 −2.125 0.552 1.326 1 0.249
[stimuli = 31.00] −0.227 0.6441 −1.489 1.036 0.124 1 0.725
[stimuli = 32.00] −0.366 0.6257 −1.592 0.860 0.342 1 0.559
[stimuli = 33.00] −0.410 0.7467 −1.874 1.053 0.302 1 0.583
[stimuli = 34.00] −1.615 0.6717 −2.931 −0.298 5.778 1 0.016
[stimuli = 36.00] −0.931 0.6474 −2.200 0.338 2.069 1 0.150
(Continued)

Table A6 | Continued
95% Wald confidence interval Hypothesis test
Parameter β SD Lower Upper Wald df P
[stimuli = 37.00] 0.881 0.7813 −0.650 2.412 1.272 1 0.259

[stimuli = 38.00] −1.978 0.5974 −3.149 −0.807 10.958 1 0.001
[stimuli = 39.00] −0.934 0.6869 −2.280 0.412 1.849 1 0.174
[stimuli = 40.00] −1.942 0.5491 −3.018 −0.866 12.513 1 0.000
[stimuli = 41.00] −0.726 0.5983 −1.898 0.447 1.471 1 0.225
[stimuli = 42.00] −0.527 0.7072 −1.913 0.860 0.554 1 0.457
[stimuli = 43.00] −2.346 0.6794 −3.678 −1.014 11.922 1 0.001
[stimuli = 44.00] −2.760 0.6862 −4.104 −1.415 16.175 1 0.000
[stimuli = 45.00] −1.974 0.6386 −3.226 −0.723 9.558 1 0.002
[stimuli = 46.00] −1.686 0.6688 −2.997 −0.375 6.355 1 0.012
[stimuli = 48.00] −0.120 0.5964 −1.289 1.049 0.041 1 0.840
[stimuli = 49.00] −0.994 0.5766 −2.124 0.136 2.971 1 0.085
[stimuli = 50.00] 1.164 0.7937 −0.392 2.719 2.150 1 0.143
[stimuli = 51.00] 0.488 0.7849 −1.050 2.027 0.387 1 0.534
[stimuli = 52.00] −0.759 0.6465 −2.026 0.508 1.379 1 0.240
[stimuli = 53.00] −1.054 0.6764 −2.380 0.271 2.430 1 0.119
[stimuli = 54.00] −0.415 0.5582 −1.509 0.679 0.553 1 0.457
[stimuli = 55.00] −0.753 0.6128 −1.954 0.448 1.509 1 0.219
[stimuli = 56.00] −2.425 0.7889 −3.971 −0.879 9.448 1 0.002
[stimuli = 57.00] −0.292 0.6341 −1.535 0.950 0.213 1 0.645
[stimuli = 58.00] −0.102 0.6570 −1.390 1.185 0.024 1 0.876
[stimuli = 59.00] −1.387 0.7481 −2.853 0.079 3.439 1 0.064
[stimuli = 60.00] 0a – – – – – 0.015
Dependent variable: recall.

a
Set to zero because this parameter is redundant.
Table A7 | Predicted recall value from repeated measure logistic regression model.
Predicted category value
Recalled 0.00 1.00 Total
Count 728 144 872

0.0 (no)
% of Total 50.5 10.0 60.5
Count 252 317 569
1.00 (yes)
% of Total 17.5 22.0 39.5
Count 980 461 1441
Total
% of Total 68.0 32.0 100.0

APPENDIX B and variance. This rule was strictly adhered to unless otherwise
A corpus of 66 peer-reviewed experimental journal articles repre- unavoidable, in which case probability signifiers were rounded
senting 4.32% of the literature available through PsycINFO, and to the reported cut-off (i.e., p < 0.05 was approximated as 0.05);
containing the keyword “vividness” was randomly compiled by a however, it should be noted that rounding was required six times
research assistant naive to the purposes of the study (see Appendix over the course of 863 entries. The resultant entries were then
references). Random selection of a relevant representative sam- categorized as either VVIQ or trial-by-trial vividness subjective
ple can be defended as a sound, reasonable meta-analytic tactic, report, and as either a neural or behavioral/cognitive objective
provided the selected sources are analyzed according to a set of pre- measure. All values within each category were summated, and
defined, a priori criteria (Rosenthal, 1991). As a prerequisite for divided by the square root of the number of entries within each
inclusivity, any statistical outcome directly pertaining to the mea- category.
sures VVIQ and trial-by-trial vividness ratings were to be utilized A non-parametric analysis examining experimental outcome
in the analysis, except those pertaining to post hoc comparisons. between VVIQ and trial-by-trial vividness ratings is presented in
The analysis consisted of two phases, a preliminary non- Figures A1A,B. The data in Figure A1A represent the number
parametric analysis, and a secondary parametric analysis. Data for of significant versus non-significant experimental outcomes for
the preliminary analysis was obtained by partitioning individual VVIQ and trial-by-trial vividness ratings for behavioral/cognitive
experimental outcomes into two 2 × 2 contingency tables. Upon objective measures. The data in Figure A1B represent the num-
partitioning each experimental outcome as either a significant or ber of significant versus non-significant experimental outcomes
non-significant experimental outcome, and as either a VVIQ or for VVIQ and trial-by-trial vividness ratings for neural objective
trial-by-trial vividness subjective report, each datum was further measures. A higher proportion of successes accompany trial-by-
categorized as either a neural or behavioral/cognitive objective trial vividness ratings for both behavioral/cognitive and neural
measure. objective measures. This relationship is especially true for studies
The same dataset from the preliminary non-parametric analy- underlying the neural origin of vividness.
sis was utilized in the secondary parametric analysis. However, The trends observed in the preliminary analysis prompted the
each binomial outcome was transformed into an exact probability use of a more sensitive statistical procedure. Because the direction-
value. Analytic accuracy was maintained by calculating proba- ality of each statistical outcome was not immediately apparent,
bilities from reported test statistics and degrees of freedom. If and degrees of freedom often exceed one for F -tests and Chi-
required, raw data was statistically analyzed anew from means square tests of significance, standard meta-analytic methodology
was decidedly insufficient for such purposes (Rosenthal, 1991).
Under these circumstances, Stouffer’s method of adding Z ’s pro-
vides a straightforward and reasonable estimate (Mosteller and
Bush, 1954; Rosenthal, 1991). Upon determining exact probabil-
ity values for each entry introduced, the values were transformed
into their standard normal deviates. These values were summated,
and divided by the square root of the number of entries within each
category. Data for the parametric analysis is shown in Figure A2.
These data show the summated Z -scores for VVIQ and trial-by-
trial vividness ratings for behavioral/cognitive and neural objective
measures.
FIGURE A1 | (A) Experimental outcomes for trial-by-trial vividness ratings

and VVIQ with respect to behavioral and/or cognitive measures. (B)
Experimental outcomes for trial-by-trial vividness ratings and VVIQ with
respect to neural measures. The dark line refers to results which reject the
null hypothesis, and the light line refers to results which fail to reject the FIGURE A2 | Summed Z -scores for vividness and VVIQ for neural and
null hypothesis. behavioral and/or cognitive measures.

As evidenced by Figure A2, two trends remain especially salient. the correlation between VVIQ scores and trial-by-trial vividness
Firstly, trial-by-trial vividness ratings are consistently greater ratings for 21 entries retrieved from six of the peer-reviewed jour-
for behavioral/cognitive and neural measures. Secondly, behav- nal articles showed an average Zr of 0.154, and variability in these
ioral/cognitive measures yield significantly greater values than values ranged from r = −0.27, to r = 0.64. Consistent with the
those which are neural. These results suggest that trial-by-trial results of experiment 2, these results support the contention that
vividness ratings are a more effective means by which to mea- trial-by-trial vividness self-reports and VVIQ scores share some
sure the subjective experience of mental imagery. Furthermore, descriptive properties of visual imagery; however, trial-by-trial
Fisher’s Z -transformation for experimental outcomes concerning vividness ratings seem to be much more resolved.
REFERENCES from spatial processing. Nat. Rev. D’Angiulli, A. (2002). Mental image of discovery and the vividness of
Adeyemo, S. A. (1998). Imagery Neurosci. 9, 182–194. generation and the contrast sensitiv- visual images. Percept. Psychophys. 9,
unvividness and social- Borkovec, T. D., and Sides, J. K. ity function. Cognition 85, B11–B19. 242–264.
psychological implications. J. (1979). The contribution of relax- D’Angiulli, A., and Reeves, A. Holmes, E. A., Lang, T. J., Moulds, M. L.,
Ment. Imagery 22, 79–98. ation and expectancy to fear reduc- (2002). Generating visual men- and Steele, A. M. (2008). Prospective
Aleman, A., Böcker, K. B. E., and de tion via graded, imaginal exposure to tal images: latency and vividness are and positive mental imagery deficits
Haan, E. H. F. (1999). Disposition feared stimuli. Behav. Res. Ther. 17, inversely related. Mem. Cognit. 30, in dysphoria. Behav. Res. Ther. 46,
towards hallucination and subjective 529–540. 1179–1188. 976–981.
versus objective vividness of imagery Burton, L. J. (2003). Examining the D’Angiulli, A., and Reeves, A. (2005). Hunt, M., Bylsma, L., Brock, J., Fenton,
in normal subjects. Pers. Individ. relation between visual imagery and “Picture theory, tacit knowledge or M., Goldberg, A., Miller, R., et al.
Differ. 27, 707–714. spatial ability tests. Int. J. Test. 3, vividness-core? Three hypotheses on (2006). The role of imagery in the
Aleman, A., and de Haan, E. H. F. 277–291. the mind’s eye and its elusive size,” maintenance and treatment of snake
(2001). Do people with better men- Campos, A., Amor, Á., and González, M. in Proceedings of the 27th Cognitive fear. J. Behav. Ther. Exp. Psychiatry
tal imagery ability make more real- Á. (2002). Presentation of keywords Science Society Annual Meeting, eds 37, 283–298.
ity monitoring errors? Curr. Psychol. by means of interactive drawings. B. Bara, L. Barsalou, and M. Buccia- Iaccino, J., and Byrne, J. (1989). Men-
Lett. 6, 7–15. Span. J. Psychol. 5, 102–109. relli (Stresa: The Cognitive Science tal layouts of concealed objects as a
Aleman,A., and de Haan, E. H. F. (2004). Campos, A., Gómez-Juncal, R., and Society), 536–541. function of imagery type and exper-
Fantasy proneness, mental imagery Pérez-Fabello, M. J. (2007). Expe- D’Angiulli, A., and Reeves, A. (2007). imental conditions. Bull. Psychon.
and reality monitoring. Pers. Individ. rience in imagery and imagery The relationship between self- Soc. 27, 402–404.
Differ. 36, 1747–1754. vividness. Imagin. Cogn. Pers. 27, reported vividness and latency Jenness, A., and Jorgensen, A. P. (1941).
Aleman, A., Nieuwenstein, M. R., 337–348. during mental size scaling of Ratings of vividness of imagery in
Böcker, K. B. E., and de Haan, E. Campos, A., López, A., and González, everyday items: phenomenolog- the waking state compared with
H. F. (2000). Mental imagery and M. A. (1999a). Effects of eyes ical evidence of different types reports of somnambulism. Am. J.
perception in hallucination-prone open/closed and order of rating on of imagery. Am. J. Psychol. 120, Psychol. 54, 253–259.
individuals. J. Nerv. Ment. Dis. 188, VVIQ scores. J. Ment. Imagery. 23, 521–551. Keenan, J. M., and Moore, R. E. (1979).
830–836. 35–43. De Pascalis, V., Marucci, F. S., Penna, Memory for images of concealed
Allbutt, J., Ling, J., Heffernan, T. Campos, A., Marcos, J. L., and Gonzáles, P. M., and Pessa, E. (1987). Hemi- objects: a reexamination of Neisser
M., and Shafiullah, M. (2008). M. Á. (1999b). Emotionality of spheric activity of 40 Hz EEG dur- and Kerr. J. Exp. Psychol. Learn.
Self-report imagery questionnaire words as related to vividness of ing recall of emotional events: dif- Mem. Cogn. 5, 374–385.
scores and subtypes of social- imagery and concreteness. Percept. ferences between low and high hyp- Keogh, L., and Markham, R. (1998).
desirable responding: auditory Mot. Skills 88, 1135–1140. notizables. Int. J. Psychophysiol. 5, Judgements of other people’s mem-
imagery, visual imagery, and think- Cartwright, D., Jenkins, J. L., Chavez, 167–180. ory reports: differences in reports
ing style. J. Individ. Differ. 29, R., and Peckar, H. (1983). Studies Dean, G. M., and Morris, P. E. as a function of imagery vivid-
181–188. in imagery and identity. J. Pers. Soc. (2003). The relationship between ness. Appl. Cogn. Psychol. 12,
Baddeley, A. D., and Andrade, J. (2000). Psychol. 44, 376–384. self-reports of imagery and spatial 159–171.
Working memory and the vividness Cocude, M., and Denis, M. (1988). Mea- ability. Br. J. Psychol. 94, 245–273. Kerr, N. H., and Neisser, U. (1983).
of imagery. J. Exp. Psychol. Gen. 129, suring the temporal characteristics Dyckman, J. M., and Cowan, P. A. Mental images of concealed
126–145. of visual images. J. Ment. Imagery 12, (1978). Imaging vividness and the objects-new evidence. J. Exp.
Barlett, C. P., and Brannon, L. A. (2007). 89–101. outcome of in vivo and imag- Psychol. Learn. Mem. Cogn. 9.
“If only...”: the role of visual imagery Conduit, R., Crewther, S. G., and Cole- ined scene desensitization. J. Con- 212–221.
in counterfactual thinking. Imagin. man, G. (2004). Spontaneous eyelid sult. Clin. Psychol. 46, 1155–1156. Kidd, L. K., and Workman, J. E.
Cogn. Pers. 26, 87–100. movements (ELMS) during sleep are Eysenck, M. (1974). Age differences in (1999). Assessment of creativity in
Belardinelli, M. O., Palmiero, M., Ses- related to dream recall on awaken- incidental learning. Dev. Psychol. 10, apparel design. Cloth. Text. Res. J. 17,
tieri, C., Nardo, D., Di Matteo, R., ing. J. Sleep Res. 13, 137–144. 936–941. 58–64.
Londei, A., et al. (2009). An fMRI Cornoldi, C., de Beni, R., Cavedon, A., Gale, A., Morris, P. E., Lucas, B., Kosslyn, S., and Alper, S. (1977). On the
investigation on image generation and Mazzoni, G. (1992). How can a and Richardson, A. (1972). Types pictorial properties of visual images:
in different sensory modalities: the vivid image be described? Charac- of imagery and imagery types: effects of image size on mem-
influence of vividness. Acta Psychol. teristics influencing vividness judg- an EEG study. Br. J. Psychol. 63, ory for words. Can. J. Psychol. 31,
(Amst.) 132, 190–200. ments and the relationship between 523–531. 32–40.
Bent, N. A., and Wick, E. (2006). Beyond vividness and memory. J. Ment. Giusberti, F., Cornoldi, C., De Beni, R., Lorey, B., Pilgramm, S., Bischoff, M.,
vividness: parental filters as moder- Imagery 16, 89–107. and Massaroni, M. (1992). Differ- Stark, R., Vaitl, D., Kindermann S., et
ators in mental imagery and mea- Cui, X., Jeter, C. B., Yang, D., Montague, ences in vividness ratings of per- al. (2011). Activation of the parieto-
sured anxiety level. J. Ment. Imagery P. R., and Eagleman, D. M. (2007). ceived and imagined patterns. Br. J. premotor network is associated with
30, 21–38. Vividness of mental imagery: indi- Psychol. 83, 533–547. vivid motor imagery–a parametric
Bird, C. M., and Burgess, N. (2008). The vidual variability can be measured Hale, S. M., and Simpson, H. M. (1971). fMRI Study. PLoS ONE 6:e20368.
hippocampus and memory: insights objectively. Vision Res. 47, 474–478. Effects of eye movements on the rate doi:10.1371/journal.pone.0020368

Mantani, T., Okamoto, Y., Shirao, N., Morrison, R. G., and Wallace, B. (2001). Riske, M. L., Wallace, B., and Allen, P. A. individual differences in imagery
Okada, G., and Yamawaki, S. (2005). Imagery vividness, creativity and the (2000). Imaging ability and eye wit- ability. J. Ment. Imagery 11,
Reduced activation of posterior visual arts. J. Ment. Imagery 25, ness accuracy. J. Ment. Imagery 24, 97–104.
cingulate cortex during imagery in 135–152. 137–148. Tracy, R. J., Roesner, L. S., and Kovac, R.
subjects with high degrees of alex- Neisser, U., and Kerr, N. (1973). Spa- Ritchey, H., and Beal, C. (1980). N. (1988). The effect of visual ver-
ithymia: a functional magnetic reso- tial and mnemonic properties of Image detail and recall: evidence sus auditory imagery on vividness
nance imaging study. Biol. Psychiatry visual images. Cogn. Psychol. 5, for within-item elaboration. J. Exp. and memory. J. Ment. Imagery 12,
57, 982–990. 138–150. Psychol. 6, 66–76. 145–161.
Markham, R., and Hynes, L. (1993). Overton, J. A. G. (2004). Correlation of Rodway, P., Gillies, K., and Schepman, Van Diest, I., De Peuter, S., Devriese,
The effect of vividness of imagery on EEG activity with subjective perfor- A. (2006). Vivid imagers are better at S., Wellens, E., Van, d. W., and
reality monitoring. J. Ment. Imagery mance on a guided imagery test: an detecting salient changes. J. Individ. Van, d. B. (2005). Imagined risk of
17, 159–170. exploratory study. J. Ment. Imagery Differ. 27, 218–228. suffocation as a trigger for hyper-
Marks, D. F., and Isaac, A. R. (1995). 28, 17–60. Sadler, P., and Woody, E. Z. (2006). Does ventilation. Psychosom. Med. 67,
Topographical distribution of EEG Pearson, J., Rademaker, R. L., and the more vivid imagery of high hyp- 813–819.
activity accompanying visual and Tong, F. (2011). Evaluating the notizables depend on greater cog- Vergeer, I., and Roberts, J. (2006). Move-
motor imagery in vivid and non- mind’s eye: the metacognition of nitive effort? A test of dissociation ment and stretching imagery during
vivid imagers. Br. J. Psychol. 86, visual imagery. Psychol. Sci. 22, and social-cognitive theories of hyp- flexibility training. J. Sports Sci. 24,
271–282. 1535–1542. nosis. Int. J. Clin. Exp. Hypn. 54, 197–208.
McClelland, A., Kemps, E., and Rauch, S. A. M., Foa, E. B., Furr, J. M., 372–391. Walczyk, J. J., and Hall, V. C. (1988). The
Tiggemann, M. (2006). Reduc- and Filip, J. C. (2004). Imagery vivid- Sheehan, P. W. (1966). Accuracy and relationship between imagery vivid-
tion of vividness and associated ness and perceived anxious arousal vividness of visual images. Percept. ness ratings and imagery accuracy. J.
craving in personalized food in prolonged exposure treatment Mot. Skills 23, 391–398. Ment. Imagery 12, 163–171.
imagery. J. Clin. Psychol. 62, for PTSD. J. Trauma. Stress 17, Slee, J. A. (1980). Individual differences Zhang, M., Weisser, V. D., Stilla, R.,
355–365. 461–465. in visual imagery ability and the Prather, S. C., and Sathian, K. (2004).
McKelvie, S. J. (1998). Effects of gen- Rehm, L. P. (1973). Relationships retrieval of visual appearances. J. Multisensory cortical processing of
der on reported vividness of visual among measures of visual Ment. Imagery 4, 93–113. object shape and its relation to men-
imagery for parents. J. Ment. Imagery imagery. Behav. Res. Ther. 11, Sutherland, M. E., Harrell, J. P., and tal imagery. Cogn. Affect. Behav. Neu-
22, 99–112. 265–270. Isaacs, C. (1987). The stability of rosci. 4, 251–259.

published: 08 November 2012
The effects of visual imagery on face identification:

an ERP study
Jianhui Wu 1*, Hongxia Duan 1,2 , Xing Tian 3 , Peipei Wang 4 and Kan Zhang 1
1
Key Laboratory of Behavioral Science, Institute of Psychology, Chinese Academy of Sciences, Beijing, China
2
University of the Chinese Academy of Sciences, Beijing, China
3
Department of Psychology, New York University, New York, NY, USA
4
Center for Higher Brain Functions, Capital Medical University, Beijing, China
Edited by: The present study tested the hypothesis that the effects of mental imagery on subsequent
Joel Pearson, The University of perception occur at a later matching stage in perceptual identification, but not in the
New South Wales, Australia
early perceptual stage as in perceptual detection. The behavioral results suggested that
Reviewed by:
the effect of visual imagery on visual identification is content-specific, i.e., imagining a
Ruthger Righart, Institute for Stroke
and Dementia Research, Germany congruent face facilitates face identification, whereas a mismatch between imagery and
Michael Wibral, Goethe University, perception leads to an interference effect. More importantly, the ERP results revealed
Germany that a more negative N2 response to the subsequent visual face stimuli was elicited over
*Correspondence: fronto-central sites in the mismatch and no-imagery conditions as compared to that in the
Jianhui Wu, Key Laboratory of
match condition, with the early P1 and N170 components independent of manipulations.
Behavioral Science, Institute of
Psychology, Chinese Academy of The latency and distribution of the neural effects demonstrate that the matching step,
Sciences, 4A Datun Road, Chaoyang but not the earlier perceptual process, is affected by the preceding visual imagery in
District, Beijing 100101, China. the context of face identification. We discuss these results in a broader context that the
e-mail: wujh@psych.ac.cn
imagery-perception interaction may depend on task demand.
Keywords: visual imagery, face identification, event-related potentials, N170, N2, matching
INTRODUCTION facilitation effect was observed during an identification task, i.e.,

Our visual perception is shaped by our previous visual expe- to give the appropriate response to different stimuli (Finke, 1986).
rience in the real world (Gilbert, 1996; Webster and MacLeod, Detection is modeled as a task that only needs to register the pres-
2011). The visual imagery process simulates perceptual represen- ence of a stimulus in an all-or-none fashion; further processing of
tations on the basis of past experience and provides a mental any specific features of stimuli are not required, and may even
template that can influence the subsequent perception (Moulton slow down the detection performance. In contrast, identification
and Kosslyn, 2009). Evidence supporting the modulation effects is modeled as a task that necessitates the processing of detailed
of mental imagery, however, demonstrates distinct directional featural information for the following matching processing, in
influence on perception. Some studies show that imagery inter- which the comparison between the bottom-up sensory informa-
feres with perception (Perky, 1910; Segal and Fusella, 1970; tion and the top-down formed template is carried out to classify
Reeves, 1981; Craver-Lemley and Reeves, 1987; Craver-Lemley the sensory input. Most of the previous studies focused on the
et al., 1997; Ishai and Sagi, 1997b; Craver-Lemley and Arterberry, effects of imagery on subsequent perceptual detection, in which
2001); whereas others show facilitation effects (Freyd and Finke, both directional modulations were found (e.g., Segal and Fusella,
1984; Farah, 1985; Ishai and Sagi, 1995, 1997a; Pearson et al., 1970; Farah, 1985). With only a few studies investigating the after-
2008). effect on identification (Finke, 1986; McDermott and Roediger,
Several factors have been proposed to reconcile the conflicting 1994; Cabeza et al., 1997; Michelon and Koenig, 2002), consis-
results about the different directional effects of mental imagery. tent facilitation effects were reported, i.e., mental imagery of a
The direction of imagery modulation has been hypothesized to be visual pattern in advance will facilitate the identification of the
content-specific. That is, whether preceding imagery facilitates or same pattern.
interferes with perception depends on how similar the imagined The neural mechanisms mediating the interaction between
and presented patterns are. Subjects were more likely to perceive a imagery and perception are also unclear. One of the questions is
stimulus when the imagined content matched the presented stim- that at which level the top-down and bottom-up processes inter-
ulus than when the two were mismatched (Peterson and Graham, act. The dominant interpretation is the perceptual level hypothesis
1974; Farah, 1985, 1989; Finke, 1986; Farah et al., 1988; Djordjevic in which this facilitation/interference effect occurs at the early
et al., 2004a,b). perceptual level, where the visual processing of visual features
Task demands can be another factor that determines the direc- in external stimuli is directly manipulated by preceding imagery
tion of interaction between imagery and perception. The interfer- (Peterson and Graham, 1974; Neisser, 1976; Freyd and Finke,
ence effect was found when the subsequent perceptual task was a 1984; Farah, 1985; Craver-Lemley and Reeves, 1987; Ishai and
simple detection task, i.e., to give a single response to any stim- Sagi, 1995; Pearson et al., 2008). However, the differential obser-
uli presented while not necessary to identify them; whereas the vations of interference and facilitation effects during detection
Frontiers in Human Neuroscience www.frontiersin.org November 2012 | Volume 6 | Article 305 | 69

Wu et al. Visual imagery and visual identification
and identification tasks (Finke, 1986), lead us to propose, along presented faces will lead to a facilitation effect as compared with
the line of a similar theory (Finke, 1986), that perceptual task the no imagery condition, and mismatch will lead to an inter-
demand may influence the occurrence of imagery-perception ference effect. For the electrophysiological recording, at least two
interaction at distinct stages along the visual information pro- processing components were proposed in the context of face
cessing stream. Specifically, the effects of imagery on perceptual recognition (Bruce and Young, 1986): the early pre-categorical
identification occur at the later stage where integrated features of structural encoding of faces reflected in the early N170 compo-
an object are matched with those stored in memory to achieve nent (e.g., Sagiv and Bentin, 2001) and a later matching process
recognition (referred to as the matching level hypothesis hence- between encoded facial representation and stored structural codes
forth in this paper); whereas the effects of imagery on perceptual (templates) presumably mediated by a fronto-central distributed
detection are presented in early perceptual processes where the N2 component that is usually elicited by a perceptual mismatch
representation of object features in establishing and spotting the from a template and detection of novelty (for a review, see Folstein
existence of features in an all-or-none fashion would be sufficient and Van Petten, 2008). Some studies reported that the earlier
in the task of detection (perceptual level hypothesis). P1 component is also related to the earlier stage of face percep-
To distinguish the perceptual level hypothesis and the matching tion (Itier and Taylor, 2004; Thierry et al., 2007). According to
level hypothesis, we need to investigate the dynamics of cogni- the matching level hypothesis, this N2 component is predicted
tive functions. It is hard, if not impossible, to separate different to be modulated by preceding imagery; whereas the modula-
cognitive stages using behavioral experiments as in most previ- tion of the P1 and N170 responses is predicted by the perceptual
ous studies, because the behavioral performance is the cumulative level hypothesis and any changes observed in these early com-
result of processing at multiple levels. Moreover, the behavioral ponents will suggest the formation of perceptual features in
measures suffer from methodological limitations, such as con- the identification task is equally affected by preceding mental
founds from experimenter expectancy effects and subjects’ tacit imagery.
knowledge (Farah et al., 1988; Pylyshyn, 2002). The event-related
potentials (ERP) technique, on the other hand, is an objective MATERIALS AND METHODS
measure that may be relatively less confounded by these strategic PARTICIPANTS
effects. More importantly, with its high temporal resolution, ERP Data from 24 participants (mean age 22 ± 1.7 years, 12 men, all
can be used to determine the time course of neural activity, mak- right-handed) were collected and analyzed. All were undergrad-
ing it possible to determine the cognitive stage at which mental uates from China Agricultural University and Beijing Forestry
imagery has effects on perception. University who gave informed consent and were paid for their
To our knowledge, only two ERP studies have been carried participation. None of them had a history of neurological or psy-
out to investigate the effects of visual imagery on perception, chiatric disorders. All reported normal hearing and normal or
using either a detection task paradigm (Farah et al., 1988) or corrected-to-normal vision. The experiment had been approved
without an active task (Ganis and Schendan, 2008). Farah et al. by the Ethics Committee of Human Experimentation in the
(1988) found larger early negativity to the subsequent visual stim- Institute of Psychology, Chinese Academy of Sciences.
uli which peaked at 160 ms over temporo-occipital sites when
the imagery matched perception compared with the mismatch STIMULI
condition. Consistently, stimuli were detected more often for the Two female faces with neutral expression from the Revision of
match than for the mismatch condition. Ganis and Schendan the Chinese Facial Affective Picture System (Gong et al., 2011)
(2008) observed that both perception and imagery affect the N170 were chosen as visual stimuli for both imagery and real presen-
response to the subsequently presented test faces. Specifically, the tation. These two pictures and one visual noise mask (Figure 1)
amplitude of N170 was enhanced when they were preceded by were presented on a computer screen placed 75 cm away from
face imagery rather than object imagery, and similar effects were the participants’ eyes and subtended at an angle of approximately
found for the non-face objects. Interestingly, Sreenivasan et al.’s 7◦ horizontally and 7.7◦ vertically. Three auditory vowels (a, o,
study (2007) also found that noise probes presented during the and u) were used as acoustic cues to indicate different tasks
delay interval of a delayed-recognition task elicited a larger N170 (imagery vs. no-imagery) as well as content in the imagery tasks.
during face relative to house working memory. These ERP experi- Specifically, two of them indicated participants to visualize differ-
ments provided neural evidence for the perceptual level hypothesis ent faces and the third one reminded participants not to imagine
for visual detection. In these two ERP experiments, however, no anything (no-imagery condition). The associations between audi-
identification task was implemented; it is still unclear at which tory cues and imagined faces/no imagery were counterbalanced
stage mental imagery affects the identification task. across participants. The duration of each sound was 600 ms and
The goal of the present ERP study was to test the proposed the intensity was adjusted to a comfortable listening level of about
matching level hypothesis in visual identification by examining the 70 dB SPL using Adobe Audition (version 1.0). Sounds were deliv-
cognitive stage(s) at which the interaction occurs between face ered binaurally through headphones by the Stim interface system
imagery and face identification. We designed a face imagery-face (Neuroscan Labs, Sterling, VA).
identification paradigm, during which participants were required
to imagine one of two faces or without any imagery followed PROCEDURE
by an identification task to visually presented face pictures. We Participants were seated in a relaxed position on a comfortable
predicted that, behaviorally, matching between imagined and chair in a dimly lit, sound-attenuated, and electrically isolated

FIGURE 1 | Schematic description of the experimental paradigm. After hearing the letter. After a random delay between 800 and 1200 ms, one of
the presentation of fixation, one of these three letter sounds was presented these two real faces was presented for 200 ms and participants were asked
and participants were asked to imagine the corresponding face vividly and to press one of the two buttons according to which of the two faces
then press the “Space” bar in the self-paced interval. For the no imagery presented on the screen. After the response a visual noise mask with
associated sound, participants were asked to simply press “Space” once duration of 200 ms was presented before the next trial began.
room. Participants completed six testing blocks while EEG was learned these associations. The second session was an imagery
recorded. Each block started with a short familiarization ses- training session in which only the auditory cues were presented
sion to remind participants of the pairing of auditory cues and and the participants were encouraged to vividly imagine the
tasks. After a fixation presented with duration between 800 and corresponding faces; the correct pairing of visual stimulus and
1200 ms, one of the auditory cues was presented and the par- auditory cue was then presented, and participants were required
ticipants were asked to press the “Space” bar after they vividly to adjust their imagination. For the no imagery condition, only
formed the image of the corresponding face. In the no-imagery a vowel sound was presented and no imagination or subsequent
condition, participants were asked to press “Space” once hearing adjustment was required. Each pair was repeated 20 times to
the letter. A random delay between 800 and 1200 ms was inserted ensure that participants were able to imagine the corresponding
before a visual face stimulus was presented for 200 ms followed by faces vividly. The third session was a face identification training
a maximal 2000 ms response window, in which participants were session in which one of these two faces was presented and the
asked to press the left button for one face and right button for participants were asked to identify the presented face by press-
another face as quickly and accurately as possible. A visual noise ing the “left” or “right” key as quickly and accurately as possible.
mask with duration of 200 ms was presented before the next trial The association of visual stimuli and response keys was coun-
began (Figure 1). terbalanced across participants. Feedback was provided following
This experiment included three conditions that differed in responses. The fourth session was an imagery-perception training
the tasks associated with the different auditory cues. Participants session with identical procedure to the testing blocks, except feed-
were asked either to skip an auditory cue (no-imagery condi- back was provided after their responses instead of a white noise
tion), or to vividly visualize a corresponding face that could be mask. For all the four training sessions, each participant received
congruent (match condition) or incongruent (mismatch con- equal exposure to both faces.
dition) with the subsequent visual stimuli. Sixty testing trials Participants completed a brief questionnaire by rating the
were included in each block, yielding 120 trials for each condi- vividness of their visual imagery on a 7-point scale (1 = very vivid
tion. The two visual face stimuli were presented equally often in imagery, 7 = no imagery at all) at the end of this experiment.
each condition. A pseudo-random presentation order was used,
so that participants could not predict the upcoming visual face EEG RECORDING AND ANALYSIS
stimuli. Sequential effects of trial-to-trial transitions were also EEG data were continuously recorded from 64 cap-mounted
counterbalanced within each block. Ag/AgCl electrodes arranged according to the 10–20 international
Four training sessions were run before the EEG recordings to placement system (Neuroscan Inc.) with an on-line reference to
ensure the correct pairing of auditory cue and imagery, as well as the left mastoid and off-line algebraic re-reference to the average
the vividness of visual imagery. The first session was a familiar- of the left and right mastoids. The EEG data were amplified with
ization session in which the auditory cues and corresponding face a bandpass filter of 0.05–100 Hz and digitized at 500 Hz. The ver-
pictures (or no picture) were presented simultaneously at least tical and horizontal electrooculogram (VEOG and HEOG) were
35 times for each pair, until participants reported that they had recorded from two pairs of electrodes: one pair placed 1 cm above

and below the left eye, and another pair placed 1 cm lateral from
the outer canthi of both eyes. Interelectrode impedances were
maintained below 5 k.
The EEG data were processed offline using the Neuroscan
4.3 software. Ocular artifacts were removed using a regression
procedure implemented in the Neuroscan software (Semlitsch
et al., 1986). Data were lowpass filtered with cutoff frequency
at 30 Hz and epochs of 400 ms in duration (including 100 ms
of pre-stimulus time as a baseline) were extracted, time-locked
to the onset of visual stimuli. Epochs exceeding ±70 μV were
considered artifacts and rejected from further analysis. Average
responses were obtained for each condition.
The peak amplitude and latency of P1 were measured at elec-
trodes PO7, CB1, O1 PO8, CB2, and O2 and were subjected to a
repeated measures Three-Way ANOVA with factors of matching
(mismatch, match and no-imagery) × laterality (left and right) ×
sites. The mean amplitude of N170 was measured in the time
window of 140–180 ms over 12 parieto-occipital sites (P7, PO3, FIGURE 2 | The mean reaction times in the face identification task.
PO5, PO7, O1, CB1, P8, PO4, PO6, PO8, O2, and CB2) and Error bars represent two standard errors of the mean (SEM). ∗∗ p < 0.01;
∗∗∗ p < 0.001.
was subjected to a repeated measures Three-Way ANOVA with
factors of matching (mismatch, match and no-imagery) × later-
ality (left and right) × sites. Given that the N170 originates in
temporal regions (e.g., Henson et al., 2003), analysis in poste-
rior and inferior temporal channels seems necessary. We thus ran The mean RT of “Space” bar presses after the onset of the
separate ANOVAs for each of the four channels (TP7, TP8, T7, auditory cue was significantly different among the three cues
and T8), though the N170 amplitude was small or the polarity [F(2, 46) = 8.50, p = 0.001]. Pairwise comparisons indicated that
was reversed in these channels. The mean amplitude of the N2 participants spent less time after the no-imagery cue than after
component was measured in the time window of 250–350 ms at the two face-imagery cues (976 ± 265 ms vs. 1358 ± 698 ms, p <
the following 21 sites: Fz, FCz, Cz, CPz, Pz, POz, Oz, F3, FC3, 0.01 and 976 ± 265 ms vs. 1359 ± 806 ms, p < 0.05), but there
C3, CP3, P3, PO3, O1, F4, FC4, C4, CP4, P4, PO4, and O2. was no significant difference between the two face-imagery cues
The N2 amplitudes were subjected to a repeated measures Three- (p = 0.98). The post-experimental questionnaire revealed that
Way ANOVA with factors of matching (mismatch, match and participants had experienced vivid face imagery (2.54 ± 0.72)
no imagery) × anterior-posterior scalp location (F, FC, C, CP, when the imagery cues were presented.
P, PO, and O) × laterality (left, midline and right). Three addi-
tional ANOVAs were carried out to directly test the distinct neural ERP RESULTS
correlates of facilitation/interference with paired matching condi- An occipital P1 was observed in all three conditions (Figure 3).
tions (match vs. mismatch, match vs. no-imagery, and mismatch ANOVA did not reveal any significant difference among condi-
vs. no-imagery). Behaviorally incorrect trials were excluded from tions for either the peak amplitude or latency of the P1 com-
analysis. The Greenhouse–Geisser correction was used to adjust ponent [F(2, 46) = 0.42, p = 0.66 and F(2, 46) = 1.62, p = 0.21,
for sphericity violations. The Bonferroni correction was applied respectively], and also no interaction effect between matching
for multiple comparisons. We also performed post-hoc Pearson conditions and laterality [F(2, 46) = 0.57, p = 0.57 and F(2, 46) =
correlation analyses in order to assess the relationship between 1.18, p = 0.31, ε = 0.72, respectively]. Additional correlation
ERP components and actual behavior (two-tailed). analysis revealed no significant correlation between RT change
and the P1 amplitude/latency differences between the paired
RESULTS comparisons of the three conditions (Ps > 0.10).
BEHAVIOR AND POST-EXPERIMENTAL QUESTIONNAIRE Typical N170 potentials were observed in responses to visual
The mean reaction time (RT) in the face identification task face stimuli over parieto-occipital sites (Figure 3). ANOVA did
was significantly different among the three matching conditions not reveal any significant difference among conditions for the
[F(2, 46) = 26.07, p < 0.001] (Figure 2). Pairwise comparisons mean amplitude of the N170 component [F(2, 46) = 0.091, p =
indicated that participants reliably responded faster on match 0.91], and also no interaction effect between matching conditions
trials than both no imagery trials (456 vs. 482 ms, p < 0.001) and laterality or sites [F(2, 46) = 0.70, p = 0.50 and F(10, 230) =
and mismatch trials (456 vs. 497 ms, p < 0.001), but slower on 1.01, p = 0.42, ε = 0.60, respectively]. Additional correlation
mismatch trials than on no imagery trials (497 vs. 482 ms, p = analysis revealed no significant correlation between RT change
0.01). The response accuracy in the face identification task was and the N170 amplitude differences across all the 12 channels
not significantly different among the three matching conditions between the paired comparisons of the three conditions (Ps >
[F(2, 46) = 1.88, p = 0.17; mismatch: 95.58%, match: 97.02%, 0.10). ANOVAs also did not reveal any significant differences
no-imagery: 96.44%]. among conditions in TP7, TP8, T7, and T8 (Ps > 0.10).

FIGURE 3 | Grand averaged ERPs illustrating the P1 and the N170 elicited by visual stimuli in three conditions.
For the amplitude of the N2 component elicited by real but not at the parieto-occipital and occipital areas (p = 0.36
faces (Figure 4), ANOVA revealed a significant main effect of and 0.52, respectively). Correlation analysis revealed a signifi-
matching (mismatch, match and no-imagery) [F(2, 46) = 5.56, cant negative correlation between RT change (no imagery vs.
p < 0.01], and a marginally significant interaction effect match) and the N2 amplitude change (no imagery-match) over
between matching factors and anterior-posterior scalp location fronto–parietal sensors (r = −0.422, p < 0.05). There was no
[F(12, 276) = 2.50, p < 0.10, ε = 0.21]. One additional ANOVA significant difference between mismatch and no imagery con-
indicated a significant main effect of matching (mismatch vs. ditions (p = 0.39) and no interaction effect between imagery
match) [F(1, 23) = 13.92, p < 0.01] and an interaction effect (mismatch and no imagery) and anterior-posterior electrodes
between matching (mismatch vs. match) and anterior-posterior (p = 0.55).
electrodes [F(6, 138) = 3.39, p < 0.05, ε = 0.29]. Post-hoc analy-
ses revealed that the mismatch condition was more negative than DISCUSSION
the match condition and this effect was broadly distributed along The present study investigated the effects of mental imagery
the anterior–posterior dimension (Ps < 0.01), but was maxi- on subsequent face identification. Behaviorally, participants
mal at the fronto-central areas. Correlation analysis revealed a responded faster in the match condition than in the no-imagery
marginally significant negative correlation between RT change condition, with the slowest RT in the mismatch condition, which
(mismatch vs. match) and the N2 amplitude change (mismatch- provides further evidence supporting the hypothesis that the
match) over anterior–posterior sensors (r = −0.347, p = 0.096) imagery-perception interaction is content-specific in an identi-
(i.e., the longer the RT change, the larger the N2 amplitude fication task. In our experiment, we demonstrated the content-
change). Another additional ANOVA indicated a significant main specific effects extending to complex visual stimuli, human faces
effect of matching (no-imagery vs. match) [F(1, 23) = 4.60, p < in this case, compared to the simple visual objects and features
0.05] and an interaction effect between matching (no-imagery vs. used previously (Finke, 1986). The results of the imagery vivid-
match) and anterior-posterior electrodes [F(6, 138) = 4.47, p < ness questionnaire suggest that the subjects successfully executed
0.05, ε = 0.27]. Post-hoc analyses revealed that the no-imagery the visual imagery. Participants pressed the “Space” bar more
condition was more negative than the match condition and rapidly to the no-imagery cue than to face-imagery cues, provid-
this effect was broadly distributed along the fronto–parietal ing further evidence for the execution of the visual imagery task
areas (Ps < 0.05), and was maximal at the fronto-central areas, as required.

is usually elicited under the S1–S2 matching task, where the S1

can be presented physically or formed from imagery (Wu et al.,
2010). This component has been generally considered to be sensi-
tive to mismatch from visual templates (for a review, see Folstein
and Van Petten, 2008). Wang et al. (2001) have revealed that the
mismatch-related N2 was elicited independently of task demands.
Results from studies using repetition paradigms also revealed
that, compared with the primed target, the unprimed target (mis-
match) elicited a more negative anterior-distributed negativity
around 250–350 ms after stimulus onset (Eddy et al., 2006; Eddy
and Holcomb, 2011). According to the model of face recogni-
tion (Bruce and Young, 1986), a sense of familiarity is achieved,
if the bottom-up sensory process matches the stored structure
code retrieved from memory. The mental imagery of a face, in
this study, is a top-down process that forms an internal perceptual
template of a face representation (Kosslyn et al., 2006; Wu et al.,
2011), and this perceptual template provides a recent context for
the subsequent visual target. When the features of visual stim-
uli match the pre-activated face template from preceding mental
imagery, the identification can be achieved faster, resulting in the
behavioral benefits in the matching condition. Whereas when the
internal image is incompatible with the target, as in the mismatch
condition, more time and higher neural cost is required to resolve
the conflict, manifested in a behavioral interference effect and
larger N2 amplitude.
Our ERP results also revealed that the fronto-central dis-
tributed N2 was observed in the no-imagery condition. There
might be two cognitive mechanisms for this result, and both
of them can be considered occurring at the matching stage of
face identification. The first one is the mismatch interpretation
of this N2 (for a review, see Folstein and Van Petten, 2008).
FIGURE 4 | The left panel illustrates the grand averaged ERPs in
mismatch and match conditions and their difference wave. The right Theories have been proposed that our perception of external
panel illustrates the grand averaged ERPs in no imagery and match objects and events is the result of bottom-up process meeting the
conditions and their difference wave. The gray areas highlight the time consistent top-down prediction (Hochstein and Ahissar, 2002).
windows of N2 (250–350 ms) used for statistical analysis. The scalp That is, the automatic predictive process is also available in the
distributions are time-locked to the peak amplitude of the difference wave
no-imagery condition but provides less specific predictions for
(Left: Mismatch minus Match; Right: No imagery minus Match).
the coming stimuli compared with the matching imagery. Such
under-specified predictions would induce greater mismatches
with the bottom-up process and hence elicit larger fronto-central
Consistent with these behavioral measures, ERP results N2 and longer RT. The second one is to explain this N2 as the
showed that, compared with the match condition, both mismatch effect of immediate context. Besides the mismatch interpretation,
and no-imagery conditions elicited a higher amplitude of the the fronto-central N2 has also been interpreted as a mecha-
fronto-central maximal N2 (250–350 ms window). No significant nism of novelty detection (for a review, see Folstein and Van
difference was observed for the P1 and N170 components. The Petten, 2008). Previous studies suggested that both long-term
observation of changes in the later N2 component but not in early context and immediate context contribute to the novelty response
visual processing components provides neural evidence support- (Daffner et al., 2000a,b). In the current study, the preceding
ing the matching level hypothesis: imagined visual stimuli affected imagery process of match trials provided immediate context for
the matching stage, rather than the earlier perceptual processing the subsequently presented face of the same trial, thus the pre-
stages in the visual identification task. Pearson correlation anal- sented face is relatively more novel in no-imagery trials than in
yses revealed that the longer the RT change, the larger the N2 the match trials; this novelty effect within the trial or in a short-
amplitude change from match to mismatch/no imagery condi- term/immediate context elicited the N2 in the no-imagery con-
tion, but not for P1 or N170 amplitude change, providing further dition. The matching stage of face identification in no-imagery
evidence for the matching level hypothesis. trials has to depend on the mechanism of the long-term memory
The mismatch condition elicited higher amplitude of the system (Grützner et al., 2010), then more time and higher neu-
fronto-central maximal N2 during the 250–350 ms window, and ral cost is required as compared with that in the match condition
this result is consistent with previous findings using repetition during which the short-term memory trace can be accessed from
paradigms. The fronto-central N2 component at this time range preceding mental imagery.

It might be argued that the difference between conditions can affected the early perceptual processing of subsequent test faces
also be interpreted as an Late Positive Complex (LPC) effect but (no task on the test faces) and such effects could generalize to
not the N2 effect, i.e., the match condition elicited more positive other object categories. The choice of only including face stimuli
potentials than both the mismatch and no-imagery conditions. in our study is because of the emphasis on testing the perceptual
The fronto-central LPC has been related to the old/new effect and matching level hypotheses in the context of face identifica-
where old stimuli elicit more positive LPC than do new stim- tion. But without stimuli from other categories as comparison
uli, and further studies suggested that this fronto-central old-new conditions, we cannot generalize the observed effects of visual
effect was an index of familiarity (for a review, see Rugg and imagery to other perceptual categories. The second limitation
Curran, 2007). In the current study, we interpreted that the no is that only two tokens of face stimuli were used in this study.
imagery condition was relatively more novel in a short-term con- This choice is because, arguably, fewer tokens to imagine would
text, and thus elicited more negative fronto-central N2 than the result in more vivid visual reconstructions after equal amounts
match condition and we interpreted it as a mechanism of nov- of training, and hence increase the effect sizes of mental imagery
elty detection, i.e., novel faces elicit more negative N2. Thus, on subsequent perception. However, participants could adapt
the trend between conditions was the same for both compo- strategies by visualizing either specific parts/features or the global
nents/interpretations. More importantly, both components have configuration of a face, thus it is not clear which processing mech-
the same scalp distribution and similar time window. Thus, we anisms would account for the observed effects of mental imagery
argue that both components and interpretations are not contrary on visual face perception. The third limitation is that the 100 ms
and may in fact reflect similar cognitive processing. For the dif- pre-face stimulus baseline might bring in inequity between the
ference between match and mismatch conditions, we focus on the imagery and no-imagery conditions. Although in both the cur-
mismatch interpretation because the current experimental design rent study and previous studies (Farah et al., 1988; Ganis and
is quite different from that used in research on the old/new effect. Schendan, 2008) participants were asked to press a button to
The distinct temporal profiles of neural correlates suggest separate the mental imagery processing period and the follow-
that task demand may be an important factor determining the ing perception period, participants could inertially perform visual
imagery-perception interaction in the hierarchical visual process. imagery and the visual trace was still available during the pre-
The present study revealed that the effects of mental imagery face stimulus interval for the imagery conditions, but not for the
on visual identification occur at a later matching stage, where no-imagery condition. Such potential overlap in the imagery con-
feature information from bottom-up processes is matched with ditions may influence the ERP baseline and result in an amplitude
that from top-down processes to achieve recognition. Thus the shift in the ERPs to the subsequent face stimuli.
matching level hypothesis is supported in a visual identification In conclusion, the effect of mental imagery on subsequent face
task. Previous ERP studies provided neural evidence that, dur- identification is content-specific, i.e., mismatch between the pre-
ing a detection task or without an active task, imagery-perception sented and imagined face leads to an interference and match leads
interaction occurs at the earlier perceptual stage of visual process- to a facilitation effect. The ERP results suggested that both facili-
ing, thus supporting the perceptual level hypothesis (Farah et al., tation and interference effects in a face identification task occur at
1988; Ganis and Schendan, 2008). Task demand modulates not a later matching stage, but not in the early perceptual processing.
only the direction that imagery affects perception (Finke, 1986)
but also the cognitive stage(s) at which this influence occurs. ACKNOWLEDGMENTS
Some limitations should be noted. The first limitation is that This work was supported by the NSF China (30900442) and the
only face stimuli were used in this study. A previous study by Project for Young Scientists Fund, IP, CAS (O9CX042004). We
Ganis and Schendan (2008) demonstrated that face imagery acknowledge Richard Carciofo for his editorial assistance.
REFERENCES Psychol. Hum. Percept. Perform. 23, Djordjevic, J., Zatorre, R. J., and Jones- the time-course of object percep-
Bruce, V., and Young, A. (1986). 3–13. Gotman, M. (2004a). Effects of tion. Psychophysiology 43, 564–568.
Understanding face recognition. Craver-Lemley, C., and Reeves, A. perceived and imagined odors on Farah, M. J. (1985). Psychophysical evi-
Br. J. Psychol. 77, 305–327. (1987). Visual imagery selectively taste detection. Chem. Senses 29, dence for a shared representational
Cabeza, R., Burton, A. M., Kelly, reduces vernier acuity. Perception 199–208. medium for mental images and per-
S. W., and Akamatsu, S. (1997). 16, 599–614. Djordjevic, J., Zatorre, R. J., Petrides, cepts. J. Exp. Psychol. Gen. 114,
Investigating the relation between Daffner, K. R., Mesulam, M. M., Scinto, M., and Jones-Gotman, M. (2004b). 91–103.
imagery and perception: evidence L. F. M., Calvo, V., Faust, R., and The mind’s nose: effects of odor and Farah, M. J. (1989). Mechanisms of
from face priming. Q. J. Exp. Holcomb, P. J. (2000a). An elec- visual imagery on odor detection. imagery-perception interaction.
Psychol. A 50, 274–289. trophysiological index of stimulus Psychol. Sci. 15, 143–148. J. Exp. Psychol. Hum. Percept.
Craver-Lemley, C., and Arterberry, unfamiliarity. Psychophysiology 37, Eddy, M. D., and Holcomb, P. J. (2011). Perform. 15, 203–211.
M. E. (2001). Imagery-induced 737–747. Invariance to rotation in depth mea- Farah, M. J., Peronnet, F., Gonon,
interference on a visual Daffner, K. R., Scinto, L. F. M., Calvo, sured by masked repetition prim- M. A., and Giard, M. H. (1988).
detection task. Spat. Vis. 14, V., Faust, R., Mesulam, M. M., ing is dependent on prime duration. Electrophysiological evidence for a
101–119. West, W. C., et al. (2000b). The Brain Res. 1424, 38–52. shared representational medium for
Craver-Lemley, C., Arterberry, M. E., influence of stimulus deviance on Eddy, M., Schmid, A., and Holcomb, P. visual images and visual percepts.
and Reeves, A. (1997). Effects of electrophysiologic and behavioral J. (2006). Masked repetition prim- J. Exp. Psychol. Gen. 117, 248–257.
imagery on vernier acuity under responses to novel events. J. Cogn. ing and event-related brain poten- Finke, R. A. (1986). Some conse-
conditions of induced depth. J. Exp. Neurosci.12, 393–406. tials: a new approach for tracking quences of visualization in pattern

identification and detection. Am. perception: psychophysical evi- Pylyshyn, Z. W. (2002). Mental condition. Neurosci. Lett. 298,
J. Psychol. 99, 257–274. dence. J. Cogn. Neurosci. 9, imagery: in search of a theory. 155–158.
Folstein, J. R., and Van Petten, C. 476–489. Behav. Brain Sci. 25, 157–182. Webster, M. A., and MacLeod, D. I.
(2008). Influence of cognitive con- Ishai, A., and Sagi, D. (1997b). Visual Reeves, A. (1981). Visual-imagery (2011). Visual adaptation and face
trol and mismatch on the N2 imagery: effects of short- and long- lowers sensitivity to hue-varying, perception. Philos. Trans. R. Soc.
component of the ERP: a review. term memory. J. Cogn. Neurosci. 9, but not to luminance-varying, Lond. B Biol. Sci. 366, 1702–1725.
Psychophysiology 45, 152–170. 734–742. visual stimuli. Percept. Psychophys. Wu, J., Mai, X., Yu, Z., Qin, S., and
Freyd, J. J., and Finke, R. A. (1984). Itier, R. J., and Taylor, M. J. (2004). 29, 247–250. Luo, Y. J. (2010). Effects of discrep-
Facilitation of length discrimination N170 or N1? Spatiotemporal differ- Rugg, M. D., and Curran, T. (2007). ancies between imagined and per-
using real and imagined context ences between object and face pro- Event-related potentials and recog- ceived sound on the N2 compo-
frames. Am. J. Psychol. 97, 323–341. cessing using ERPs. Cereb. Cortex nition memory. Trends Cogn. Sci. 11, nent of the event-related potential.
Ganis, G., and Schendan, H. E. (2008). 14, 132–142. 251–257. Psychophysiology 47, 289–298.
Visual mental imagery and percep- Kosslyn, S. M., Thompson, W. L., Sagiv, N., and Bentin, S. (2001). Wu, J., Yu, Z., Mai, X., Wei, J., and
tion produce opposite adaptation and Ganis, G. (2006). The Case Structural encoding of human and Luo, Y. J. (2011). Pitch and loudness
effects on early brain potentials. for Mental Imagery, New York, NY: schematic faces: holistic and part- information encoded in auditory
Neuroimage 42, 1714–1727. Oxford University Press. based processes. J. Cogn. Neurosci. imagery as revealed by event-related
Gilbert, C. D. (1996). Plasticity in visual McDermott, K. B., and Roediger, H. L. 13, 937–951. potentials. Psychophysiology 48,
perception and physiology. Curr. 3rd. (1994). Effects of imagery on Segal, S. J., and Fusella, V. (1970). 415–419.
Opin. Neurobiol. 6, 269–274. perceptual implicit memory tests. Influence of imaged pictures and
Gong, X., Huang, Y. X., Wang, Y., and J. Exp. Psychol. Learn. Mem. Cogn. sounds on detection of visual and
Conflict of Interest Statement: The
Luo, Y. J. (2011). Revision of the 20, 1379–1390. auditory signals. J. Exp. Psychol. 83,
authors declare that the research
Chinese facial affective picture sys- Michelon, P., and Koenig, O. (2002). 458–464.
was conducted in the absence of any
tem. Chin. Ment. Heal. J. 25, 40–46. On the relationship between visual Semlitsch, H. V., Anderer, P., Schuster,
commercial or financial relationships
Grützner, C., Uhlhaas, P. J., Genc, E., imagery and visual perception: evi- P., and Presslich, O. (1986). A solu-
that could be construed as a potential
Kohler, A., Singer, W., and Wibral, dence from priming studies. Eur. tion for reliable and valid reduc-
conflict of interest.
M. (2010). Neuroelectromagnetic J. Cogn. Psychol. 14, 161–184. tion of ocular artifacts applied to
correlates of perceptual clo- Moulton, S. T., and Kosslyn, S. M. the P300 ERP. Psychophysiology 23,
sure processes. J. Neurosci. 30, (2009). Imagining predictions: 695–703. Received: 29 April 2012; accepted: 23
8342–8352. mental imagery as mental emula- Sreenivasan, K. K., Katz, J., and October 2012; published online: 08
Henson, R. N., Goshen-Gottstein, tion. Philos. Trans. R. Soc. Lond. B Jha, A. P. (2007). Temporal November 2012.
Y., Ganel, T., Otten, L. J., Quayle, Biol. Sci. 364, 1273–1280. characteristics of top-down mod- Citation: Wu J, Duan H, Tian X, Wang
A., and Rugg, M. D. (2003). Neisser, U. (1976). Cognition and ulations during working memory P and Zhang K (2012) The effects of
Electrophysiological and haemody- Reality. San Francisco, CA: W. H. maintenance: an event-related visual imagery on face identification:
namic correlates of face perception, Freeman. potential study of the N170 com- an ERP study. Front. Hum. Neurosci.
recognition and priming. Cereb. Pearson, J., Clifford, C. W., and Tong, ponent. J. Cogn. Neurosci. 19, 6:305. doi: 10.3389/fnhum.2012.00305
Cortex 13, 793–805. F. (2008). The functional impact of 1836–1844. Copyright © 2012 Wu, Duan, Tian,
Hochstein, S., and Ahissar, M. (2002). mental imagery on conscious per- Thierry, G., Martin, C. D., Downing, P., Wang and Zhang. This is an open-
View from the top: hierarchies and ception. Curr. Biol. 18, 982–986. and Pegna, A. J. (2007). Controlling access article distributed under the terms
reverse hierarchies in the visual sys- Perky, C. W. (1910). An experimental for interstimulus perceptual vari- of the Creative Commons Attribution
tem. Neuron 36, 791–804. study of imagination. Am. J. Psychol. ance abolishes N170 face selectivity. License, which permits use, distribution
Ishai, A., and Sagi, D. (1995). Common 21, 422–452. Nat. Neurosci.10, 505–511. and reproduction in other forums, pro-
mechanisms of visual imagery and Peterson, M. J., and Graham, S. E. Wang, H., Wang, Y., Kong, J., Cui, L., vided the original authors and source
perception. Science 268, 1772–1774. (1974). Visual detection and visual and Tian, S. (2001). Enhancement are credited and subject to any copy-
Ishai, A., and Sagi, D. (1997a). imagery. J. Exp. Psychol. 103, of conflict processing activity in right notices concerning any third-party
Visual imagery facilitates visual 509–514. human brain under task relevant graphics etc.

published: 14 September 2012
doi: 10.3389/fpsyg.2012.00329
Electrophysiological potentials reveal cortical mechanisms

for mental imagery, mental simulation, and grounded
(embodied) cognition
Haline E. Schendan 1,2 * and Giorgio Ganis 1,2,3
1
School of Psychology, University of Plymouth, Plymouth, Devon, UK
2
Massachusetts General Hospital, Athinoula A. Martinos Center for Biomedical Imaging, Charlestown, MA, USA
3
Department of Radiology, Harvard Medical School, Boston, MA, USA
Edited by: Grounded cognition theory proposes that cognition, including meaning, is grounded in sen-
sorimotor processing. The mechanism for grounding cognition is mental simulation, which
is a type of mental imagery that re-enacts modal processing. To reveal top-down, corti-
Reviewed by:
Fred Mast, Universität Bern, cal mechanisms for mental simulation of shape, event-related potentials were recorded
Switzerland to face and object pictures preceded by mental imagery. Mental imagery of the identi-
Karla K. Evans, Harvard Medical cal face or object picture (congruous condition) facilitated not only categorical perception
School, USA
(VPP/N170) but also later visual knowledge [N3(00) complex] and linguistic knowledge
Adam Reeves, Northeastern
University, USA (N400) for faces more than objects, and strategic semantic analysis (late positive complex)
*Correspondence: between 200 and 700 ms. The later effects resembled semantic congruity effects with
Haline E. Schendan, School of pictures. Mental imagery also facilitated category decisions, as a P3 peaked earlier for
Psychology, University of Plymouth, congruous than incongruous (other category) pictures, resembling the case when identical
Drake Circus, Plymouth, Devon, PL4
pictures repeat immediately. Thus mental imagery mimics semantic congruity and imme-
8AA, UK.
e-mail: haline.schendan@plymouth. diate repetition priming processes with pictures. Perception control results showed the
ac.uk opposite for faces and were in the same direction for objects: Perceptual repetition adapts
(and so impairs) processing of perceived faces from categorical perception onward, but
primes processing of objects during categorical perception, visual knowledge processes,
and strategic semantic analysis. For both imagery and perception, differences between
faces and objects support domain-specificity and indicate that cognition is grounded in
modal processing. Altogether, this direct neural evidence reveals that top-down processes
of mental imagery sustain an imagistic representation that mimics perception well enough
to prime subsequent perception and cognition. Findings also suggest that automatic men-
tal simulation of the visual shape of faces and objects operates between 200 and 400 ms,
and strategic mental simulation operates between 400 and 700 ms.
Keywords: mental imagery, visual shape perception, object categorization, face identification, semantic memory
priming, visual knowledge, embodiment and grounded cognition, event-related potential
INTRODUCTION motivation, intention; Pulvermuller, 1999; Barsalou, 2008). Like

Mental imagery is the ability to reactivate and manipulate imagistic theories of mental imagery, grounded cognition theory
modality-specific mental representations when current sensory challenges the dominant symbol systems paradigm inspired by
stimulation or overt motor action is absent, and this ability can formal theories of logic, language, and computation that proposes
be associated with the subjective experience of perceiving or act- that amodal symbol representations, which are independent from
ing within one’s mental world (e.g., “seeing with the mind’s eye”). the sensorimotor processes, support language, thinking, atten-
Mechanisms of mental imagery have been proposed in large scale tion, memory, and meaning (Fodor, 1983; Johnson-Laird, 1983;
network theories (Kosslyn et al., 2006). At an abstract level, such Pylyshyn, 2003). Recently, however, tests of grounded cognition
imagistic theories propose that, during mental imagery, modality- theory have yielded compelling evidence that modal processing
specific, long-term memory is reactivated in a top-down manner affects cognition, including meaning, even when task irrelevant,
and maintained via working memory processes so that they can be and vice versa (e.g., Tucker and Ellis, 1998; Wilson, 2002; Vigliocco
inspected and manipulated to achieve a task goal (Kosslyn, 1994; et al., 2006; Barsalou, 2008; Fischer and Zwaan, 2008; Kemmerer
Kosslyn et al., 2001; Ganis et al., 2003; Ganis and Schendan, 2011). et al., 2008; Chatterjee, 2010; Anderson et al., 2011; Hirschfeld and
Notably, the mechanisms proposed in these theories of mental Zwitserlood, 2011; Meteyard et al., 2011). However, little is yet
imagery resemble those in grounded (embodied) cognition theory, known about when, how, and how much cognition is grounded
which proposes that cognition is grounded in modal processing of and about the brain mechanisms (Mahon and Caramazza, 2009;
sensorimotor information and introspective states (e.g., emotion, Chatterjee, 2010; Rumiati et al., 2010; Meteyard et al., 2011). The
www.frontiersin.org September 2012 | Volume 3 | Article 329 | 77

Schendan and Ganis ERPs of shape mental imagery
main proposal for how cognition is grounded is mental simu- 1999; Puce et al., 1999). This experiment thus reveals when top-
lation, defined as the “re-enactment of perceptual, motor, and down processes for mental imagery of visual shape can ground
introspective states acquired during experience with the world, cognition of faces and other objects. Such neurophysiological
body, and mind” (Barsalou, 2008, 2009). The present experiment markers will be crucial for future work on when, how, and how
aims to reveal the cortical dynamics of mental imagery mecha- much top-down cortical processes of mental simulation ground
nisms that may ground cognition in mental simulations. Crucially, cognition.
the cortical dynamics of mental imagery and mental simulations Findings from this experiment have been reported already
that ground cognition are almost entirely unknown because elec- for early ERPs before 200 ms (Ganis and Schendan, 2008), that
tromagnetic brain sensing methods, which reveal neural activity is, the vertex positive potential and its occipitotemporal N170
directly with the required high time precision [milliseconds (ms)], counterpart (VPP/N170), which are associated with categorical
have not been applied, as done here. perception. The goal here was to analyze the later ERPs that were
An important distinction in grounded cognition theory to not analyzed previously and are associated with knowledge, mean-
consider is that between non-conscious automatic simulations ing, and category decisions. These abilities are the main targets for
implicated, for example, in constructing meaning from lan- grounded cognition explanations. The main hypothesis is that, if
guage (Bub and Masson, 2010; Wassenburg and Zwaan, 2010) mental simulation constructs cognition, including meaning, then
and conscious effortful simulations, such as mental imagery, mental simulation of modal processes induced by mental imagery
that result from conscious representations constructed in work- (and the associated cortical dynamics revealed by the ERPs) should
ing memory strategically (Kosslyn et al., 2006; Barsalou, 2008, resemble those associated with cognitive and semantic effects.
2009). Critically for the present study, both types of simula- If true, then this would constitute crucial evidence that links
tion share common representations (Barsalou, 2008; Moulton mental simulation of modal processing (using top-down mental
and Kosslyn, 2009). We hypothesize that, at the level of brain imagery mechanisms) with cognition and meaning, as proposed in
mechanisms, the top-down feedback mechanisms that support grounded cognition theory. To assess cognitive effects, this study
automatic simulation are a subset of those that support mental capitalizes on ERP markers of face and object cognition (Neu-
imagery (and conscious effortful simulations; Ganis and Schen- mann and Schweinberger, 2008; Schendan and Maher, 2009). To
dan, 2011). Specifically, we propose that non-conscious automatic assess semantic effects, in particular, this study capitalizes also
simulations unfold via reflexive top-down signals from higher to on well-studied semantic congruity and priming effects. These
lower level areas along modal information processing pathways effects are thought to reflect processing meaning in the semantic
(e.g., the ventral stream for processing visual objects): Perceiv- memory system (Rossell et al., 2003) and to operate via the same
ing a stimulus triggers these processes reflexively. These same automatic top-down processes implicated in automatic mental
processes are triggered by effortful, strategic, task-oriented, top- simulation (Collins and Loftus, 1975; Franklin et al., 2007; Kutas
down signals from the prefrontal cortex during mental imagery and Federmeier, 2011).
(Ganis and Kosslyn, 2007), which also triggers them most strongly To predict the specific pattern of ERP cognitive, semantic con-
due to the deliberate, targeted nature of the task requirements. gruity, and priming effects that mental imagery could produce, this
Thus, studying mental imagery provides a powerful way to report capitalizes on the two-state interactive (2-SI) account of the
reveal the maximum set of top-down feedback mechanisms that brain basis of visual object cognition (Schendan and Kutas, 2003,
support mental simulation, including non-conscious automatic 2007; Schendan and Maher, 2009) and extends it into a multi-state
simulation. interactive (MUSI) account. This framework proposes that pos-
To ensure that mental imagery mechanisms underlie the effects, terior object processing areas activate at multiple times in brain
this experiment used a validated mental imagery task (Kosslyn “states” serving distinct functions. State 1: Initial activity in object
et al., 2006). The key task elements are that subjects first mem- processing areas feeds forward from occipital to temporal cortex
orize pictures of faces of real people or objects extensively and between ∼120 and ∼200 ms when a visual object is coarsely per-
then practice visualizing these pictures mentally. Afterward, dur- ceptually categorized, indexed by the VPP/N170 (Schendan et al.,
ing the mental imagery task, the name of the person (or object) 1998; Schendan and Lucia, 2009). State 2: Object processing areas
cues mental visualization of the associated picture of the face (or activate again but in a sustained, interactive manner dominated
object). The critical and novel element of this design is that, after by feedback and recurrent processing among these areas and ven-
subjects report (by pressing a key) that they have generated a trolateral prefrontal cortex (VLPFC) between ∼200 and 500 ms,
vivid mental image, a target picture appears 200 ms later. This indexed by a frontal N3(00) complex (often labeled frontal N400).
picture is either identical to the picture that they had learned to The N3 is specific to processing pictures of a face or object and non-
visualize (congruous condition) or different from it, being from linguistic information (e.g., shape; Barrett and Rugg, 1989, 1990;
the opposite category (i.e., if a face was visualized, an object McPherson and Holcomb, 1999; Nessler et al., 2005) and is the first
is shown, and vice versa: incongruous condition). The impact of ERP to modulate according to visual cognitive factors that simi-
the imagined picture on the target picture is used to define the larly affect object processing areas and VLPFC, including semantic
cortical dynamics of mental imagery. The advantage to com- memory (Barrett and Rugg, 1990; Zhang et al., 1995; Doniger
paring mental imagery of these two categories is that faces and et al., 2000, 2001; Schendan and Kutas, 2002, 2003, 2007; Phil-
objects recruit different posterior visual processing areas (Hasson iastides and Sajda, 2006, 2007; Philiastides et al., 2006; Schendan
et al., 2003; Downing et al., 2006) and are associated with dis- and Lucia, 2009, 2010; Schendan and Maher, 2009). For exam-
tinct ERP signatures (e.g., Schendan et al., 1998; Allison et al., ple, the N3 becomes more negative with greater mental rotation
Frontiers in Psychology | Perception Science September 2012 | Volume 3 | Article 329 | 78

(Schendan and Lucia, 2009), worse speed and accuracy of category and Knight, 1994). In addition, we report results of new analyses
decisions, greater stimulus atypicality and impoverishment (e.g., of data from the perception control experiment conducted in the
atypical views, visual degradation; Doniger et al., 2000; Schendan same group of participants (Ganis and Schendan, 2008) that had
and Kutas, 2002, 2003; Johnson and Olshausen, 2003), and implicit not been explored previously (i.e., face ERPs after 500 ms and all
memory (repetition priming, i.e., better decisions for repeated object ERPs). As ERPs to faces starting from the early VPP/N170
than new objects; Henson et al., 2004; Schendan and Kutas, 2007) until 500 ms show perceptual adaptation (i.e., reduced for con-
primarily for meaningful objects (e.g., dog; Voss et al., 2010). Such gruous; Ganis and Schendan, 2008), the N3, N400, and LPC ERPs
effects also localize to object processing areas (David et al., 2005, should likewise show adaptation.
2006; Philiastides and Sajda, 2006, 2007; Sehatpour et al., 2008;
Schendan and Maher, 2009; Schendan and Lucia, 2010; Clarke MATERIALS AND METHODS
et al., 2011). Later in state 2, the well-studied, centroparietal N400 Participants, materials, design, procedures, and electroencephalo-
between 300 and 500 ms reflects interactive activation of seman- graphic (EEG) recording methods for these mental imagery and
tic memory in anterior temporal cortex and VLPFC, especially in perception control experiments were detailed previously (Ganis
response to words (i.e., linguistic stimuli; Marinkovic et al., 2003; and Schendan, 2008). This section summarizes key aspects of
Lau et al., 2008; Kutas and Federmeier, 2011). Both mid-latency the methods for understanding this report. The mental imagery
negativities (N3 and N400) are more negative for stimuli that experiment started with memorization of pictures of 11 faces
are incongruous (i.e., non-matching) relative to congruous (i.e., of real people and 11 common real objects across 13 exposures
matching) with the immediately preceding semantic context based each. Next, participants practiced mentally visualizing each mem-
on a sentence or scene (Ganis et al., 1996). A similar account has orized picture three times. For this imagery practice, the name
been proposed for face cognition (Neumann and Schweinberger, appeared followed by a gray screen during which subjects visual-
2008; Burton et al., 2011). ized the picture of the named face (object). Once they had done
The MUSI account revises this story by adding State 3 that so, they pressed a key to see the actual picture in order to adjust
operates after about 400 to 500 ms and performs internal eval- their mental image. For the mental imagery test (Figure 1A),
uation and verification processes, including conscious effortful the name appeared for 300 ms followed by a gray screen dur-
mental simulations. For example, later verification of category ing which subjects visualized mentally the associated memorized
decisions, more complex semantic processes, and episodic mem- picture. As soon as they had generated this mental image, they
ory have been associated with a posterior late positive complex pressed a key. Two-hundred milliseconds after the key press the
(LPC) after ∼500 ms (Schendan and Kutas, 2002; Rugg and test picture appeared for 300 ms. The test picture was either the
Curran, 2007; Voss et al., 2010) that is, instead, more positive picture of the face (object) that was visualized mentally or a pic-
to incongruous than congruous semantic contexts with objects, ture from the other category [i.e., an object (a face)]. The two
videos, and faces (Ganis et al., 1996; Schendan and Kutas, 2002; by two design included within-subject factors of image congruity
Ganis and Kutas, 2003; Sitnikova et al., 2009, 2010). Altogether, (congruous, incongruous) by category type of target picture (face,
this predicts that the N3 and N400 will be more negative and object). There were 55 trials for each of the four critical combi-
the LPC will be more positive when mental imagery does not nations (image-picture target pairings) of congruous (face–face;
match the current picture (incongruous) than when it does match object–object) and incongruous (object-face; face object) condi-
(congruous). Finally, cortical sources of the ERPs to faces and tions. Stimuli were presented on a 2100 cathode-ray tube monitor
objects and associated congruity effects should differ because (1,024 × 768, 150 Hz refresh, Dell P1130) using custom-made
these two categories recruit distinct cortical areas (Hasson et al., StimPres2.0 software for the Neurocog system that ensures pre-
2003). cise stimulus time-locking to the EEG (Holcomb, 2003). EEG was
Prior work on mental imagery and semantic congruity and recorded at 250 Hz (bandpass .01–100 Hz) from 32 Ag/AgCl elec-
priming would not necessarily predict such later effects, how- trodes attached to a plastic cap (Figure 1B) and electrodes attached
ever. The only prior ERP study with a similar mental imagery via adhesive to the nose, right and left mastoids, underneath the
task revealed no effects after 300 ms, predicting no effects here right eye, and lateral to each eye. The perception control experi-
(Farah et al., 1988), but that study involved imagery of two let- ment with the same subjects was identical, except for the following.
ters, which have minimal meaning. In contrast, the present work There were no memorization and imagery practice sessions, and
aims to reveal mental imagery for stimuli with richer seman- participants were shown a picture instead of a name before the test
tic content. Studies of semantic congruity and priming effects picture: The two pictures were shown consecutively with a 200 ms
would also predict no ERP effects of mental imagery after 200 interstimulus interval (ISI), and the first picture was shown until
or 300 ms because the slow stimulus timing and cross modal con- the participant pressed a key to report that they had identified the
ditions here (i.e., word cue followed by picture) violate standard object (or face). There were 66 trials in each of the four critical
methods for producing such semantic effects, as detailed in the dis- conditions. The primary goal of the control experiment was to
cussion. To anticipate, results reveal effects not only on the early replicate perceptual adaptation of the early VPP/N170, and so, in
VPP/N170 during categorical perception, as reported previously order to focus on higher-level face (object) processing in poste-
(Ganis and Schendan, 2008), but also later ERPs (N3, N400, and rior category specific cortex, instead of low-level simple features,
LPC) linked to knowledge, meaning, and categorization, as well the first and second pictures were never identical, even in face–
as a P3(00) linked to immediate repetition priming of perceived face and object–object (congruous) trials (Ganis and Schendan,
faces and objects (Bentin and McCarthy, 1994; Nielsen-Bohlman 2008).

except bilateral eyes. Maps of voltage distribution across the head

A Mental imagery were produced using EEGLab software. Eighteen subjects were
1,700 analyzed; note, due to a scripting error, 1 of the original 19 subjects
was not analyzed, but performance results and visual inspection
of ERP patterns showed that all results remained the same with
300 and without this subject.
Data were submitted to analyses of variance (ANOVAs) with
200 within-subject factors of congruity and category type (type). For
ERPs, ANOVAs assessed separately the lateral pairs (1–26) and
midline sites (27–32). Lateral ANOVAs included within-subject
Self-paced
factors of Electrode (13 levels) and Hemisphere (left, right). Mid-
line ANOVAs included within-subject factors of Electrode (3 lev-
300 els) and Site [frontocentral (27–29), occipitoparietal (30–32)]. If
BRAD PITT Mauchley’s test indicated violations of sphericity, the Greenhouse–
(ms) Geisser correction was applied to the p-value. For brevity, only
critical Congruity and Type effects are reported, as scalp location
effects alone are not of theoretical interest.
B
nose Mean ERP amplitudes were measured within time periods after
200 ms chosen based on prior studies. (1) As N3(00) complex
1 2 components can vary in functional modulation in ∼100 ms time
Fpz
periods between 200 and 500 ms (Schendan and Maher, 2009),
5 27 6 analyses assessed separately (a) 200–299 ms (frontopolar P250 and
~F73 4~F8 related polarity inverted occipitotemporal Ncl/N250; e.g., Doniger
Fz et al., 2000; Federmeier and Kutas, 2002; Sehatpour et al., 2006;
11 12 Schendan and Lucia, 2010), (b) 300–399 ms (frontal N350; e.g.,
9 10
~T37 8~T4 Schendan and Kutas, 2002, 2003, 2007), and (c) 400–499 ms (fron-
28
15 Cz 16 topolar N450 and frontocentral N390; e.g., Barrett and Rugg, 1989;
13 14 Ganis and Kutas, 2003; Schendan and Maher, 2009). The latter two
EAR 29 EAR
21 22
times (300–500 ms) also included the N400 (Ganis et al., 1996). (2)
17 18
~P9 19 20 ~P10 The LPC was assessed from 500 to 699 ms (Heil, 2002; Schendan
~P7 30 ~P8 and Lucia, 2009). (3) Continuation of effects was assessed from
25 26 700 to 899 ms (Schendan and Maher, 2009).
~O1 31 Oz ~O2
~PO9 ~PO10 To isolate effects, focal spatiotemporal analyses were run on sites
23 32 24
and times for which the face (object) cognition-, congruity-, or
priming-related ERP was maximal and overlapped least with oth-
FIGURE 1 | Methods. (A) Diagram of an experimental trial for the test of
ers; these location choices were confirmed and adjusted by visual
mental imagery for faces and objects (only a congruous trial of face
imagery – face picture is shown). An appropriate face (or object, not shown) inspection. (1) From 200 to 299, 300 to 399, and 400 to 499 ms,
was visualized mentally upon seeing the corresponding name (which was respectively, (a) pair 1-2 assessed the frontopolar P250, N350, and
on the screen for 300 ms). Subjects pressed a key as soon as they had N450, (b) pair 11-12 assessed the frontocentral N350 and N390,
generated a vivid mental image, and 200 ms after this key press, the test (c) and pair 17-18 assessed their polarity inverted, occipitotempo-
picture appeared, which was either the identical picture that had been
imaged mentally (congruous condition) or not (in which case the picture
ral counterparts (Scott et al., 2006; Schendan and Maher, 2009).
was from the other category: incongruous condition). There was no task on (2) These times (200–499 ms) were also assessed at pair 19-20 for
the test pictures. Perception trials (not shown) had a parallel structure to the centroparietal N400 (Ganis et al., 1996); note, 200–299 ms was
imagery, except that the trial started with perception of a picture of a face included as visual inspection suggested an N400 onset before its
(object) was presented until subjects pressed a key as soon as they typical 300 ms start. (3) Centroparietal pair 19-20 also assessed
identified it. (B) Thirty-two-channel geodesic montage for EEG recording.
Circles show each electrode location with its numerical label. Actual or
the LPC from 500 to 699 ms. (4) Visual inspection indicated that
approximate (∼) locations of 10–10 sites are italicized and in red; posterior a P3 peaked earlier for congruous (∼375 ms) than incongruous
sites 23, 24, and 32 lie 1 cm below the inion. stimuli (∼500 ms). Consequently, early on, the congruity effect
is more positive for congruous than incongruous from 300 to
400 ms, and later, from 400 to 700 ms, the effect is in the opposite
ERP ANALYSES direction. To capture this, local positive peak latency (i.e., highest
ERPs were calculated by averaging EEG to each condition, exclud- peak within ±20 ms to avoid spurious peaks due to high frequency
ing trials with above threshold muscle activity, blinks, eye, and noise) between 300 and 699 ms was assessed in each condition at
other movement artifacts, time-locking to test picture onset midline occipitoparietal site 30, based on the location of simi-
with a 100 ms pre-stimulus baseline. For analyses, ERPs were lar immediate repetition effects on the P3 (Bentin and McCarthy,
re-referenced to the mean of both mastoids, and, for visual com- 1994). P3 mean amplitude was also assessed from 300 to 399, 400
parison with prior work, also to the average of all electrodes, to 499, and 500 to 699 ms at site 30. (5) Since visual inspection

suggested frontal effects continued after 500 ms, frontopolar and MENTAL IMAGERY ERPs
frontocentral focal analyses were also run from 500 to 699 and Mental imagery results for faces during the first 500 ms and objects
700 to 899 ms. For focal analyses, which are more precise (albeit during the first 200 ms were reported previously (Ganis and Schen-
less comprehensive) than omnibus analyses, congruity by category dan, 2008). The new ERP results here reveal congruity effects
interactions were assessed further using planned simple effect tests to faces after 500 ms (Figure 2), those to objects after 200 ms
of congruity for each category condition. (Figure 3), and category type effects (Figures 6 and 7) and inter-
For perception control ERPs, analyses were the same as for actions of congruity by type after 200 ms (Figures 4–6). For
imagery, except for the following. Face ERP analyses through comparison with previous work, ERPs are also plotted with the
500 ms were already carried out and reported (Ganis and Schen- common average reference (Figures 8 and 9). For brevity, (a) only
dan, 2008) and so not duplicated here. ERPs for faces after 500 ms congruity and category type effects, which are of theoretical inter-
and those for objects after 200 ms were analyzed separately, and, est, are reported, (b) degrees of freedom (df) are listed only for the
for brevity in reporting these control data, comparisons between first report of each effect, and (c) planned contrasts for omnibus
faces and objects are reported only for focal spatiotemporal analy- results are not reported, except to note that they supported the
ses, and omnibus analyses are not reported, but they confirmed corresponding focal spatiotemporal results. For the focal results
the focal results. [all dfs (1, 13)], any interactions of congruity and category type
were followed with corresponding contrast ANOVAs that assessed
SOURCE ESTIMATION the congruity effects to objects and faces, separately.
Source estimation methods evaluated whether distinct sources
underlie congruity effects between categories and category dif- 200–500 ms: N3, N400, AND P3
ferences. The inverse problem of localizing the cortical sources Negativity on the N3 was greater for incongruous than congru-
of electromagnetic data recorded from the scalp has no unique ous imagery, and the N400 showed this pattern only for faces
solution without additional constraints. Standardized low resolu- (Figures 2–4). Negativity was greater for objects than faces for
tion brain electromagnetic tomography (sLORETA) estimates the the N3, regardless of congruity, and the N400 showed this pattern
sources (Pascual-Marqui, 2002) by making a maximum smooth- for congruous imagery but showed the opposite for incongruous
ness assumption to compute the three-dimensional (3D) distri- imagery (Figure 6), consistent with the N400 congruity effect for
bution of current density using a standardized, discrete, 3D dis- faces but not objects. Congruity effects had a frontocentral max-
tributed, linear, minimum norm inverse solution. Localization is imum for faces (Figure 2B) and a frontopolar maximum instead
data-driven, unbiased (even with noisy data), and exact but has low for objects (Figure 3B), and, accordingly, object and face categories
spatial precision due to smoothing assumptions resulting in highly differed mainly frontally (Figure 7). All effects inverted polarity
correlated adjacent cortical volume units. A realistic head model over occipitotemporal sites. Accordingly, omnibus results (Table 1)
constrains the solution anatomically using the structure of corti- showed congruity and category type effects, and interactions of
cal gray matter from the Montreal Neurological Institute (MNI) congruity by category type from 200 to 400 ms in lateral and mid-
average of 152 human brains as determined using a probabilistic line ANOVAs and also from 400 to 500 ms in lateral ANOVAs, but
Talairach atlas. showed only category type and congruity by category type interac-
sLORETA software computed the sources of the grand aver- tions in midline ANOVAs from 400 to 500 ms when N3 congruity
age ERP difference waves using data from all sites, except nose effects to objects ended.
and eyes (Pascual-Marqui, 2002). ERP difference data are analo-
gous to the signal changes between fMRI conditions, and, thus, N3 complex
limit the sources to those that could reflect fMRI differences, and Focal spatiotemporal results (Table 2) confirmed frontopolar N3
difference waves can reveal weaker sources better (Luck, 2005). congruity effects for objects, frontocentral N3 congruity effects
Data were analyzed with bandpass filter of 0.01–20 Hz, based for both categories, and centroparietal N400 congruity effects
on the validated sLORETA analyses reported previously for the for faces, and occipitotemporal polarity inversion of congruity
VPP/N170 (Ganis and Schendan, 2008). Electrode coordinates effects for objects, as well as category type effects. Specifically,
were digitized using an infrared digitization system, and imported results showed type effects on the entire N3 complex from 200 to
into LORETA-Key software. This coordinate file was converted 500 ms, as frontal negativity and occipitotemporal positivity were
using sLORETA electrode coordinate conversion tools. The trans- greater for objects than faces (Figure 7). Congruity was signifi-
formation matrix was calculated with a regularization parameter cant at frontopolar and frontocentral sites from 200 to 400 ms,
(smoothness) corresponding to the signal-to-noise (SNR) ratio and, at occipitotemporal sites, congruity was marginal from 200
estimated for each difference wave separately at each 100 ms time to 300 ms [F (1, 17) = 3.98, p = 0.062]; note, while occipitotempo-
period of interest from 200 to 900 ms relative to the 100 ms ral congruity was also significant from 400 to 500 ms, this was due
baseline. to the start of the posterior LPC. Congruity interacted with type
from 200 to 300 ms at frontopolar sites and from 200 to 500 ms at
RESULTS frontocentral sites (Figures 2 and 3). Planned simple effects tests
These results cover all times and comparisons not analyzed for of the congruity effect for each type showed that this was because
our prior report on this study, which focused mostly on face congruity effects were largest for objects at frontopolar sites and
ERPs before 200 ms during imagery and perception (Ganis and largest for faces at frontocentral sites and inverted polarity occipi-
Schendan, 2008). totemporally for objects. Specifically, Table 2 shows congruity was

A Face Mental Imagery
3 5 1 27 2 N3 6 4
N3
7 9 11 12 10 8
28
VPP
13 15 16 14
29
N170
17 19 21 30 22 20 18
N400 LPC
P3
Incongruous
Congruous
31
2 µV
P3
23 25 26 24
0 400 800 ms
N400 32
P3
B Max - 200-300 300-400 400-500 500-700 700-900 ms
Max +
Congruity (Incongruous - Congruous)
FIGURE 2 | Mental imagery congruity effects for faces. (A) Grand average ERPs between −100 and 900 ms at all 32 channels shown filtered low-pass 30 Hz.
(B) Maps of voltage across the scalp for difference waves of the ERPs in (A). VPP, vertex positive potential; LPC, late positive complex.
significant at frontopolar sites for objects from 200 to 300 ms and sites were run on later times. Results (Table 2) at frontopolar sites
for both category types from 300 to 400 ms, and at frontocentral confirmed no congruity effects there after 400 ms and type effects
sites for both from 200 to 400 ms and then later only for faces from ongoing between 200 and 900 ms. Frontocentral results confirmed
400 to 500 ms. At occipitotemporal sites, congruity was significant that type effects continued, and congruity effects remained larger
only for objects from 200 to 300 ms during the N3 and later from for faces than objects until 900 ms, and, indeed, were significant
400 to 500 ms when LPC effects start. only for faces from 400 to 900 ms. In sum, frontopolar and occip-
These results and visual inspection indicated that, while fron- itotemporal object congruity effects occurred between 200 and
topolar N3 congruity effects for objects ended by 400 ms, fronto- 400 ms, whereas frontocentral face congruity effects occurred from
central congruity effects to faces, as well as type effects at these sites, 200 to 900 ms, as did type effects at these times and sites, and
continued after 500 ms. To assess this, focal analyses of these frontal occipitotemporal sites also showed LPC effects after 400 ms.

A Object Mental Imagery

N3
3 5 1 27 2 6 4
N3
7 9 11 12 10 8
28
VPP
13 15 16 14
29
N170
17 19 21 30 22 20 18
LPC
Incongruous P3
Congruous
31 P3
2 µV
23 25 26 24
0 400 800 ms
32
B Max - 200-300 300-400 400-500 500-700 700-900 ms
Max +
Congruity (Incongruous - Congruous)
FIGURE 3 | Mental imagery congruity effects for objects. (A) Grand average ERPs between −100 and 900 ms at all 32 channels shown filtered low-pass
30 Hz. (B) Maps of voltage across the scalp for difference waves of the ERPs in (A) within each analysis time period.
N400 and, instead, N3 congruity effects for objects with inverted polarity
Focal spatiotemporal results (Table 2; Figure 4C) at centroparietal at these sites (i.e., most negative for congruous) as observed at adja-
sites showed a category type effect from 400 to 500 ms, congruity cent occipitotemporal sites (Figure 3). Later, from 300 to 400 ms,
effects from 300 to 500 ms, and congruity by type interactions congruity affected faces in the predicted direction (i.e., most nega-
from 200 to 500 ms. The N400 for faces was larger over the left tive for incongruous), as N400 congruity was significant for faces.
hemisphere, as demonstrated by three-way interactions of type From 400 to 500 ms, posterior LPC congruity effects began: Con-
by congruity by hemisphere that were significant from 300 to gruity was significant for objects and congruity by hemisphere
400 ms and marginal before and after [200–300 ms F (1, 17) = 3.05, was marginal for faces, as the effect was larger on the right (F [1,
p = 0.099; 400–500 ms F = 3.89, p = 0.065]. Planned simple effects 17] = 4.12, p = 0.058). In sum, centroparietal sites showed occip-
tests (Table 2) from 200 to 300 ms showed no N400 effects for faces, itotemporal polarity inversion of N3 congruity effects for objects,

Mental Imagery Congruity Mental Imagery Congruity

Incongruous Incongruous
Congruous Congruous
Faces Objects Objects
A Faces
A N3 N3
P3
Fronto-
polar
right
P3
N3
P3
1 µV P3
N3 Fronto-
central P3
0 300 600 ms
right
VPP
Incongruous - Congruous
max +
1 µV B 400-500 500-600 ms 400-500 500-600 ms
0 200 400 ms
VPP
Incongruous - Congruous
B 200-300 300-400 ms max + 200-300 300-400 ms
0
C LPC
0
C N400 & P3
Centro- LPC
parietal LPC
N400
left
N1 FIGURE 5 | Mental imagery congruity effects on the P3 and late

Parietal positive complex (LPC). Grand average ERPs between −100 and 700 ms
left show congruity effects (A) on a P3 at a midline parietal site (30) and (C) on
a late positive complex (LPC) at a right centroparietal site (20). Filtered
low-pass 30 Hz. Shading captures the 500–700 ms time when the LPC is
P1 maximal. (B) sLORETA sources of congruity difference waves at times of
the P3 and LPC. Images plot the magnitude of the estimated current
density based on the standardized electrical activity in each of 6,239 voxels
P3 of 5 mm3 size.
P3
FIGURE 4 | Mental imagery congruity effects on the N3 and N400 and

P3. Grand average ERPs between −100 and 400 ms at (A) right frontopolar
showed category type and congruity effects from 500 to 900 ms
(2) and right frontocentral (12) N3 sites and (C) left centroparietal N400 (19) and congruity by type interactions from 500 to 700 ms (Table 1),
and parietal P3 (25) sites. Filtered low-pass 30 Hz. VPP, vertex positive reflecting continuing frontal type effects and frontocentral con-
potential. (B) sLORETA sources of congruity difference waves at times of gruity effects with faces, as reported above for frontal focal results,
the N3, N400, and P3. Images plot the magnitude of the estimated current
and continuing LPC type effects until 900 ms and congruity effects
density based on the standardized electrical activity in each of 6,239 voxels
of 5 mm3 size.
until 700 ms. Focal results at centroparietal pair 19-20 confirmed
the posterior distribution of the LPC, showing category type
and congruity effects (Table 2); as congruity and type did not
N400 congruity effects for faces, and, after 400 ms, the start of interact, congruity affected both category types. Planned simple
posterior LPC congruity effects (Figures 2–4). effects tests (Table 2) showed that congruity effects continued
until 700 ms for objects and were in the same direction for faces
500–900 ms: LPC (i.e., more positive for incongruous) but did not reach significance
As the earlier centroparietal focal results indicated, LPC congruity [500–700 ms: congruity by hemisphere, F (1, 17) = 2.28, p = 0.15]
effects began around 400 ms. Posterior positivity is greater for perhaps due to ongoing frontal congruity effects in the opposite
incongruous than congruous pictures (Figures 2, 3, and 5C) and direction that may partly cancel out the posterior LPC effect for
for objects than faces (Figure 7). Accordingly, omnibus results faces.

repetition priming that makes the P3 earlier and larger (Bentin

Mental Imagery Category and McCarthy, 1994). Likewise, here, the P3 appeared to peak
Objects earlier for congruous than incongruous stimuli, resulting in a
Faces P3 that is more positive for congruous than incongruous ini-
Congruous Incongruous tially and then later shows the opposite (Figures 2, 3, and 5A).
N3
A N3 Results of ANOVAs on local positive peak latency data at midline
N3 occipitoparietal site 30 confirmed that the P3 peaked earlier for
Fronto- congruous (426 ms) than incongruous (496 ms) stimuli [con-
polar
gruity, F (1, 17) = 11.41, p = 0.004], regardless of category. Peak
right
latency captured the P3 pattern better than mean amplitude due
to overlapping N400 and LPC effects. P3 mean amplitude results
at site 30 showed only that the P3 was more positive for congru-
N3 ous than incongruous faces from 300 to 700 ms and marginally
the opposite (more positive for incongruous than congruous) for
N3
Fronto- objects from 400 to 500 ms; there were significant effects of con-
central
gruity (300–400 ms, F = 20.61, p < 0.001) and congruity by type
right
(300–700 ms: F s > 13, ps < 0.003) due to congruity being signif-
icant for faces (300–500 ms, F s > 7.45, ps < 0.015) and marginal
for objects (400–500 ms, F = 3.63, p = 0.074).
VPP
MENTAL IMAGERY SOURCES
Because faces and objects recruit distinct occipitotemporal areas
VPP
(Hasson et al., 2003), the cortical sources of mental imagery should
Object - Face
differ between these categories. Estimated cortical sources of each
B 200-300 300-400 ms max + 200-300 300-400 ms
of the four difference waves were consistent with known prefrontal
and posterior face (object) processing areas. MNI coordinates are
reported for the maximum activated region (x y z) and up to four
anatomically distinct sources, the Brodmann’s areas (BA) for all,
and the BA for up to four secondary sources that are contiguous
0 with the maximum; this captured all clear sources.
C N400 & P3
Centro- Congruity (incongruous–congruous)
parietal N400
For faces, results were consistent with ERP and fMRI evidence for
left prefrontal and temporal lobe generators during recognition and
priming of faces (Henson et al., 2003). Specifically, congruity for
N170 faces from 200 to 400 ms during the N3 and N400 (and over-
N170
lapping P3) reflects sources in medial prefrontal cortex, VLPFC,
Parietal and superior temporal cortex: Figure 4B shows sources observed,
left from 200 to 300 ms, in medial frontal gyrus [BA9 at 5 45 25 and,
N400 from 300 to 400 ms in VLPFC (BA47 at 15 35 −30; BA11)] and
superior temporal gyrus (STG; BA38). After 400 ms, P3 and LPC
congruity reflect sources in prefrontal and middle temporal cor-
1 µV
tex: Figure 5B shows sources observed in middle temporal gyrus
P3 0 200 400 ms (400–500 ms BA21 at 70 −35 −5; 500–700 ms BA21 at 70 −35
−10) and, after 500 ms, also in medial and superior frontal gyrus
FIGURE 6 | Mental imagery category effects on the N3 and N400. Grand (not shown; 500–600 ms, BA9/10 at 0 55 25; 600–700 ms, BA9/10
average ERPs between −100 and 400 ms at (A) right frontopolar (2) and at 5 60 30).
right frontocentral (12) N3 sites and (C) left centroparietal N400 (19) and For objects, N3 congruity effects occurred from 200 to 400 ms
parietal P3 (25) sites. Filtered low-pass 30 Hz. VPP, vertex positive potential.
and so this time was of primary interest. Results were consistent
(B) sLORETA sources of category difference waves at times of the N3,
N400, and P3. Images plot the magnitude of the estimated current density
with N3 and fMRI evidence for ventral object processing stream
based on the standardized electrical activity in each of 6,239 voxels of generators during categorization and priming (Henson et al., 2004;
5 mm3 size. Schendan and Stern, 2008; Schendan and Maher, 2009; Schen-
dan and Lucia, 2010). Specifically, during the N3 (Figure 4B),
sources were observed, from 200 to 300 ms, in middle tempo-
P3 PEAK LATENCY AND AMPLITUDE ral gyrus (BA21 at 70 −35 −10) extending to inferior (BA37)
Visual inspection revealed prominent parietal P3-like peaks and superior temporal (BA22) and fusiform (BA37) and middle
between 300 and 700 ms, resembling immediate perceptual occipital gyri (BA19) and, from 300 to 400 ms, at the junction

Mental Imagery Category

A (Objects - Faces)
Congruous
Max -
B Max +
Incongruous
Time 200-300 300-400 400-500 500-700 700-900 ms
FIGURE 7 | Volt maps of mental imagery category effects. Maps plot voltage interpolated across electrode locations in each analysis time window for
difference waves that define the category effect on (A) Congruous and (B) Incongruous trials, as highlighted in Figure 6 for the N3, N400, and P3. Maps
produced using EEGLab software.
Face Mental Imagery
3 5 1 27 2 6 4
VPP
7 9 11 12 10 8
N3 N3
28
13 15 16 14
29
17 21 27 22 18
30
19 20
Incongruous 25 31 26
N170
Congruous
0 400 800 ms
−1 µV 23 32 24
FIGURE 8 | Mental imagery congruity effects for faces with common average reference. Same as Figure 2A, except ERPs were re-referenced to the
common average and plotted positive up for comparison with some other work.
of posterior fusiform and inferior occipital gyri (BA18 at 25 −90 400–500 ms, in VLPFC (BA47 at 20 30 −30), (ii) 400–700 ms, in
−25 and 35 −90 −20) extending posteriorly to lingual (BA17) fusiform (400–500 ms, BA37 at 55 −55 −25; 500–700 ms, BA20
and anteriorly to fusiform (BA20 at −45 −25 −30; BA 37) and at 55 −40 −30; BA36 at all times; BA19 at 400–500 ms), (iii)
parahippocampal gyri (BA36). Afterward, during the later P3 400–900 ms, in inferior (BA20 and 37; a maximum also from
peak to incongruous objects and the LPC (Figure 5B), various 800–900 ms at BA20 at −60 −55 −20) and middle temporal gyri
ventral stream sources continued, and prefrontal ones occurred (BA20 until 700 ms), (iv) 600–700 ms, in parahippocampal gyrus
initially from 400 to 500 ms: Sources were observed from (i) (BA36).

Object Mental Imagery
3 5 1 27 2 6 4
N3 N3
VPP
7 9 11 12 10 8
N3 N3
28
13 15 16 14
29
17 21 22 18 N170
30
19 20
Incongruous
Congruous 31
25 26
0 400 800 ms
−1 µV 23 32 24
32
FIGURE 9 | Mental imagery congruity effects for objects with common average reference. Same as Figure 3A, except ERPs were re-referenced to the
common average and plotted positive up for comparison with some other work.
Table 1 | F -values for significant effects in lateral (lat) and midline (mid) omnibus ANOVAs with congruity (C) and category type (T) factors at
each time period.
ERP N3 N3, N400, P3 LPC
Time (ms) 200–300 300–400 400–500 500–700 700–900
Source Lat Mid Lat Mid Lat Mid Lat Mid Lat Mid
Type 14.52** 5.78* 13.03** 8.84** 13.71** 13.26** 20.26** 27.61** 32.67** 32.8**
T×E 23.57** 20.59** 11.91** 11.6** 9.23** 7.78* 7.62** 6.19* 7.5** 8.24*
T×H – 43.05** – 20.17** 28.32** 13.15** 49.96** 5.49* 9.53** 8.04**
T×E×H 2.63* – – – – 6.04* 2.53* 12.56** 2.57* 10.1**
Congruity 12.99** 11.98** 28.72** 26.52** – – – – – 5.94*
C×E 18.29** 24.73** 13.52** 14.7** 3.62* – 9.38** 17.72** – –
C×H – 20.38** – 6.83** – – – 8.84** – –
C×E×H – – – 10.4** – – – – – –
T×C – 11.62** 8.05* 16.65** 23.51** 23.72** 8.18* 14.38** – –
T×C×E 10.16** – 9.66** – 6.05** – 6.89** – – –
T×C×E×H – 15.93** – 19.21** – 7.72** – 10.93** – –
**p < 0.01, *p ≤ 0.05, –p > 0.05. E, electrode; H, hemisphere for lateral or anterior-posterior site for midline.
Category type (object–face) (maximum 200–300 ms: BA20 at 60 −35 −20; maximum 400–
As expected for domain-specificity (Downing et al., 2006), objects 500 ms: BA37 at 60 −55 −10) and (ii) middle temporal gyri
and faces differed primarily in object and face-sensitive areas of (maximum 300–400 for BA20 at 60 −45 −20), (iii) in fusiform
the ventral visual pathway (Figure 6B shows only 200–400 ms as gyrus at most times (BA36, 37: 200–500 ms), and (iv) in middle
later sources remained similar). Specifically, incongruous stimuli occipital gyrus from 300 to 500 ms (BA37; BA19). Likewise, con-
showed sources (i) continuously until 900 ms in inferior temporal gruous stimuli also showed category differences in these regions:

Table 2 | F -values for significant effects of congruity (C) and category For faces (Figure 10), adaptation continued until 900 ms [i.e.,
type (T) at lateral electrode pairs and time periods in focal ANOVAs more negative anteriorly and more positive at occipitotempo-
(upper) and corresponding planned simple effects tests of congruity ral sites for congruous (adapted) than incongruous], as shown
for each category type (lower). by significant effects of congruity [700–899 ms F (1, 17) = 8.47,
p = 0.01], congruity by electrode [500–699 ms F (12, 204) = 8.91,
Start time (ms) 200 300 400 500 700 p = 0.001; 700–899 ms F = 15.88, p < 0.001], congruity by hemi-
sphere [500–699 ms F (1, 17) = 15.63, p = 0.001; 700–899 ms
FRONTOPOLAR (PAIR 1–2) FOCAL ANOVA
F = 11.72, p = 0.003], and congruity by electrode by hemisphere
T 45.13** 15.22** 13.29** 5.80* 5.76*
[700–899 ms F (12, 204) = 2.37, p = 0.048]. Objects largely showed
C 11.80** 16.96** – – –
the opposite (Figure 11), resembling instead ERPs during rapid
T×C 5.20* – – – –
masked repetition priming that is associated with faster response
Congruity effect for each category
times when the prime is more similar, relative to different from,
Objects 17.38** 12.28** – – –
the target (i.e., a positive priming benefit; Forster and Davis, 1984;
Faces – 5.34* – – –
Eddy et al., 2006). Such masked priming is associated with mod-
FRONTOCENTRAL (PAIR 11–12) FOCAL ANOVA
ulations of the VPP (labeled N190 in such work), frontal N3, and
T 17.37** 13.84** 10.56** 18.08** 29.59**
centroparietal N400, which are more negative for incongruous
C 23.87** 26.35** – 9.80** 7.41*
than congruous objects; an occipitotemporal N170 (labeled P190)
T×C 5.31* 13.60** 21.49** 12.26** –
and occipitotemporal N3 counterpart show the opposite (Eddy
et al., 2006; Eddy and Holcomb, 2010). However, the present results
Objects 15.47** 7.94* – – –
would suggest that such positive priming benefits any category
Faces 19.96** 31.84** 9.13** 15.25** 9.17**
within the domain of non-face objects (i.e., between different
OCCIPITOTEMPORAL (PAIR 17–18) FOCAL ANOVA
non-face, basic, object categories) relative to the cross domain
T 9.10** 9.37** 10.79** – –
case (i.e., between faces and non-face objects), whereas all ERP
T×H – – 7.36* 10.13** 4.85*
masked object priming studies to date compared identical object
C – – 4.97* – –
pictures to unrelated non-face objects (Eddy et al., 2006, 2007;
Eddy and Holcomb, 2009, 2010, 2011). To capture this similar-
Objects 5.22* – 7.05* 7.76* –
ity between the present object perception results and the prior
CENTROPARIETAL (PAIR 19–20) FOCAL ANOVA
masked object priming findings, the times and sites of masked
T – – – 8.19* 7.33*
priming effects were analyzed (Eddy et al., 2006; Eddy and Hol-
T×H – – 23.27** 33.56** 30.06**
comb, 2010). Results confirmed the similarity of the present results
C – 10.42** 5.34* 6.32* –
to those in studies of masked priming: From 100 to 250 ms at
T×C 12.27** 5.91* 9.51** – –
frontocentral pair 11-12 for the VPP (i.e., N190), congruity was
T×C×H – 4.63* – – –
significant [F (1, 17) = 29.34, p < 0.001] and marginal at pair 23-24
for the occipitotemporal N170 (i.e., P190; F = 3.44, p = 0.081), sig-
Objects 9.36** – 23.50** 13.82** –
nificant from 250 to 350 ms for the N3 at frontocentral pair 11-12
Faces – 12.30** – – –
(F = 21.84, p < 0.001), and 350–500 ms for the N400 at centropari-
**p < 0.01. *p ≤ 0.05. –p > 0.05. H, hemisphere. Results for 100 ms time periods etal pair 19-20 (F = 23.49, p < 0.001), but the centroparietal effect
from 200 to 500 ms; results for 200 ms time periods after 500 ms. appeared to reflect only the overlapping LPC, suggesting minimal
N400 modulation if any.
Sources were observed (i) in middle temporal gyrus (200–300 ms For completeness, the same focal spatiotemporal analyses com-
maximum for BA37 at 50 −40 −15; BA20), (ii) from 200 to 300 ms pared perceptual congruity between types, as done for imagery,
in fusiform gyrus (BA36/37), (iii) at most times in inferior tempo- and a 140–180 ms time window was added to assess the VPP/N170,
ral gyrus (BA37 at −60 −65 −10; BA20: 300–500 ms), (iv) from as had been done previously for faces (Ganis and Schendan,
300 to 400 ms in middle occipital gyrus (BA19/37). In addition, 2008). Results confirmed that perceptual congruity effects differed
congruous stimuli showed sources of category differences in (v) between categories during the VPP/N170, frontal N3, centropari-
STG from 400 to 500 ms (BA22 at 70 −25 5; BA41/42), consistent etal N400, and posterior LPC. Specifically, at all times after 200 ms
with superior temporal face-specific processes (Puce and Perrett, [all dfs (1, 17)], frontopolar ERPs showed effects of type (type
2003). at 500–899 ms F s > 5.72, ps < 0.03; type by hemisphere at 200–
899 ms F s > 11, ps < 0.005) and congruity (200–899 ms: congruity
PERCEPTION CONTROL ERPs F s > 43, ps < 0.001; congruity by hemisphere F s > 10, ps < 0.006),
Results have already been reported for face ERPs before 500 ms and their interaction (type by congruity by hemisphere at 200–
and comparisons of the early VPP/N170 between perception and 899 ms F s > 4.99, ps < 0.04). Frontocentral ERPs showed effects
mental imagery of faces, demonstrating typical perceptual adap- of type during the VPP and after 500 ms (140–180 and 500–
tation reduction of the VPP/N170 for repeated faces, as well 899 ms F s > 7.96, ps < 0.02), effects of congruity at all times
as adaptation of later ERPs until 500 ms (Ganis and Schen- (congruity at 140–180 and 200–899 ms F s > 6.2, ps < 0.03; con-
dan, 2008). Here, we focus on results for faces after 500 ms, gruity by hemisphere at 200–299 ms F s > 4.91, ps < 0.042), and
results for objects, and comparisons between faces and objects. their interaction during the VPP and N3 (type by congruity by

Face Perception
3 5 1 27 2 6 4
N3
7 9 11 12 10 8
28
13 15 16 14
VPP
29
Incongruous
Congruous
N170
17 19 21 30 22 20 18
LPC
23 25 31 26 24
2 µV
32
0 400 800 ms
FIGURE 10 | Perception congruity effects for faces. Grand average ERPs between −100 and 900 ms at all 32 channels shown filtered low-pass 30 Hz. LPC,
late positive complex. For congruous, pictures of two different people’s faces were presented sequentially, and for incongruous, a picture of a non-face object
preceded a picture of a face.
hemisphere at 140–180 and 200–299 ms F s > 6.31, ps < 0.01). than before (Ganis and Schendan, 2008). Results of an ANOVA
Occipitotemporal ERPs showed effects of type on the N170 with experiment (imagery, perception), congruity, and category
and LPC (type at 140–180 and 400–499 ms F s > 4.65, ps < 0.05; factors demonstrated that timing differences cannot explain cat-
type by hemisphere 200–299 ms F s > 12, ps < 0.003), and only egory and congruity effects. Mental imagery was slower than
the N170 showed effects of congruity (140–180 ms, F = 15.81, perception [F (1, 17) = 267.30, p < 0.001], as subjects took a long
p = 0.001) and the interaction (type by congruity at 140–180 ms, time, 3,889 ms, from the onset of the word cue to report gen-
F = 16.80, p = 0.001). Centroparietal sites showed effects of type eration of the mental image, and only 1,070 ms from the onset
(type by hemisphere at 200–499 ms, F s > 6.02, ps < 0.03), con- of the first picture during perception to report identification of
gruity (400–499 ms F s > 6.77, ps < 0.002), and interactions of the face (categorization of the object); note, with the 200 ms ISI
type by congruity (200–499 ms F s > 7.21, ps < 0.02), but reflected included, these times corresponded to an average, stimulus onset
N400 adaptation for faces and instead overlapping LPC prim- asynchrony (SOA) of 4,089 ms for imagery (SD = 792 ms; range
ing for objects. After 300 ms, the LPC at posterior pair 25-26 2,548–5,252 ms) and 1,270 ms for perception (SD = 346 ms; range
showed effects of type (300–499 ms F s > 8.31, ps < 0.02) and 754–2,113 ms). Critically, these SOA times were indistinguishable
the interactions (type by congruity at 500–899 ms, F s > 4.94, between category and congruity conditions (F s < 1, ps > 0.8), and,
ps < 0.05; type by congruity by hemisphere at 300–399, F s > 6.56, no interaction was significant (F s < 2.1, ps > 0.17). Specifically,
ps < 0.03). for mental imagery, SOAs were similar between categories and
between congruous (faces 4,071 ms; objects 4,107 ms) and incon-
PERFORMANCE gruous conditions (faces 4,081 ms; objects 4,096 ms). Likewise, for
As stimulus timing is critical to interpreting the results, time to perception, SOAs were similar between categories and between
report mental image generation (mental imagery experiment) or congruous (faces 1,311 ms; objects 1,224 ms) and incongruous
identification (perception experiment) is reported in more detail conditions (faces 1,247 ms; objects 1,298 ms).

Object Perception
3 5 1 27 2 N3 6 4
N3
7 9 11 12 10 8
28
13 15 16 14
VPP
29
Incongruous
Congruous
N170
17 19 21 30 22 20 18
LPC
23 25 31 26 24
2 µV
32
0 400 800 ms
FIGURE 11 | Perception congruity effects for objects. Grand average ERPs between −100 and 900 ms at all 32 channels shown filtered low-pass 30 Hz. LPC,
late positive complex. For congruous, pictures of two different objects were presented sequentially, and for incongruous, a picture of a face preceded a picture
of an object.
DISCUSSION shows the opposite congruity effect from the N3 and N400, being
The findings reveal the cortical dynamics of ongoing top-down instead more positive for incongruous than congruous stimuli
processes of mental imagery of visual shape during later knowl- until 700 ms. This pattern of effects resembles a combination of
edge, meaning, and decision processing of a perceived picture. ERP effects of semantic congruity (on N3, N400, and LPC) and
Overall, ongoing mental imagery facilitates categorical percep- immediate repetition priming (on N3, N400, and P3). In con-
tion during the early VPP/N170, as reported previously (Ganis trast, perception shows a different pattern of congruity effects,
and Schendan, 2008), and higher cognitive processes during later as predicted due to the bottom-up contributions to perception
ERPs, as reported here. Specifically, the frontal N3(00) complex, but not imagery. Further, the pattern differs between categories:
which indexes object and face cognition, knowledge, and category Faces show adaptation; objects show rapid priming effects in
decisions from 200 to 400 ms and the centroparietal linguistic the opposite direction. Altogether, these findings indicate that
N400 index of semantic memory from 300 to 500 ms are more top-down processes of mental imagery can induce a powerful
negative for incongruous than congruous stimuli. Notably, N3 imagistic mental representation of visual shape that mimics top-
and N400 effects dissociate from each other not only temporally down processes recruited also for picture perception and facilitates
(i.e., earlier for N3) and spatially (i.e., more frontal for N3) but knowledge, meaning, and categorization processes.
also in how congruity effects differ between categories. N3 effects
are frontopolar for objects and frontocentral for faces and asso- MENTAL IMAGERY
ciated with different brain sources between categories. While the Next we consider the evidence that these mental imagery effects are
N400 congruity effect is small but clear for faces, it is smaller, related to semantic congruity and immediate repetition priming
and, indeed, minimal and not clearly evident for objects. In addi- processes and the implications of this for how mental simulation
tion, the parietal P3 peaks earlier for congruous (∼400 ms) than can ground cognition in shape processing. First, it is important to
incongruous (∼500 ms) stimuli. After 400 ms, the posterior LPC be reminded of key, unique aspects of the present methods (Ganis

and Schendan, 2008). (a) People had extensive training generat- A second reason these findings are novel is that the procedure
ing mental images of each picture of a real person or object, a here of having a word (the name) precede a picture is used in
standard procedure used in validated mental imagery tasks. (b) cross modal priming studies. Critically, for word-picture priming,
The name (i.e., written word) for the person or object cued sub- behavioral evidence has been mixed and, if such cross modal prim-
jects to visualize mentally each associated trained picture. (c) A ing is found, it occurs mainly with much shorter delays than used
picture probed ongoing mental imagery of faces and objects, and here. Such studies typically use an SOA of about 1 s (Bajo, 1988)
ERPs were recorded to this picture, which either was the picture and often much less (Carr et al., 1982) but also often mask the
subjects were cued to visualize mentally or another picture from prime. At such short SOAs, word-picture priming can be compa-
the opposite category. Further, two categories (faces and objects) rable (Bajo, 1988) or much less than within modality (e.g., picture
that are supported by different visual processing areas (Hasson to picture; Carr et al., 1982). At slightly longer SOAs of less than
et al., 2003; Downing et al., 2006; Tsao et al., 2006) were used to about 2 s, priming can be absent (Biggs and Marmurek, 1990).
manipulate congruity, and the opposite category was assigned to Like behavioral effects, ERP effects of cross modal priming (on
the incongruous condition. Consequently, congruity effects reflect the N3, N400, and P3/LPC) have been found most consistently in
large differences in shape processing. This is because congruent studies using SOAs briefer than 500 ms or unmasked primes com-
face imagery (identical face) maximally affects face processes, while posed of multiple words in sentences or noun phrases (Ganis et al.,
incongruent imagery (object) minimally affects face processes, 1996; Federmeier and Kutas, 2001, 2002; Stanfield and Zwaan,
and analogously for object imagery. (d) The delay between the 2001; Zwaan et al., 2002; Hirschfeld et al., 2012). Cross modal
name cue and target picture was relatively long (SOA 4,089 ms, priming from a single word to a picture is more variable but has
on average). (e) No task was performed on the target picture to been found at SOAs ranging from 120 to 700 ms on the N400
minimize decision and response related ERPs that can complicate (and perhaps P3/LPC) between 350 and 550 ms when the prime
interpretation of the waveform, thereby defining knowledge and is unmasked (Auchterlonie et al., 2002; Johnson and Olshausen,
semantic memory processes as clearly as possible. While a limita- 2003, 2005; Dobel et al., 2010; Kiefer et al., 2011) and masked
tion of this study is that the target picture was not followed by a (Blackford et al., 2012). In contrast, N3 cross modal priming has
task that assessed the mental imagery, evidence that subjects did been found only in studies that (a) mask the word prime, use
as instructed is that they took much longer (2,819 ms) to visualize the shortest SOAs (120 ms or less), and overt naming, or (b) use
the named picture in the mental imagery experiment than to iden- long SOAs of about 1–2 s and name verification (Johnson and
tify the perceived face (object) picture in the perception control Olshausen, 2003, 2005). Notably, for priming from a word to a
experiment. picture, visually impoverishing the objects (by occlusion or frag-
mentation) yields a more frontopolar distribution of congruity
Mental imagery produces semantic congruity effects and facilitates effects. This scalp distribution is consistent with the frontopolar
cross modal priming N3 in studies of object cognition and priming with non-canonical
The most important finding here is that mental imagery produces views, fragmented real objects, and pseudo objects (Holcomb and
ERP effects that resemble N3, N400, and LPC effects observed in McPherson, 1994; Schendan et al., 1998; McPherson and Hol-
studies of short latency, cross modal priming, semantic congruity, comb, 1999; Schendan and Kutas, 2002, 2003, 2007; Schendan and
and semantic priming phenomena. No prior behavioral or ERP Lucia, 2009; Schendan and Maher, 2009), which recruit top-down
study on these phenomena or mental imagery would have pre- processes more (Michelon et al., 2003; Ganis et al., 2007), and
dicted this finding, as mentioned in the introduction. One reason as found here for mental imagery of objects. In sum, behavioral
is that the timing for mental imagery here is well beyond that for and ERP (N3, N400, P3/LPC) effects of cross modal priming can
automatic spreading activation associated with semantic priming, occur at shorter SOAs, with multiple words as the prime, and dur-
which is thought to underlie semantic congruity effects and to ing naming tasks, but, crucially, none of these procedures apply to
reveal semantic memory processes (Kutas and Federmeier, 2011). the mental imagery task used here.
Consider that, in the typical semantic priming task, two related In contrast, for long SOAs well beyond about 1 s and more like
words are presented sequentially with a brief delay (usually under mental imagery here, behavioral cross modal priming from a sin-
1 s): A target word (e.g., “doctor”) is preceded by a prime word gle word to a picture has not been found (Morton, 1979; Warren
that is different and either semantically related (“nurse”) or unre- and Morton, 1982) or is much smaller than that within modality
lated (“truck”) to the target. Response times are faster to targets (Carr et al., 1982). The ineffectiveness of word primes for picture
preceded by primes that are semantically related (congruous) than targets at long delays, however, can be overcome, by (a) vary-
unrelated (incongruous). Findings with SOAs between prime and ing prime modality only between- (i.e., not within-) subjects, (b)
target of under 500 ms reflect automatic spreading activation in blocking prime modality (Brown et al., 1991), or (c), critically here,
the semantic network, whereas SOAs between 500 and 1000 ms or instructing subjects to use mental imagery. A word that is used to
so reflect controlled semantic analysis (Rossell et al., 2003; O’Hare cue mental imagery during a study session primes later perfor-
et al., 2008). Thus for the timing in our study (for both imagery mance with the picture at a long delay (minutes) on an implicit
and perception), the results can reflect only controlled seman- memory test with objects (McDermott and Roediger, 1994) or
tic processing. In contrast, most semantic priming work studied famous faces (Cabeza et al., 1997) and can do so as well or better
automatic spreading activation using short SOAs under 500 ms than a perceived picture (Michelon and Koenig, 2002; Michelon
and so evidence of behavioral and ERP priming with longer delays and Zacks, 2003). However, it is important to note that these prim-
is scarce and more so for the much longer delays here. ing studies do not use the picture target at test to reveal ongoing

mental imagery sustained within working memory, as herein, but memory in both. Immediate repetition priming is due to working
rather its long-term consequences for a memory test much later memory for the first image that is sustained across the brief delay
(i.e., beyond the time limits for working memory). Notably, prior until the second image appears (Bentin and McCarthy, 1994).
ERP studies, in which a single word could prime a picture target at This working memory facilitates categorization of the repeated
a long delay, have not used blocking or mental imagery procedures, percept with minimal or no reactivation of semantic memory,
and, accordingly, no behavioral or ERP priming effects were found. minimizing the N400. Likewise, the priming effect on the frontal
Although episodic recognition does show effects (Kazmerski and N3 could indicate that visual knowledge and cognitive decision
Friedman, 1997; Spironelli et al., 2011), these do not apply here processes for objects and faces are also largely bypassed. Immediate
because we assess ongoing mental imagery, not the consequences repetition effects on the P3 reflect modality-specific (i.e., visual)
for later episodic recollection. Further, even if N400 effects are working memory that speeds the category decision (Bentin and
found at such long lags, they likely reflect morphological (linguis- McCarthy, 1994; Nielsen-Bohlman and Knight, 1994; Zhang et al.,
tic) representations, not semantic or phonological representations 1995). Consistent with maintaining modal visual information in
which do not seem to survive lags beyond SOAs of 300 ms (Feld- working memory, P3 facilitation is not associated with semantic
man, 2000; Koester and Schiller, 2008). This would suggest that priming, as reviewed above. Further, immediate repetition effects
N400 effects for mental imagery here reflect linguistic, not seman- on the P3 are likely also due to having subjects perform a task
tic, memory representations (Kousta et al., 2011), but future work on the pictures, which was often episodic recognition, as such
needs to assess this. task requirements maximize P3 and other late posterior positivi-
ties (Dien et al., 2004). Mental imagery had no task requirements
Mental imagery primes perception like immediate picture repetition for the target picture, but the practice session required subjects
does to assess how well their mental image matched the picture; thus
Altogether, these direct neurophysiological findings are consistent subjects likely continued to do so incidentally during the men-
with behavioral evidence that mental imagery facilitates object tal imagery test, resulting in P3 facilitation despite no overt task.
categorization via priming mechanisms (Peterson and Graham, Thus, mental imagery can simulate the top-down cortical dynam-
1974). Most striking is the finding that the N3, N400, and P3 con- ics that are produced by an actual perceived picture, and the
gruity effects mimic ERP immediate repetition priming, providing episodic memories encoded during training and practice contain
direct neurophysiological evidence that mental imagery can affect visual details sufficient to enable mental imagery representations
neural processing like actual perception of a picture can. When to operate like an actual perceived picture (as in immediate repeti-
the exact same image repeats immediately with no intervening tion priming). This provides strong and direct neurophysiological
stimuli, ERPs after 300 ms become earlier and larger for objects support for the pictorial theory of mental imagery and implicates
(Nielsen-Bohlman and Knight, 1994; Zhang et al., 1995), faces, these strong pictorial representations in episodic memory of per-
and words (Bentin and McCarthy, 1994; Schendan et al., 1997). sonally experienced, autobiographical information that depends
The P3 is more positive and peaks earlier (∼400 ms) and the fol- upon the mediotemporal lobe (Ganis and Schendan, 2011). This
lowing N400 is more positive for repeated (akin to congruent) finding also constitutes evidence for the visual detail achievable
than unrepeated (akin to incongruent) faces and objects, and a by the episodic memory system. Such evidence will be crucial for
later P3 or LPC, peaking around 500 ms, is instead more posi- developing theories of mental simulation for episodic memory
tive for unrepeated than repeated pictures. These ERP effects of (Schacter et al., 2008).
perceptual immediate repetition priming have been observed at
relatively short SOAs of 1200–3500 ms, which is much longer than Reflexive top-down processes for mental imagery support
the 500 ms SOA necessary to observe automatic spreading activa- automatic mental simulation
tion in semantic priming and slightly longer than the about 1 s SOA By using faces and objects, which have partially non-overlapping
for cross modal priming (without special conditions like mental visual processing pathways, these mental imagery findings define
imagery). For objects, most studies could not or did not assess the largest possible set of top-down mechanisms that support
frontal ERPs, but one study also shows the frontal N3 is more pos- mental simulation of face (object) shape, including non-conscious
itive for repeated than unrepeated items (SOA 2400 ms; Henson automatic simulation. Mental simulation has been proposed to
et al., 2004). This study also reported VPP/N170 repetition adap- operate via a pattern completion process that re-enacts modal pro-
tation for objects (i.e., smaller for repeated), but this adaptation cessing that had occurred during learning when later retrieving the
direction is the opposite of the later repetition priming effects in memory (Barsalou, 2009). We proposed that, at the level of brain
that study and of the mental imagery effects here, and no other mechanisms, the top-down feedback mechanisms that support
immediate repetition study found effects before 200 ms (Nielsen- automatic simulation are a subset of those that support mental
Bohlman and Knight, 1994; Zhang et al., 1995). Altogether, these imagery (Ganis and Schendan, 2011). Specifically, automatic sim-
findings indicate that mental imagery mimics the pattern of imme- ulations unfold via reflexive top-down signals from higher to lower
diate repetition priming of perceived pictures on the N3, N400, P3, level areas along modal information processing pathways, such as
and LPC. However, mental imagery also enhances the VPP/N170, the ventral stream: Perceiving a stimulus triggers these processes
unlike immediate repetition priming, which typically has little or reflexively (Ganis and Kosslyn, 2007). Through such distributed
no effect on early ERPs. multi-regional activity, seeing an object or reading its name (e.g.,
The similarity between mental imagery effects and ERP imme- “dog”), for example, re-enacts associated modal features that were
diate repetition priming is consistent with the role of working stored during earlier learning experiences (e.g., its shape, color,

motion, actions with it), thereby constructing cognition, mem- effects on the VPP/N170 extend to later ERPs. The direction of
ory, and meaning. This is consistent with the MUSI account that these effects suggests their interpretation. Consider that all these
proposes top-down feedback processes after 200 ms have the great- adaptation effects go in the opposite direction compared to the
est role in visual cognition, constructing knowledge, meaning, facilitation effects found for mental imagery: Adaptation reduces
memory, and decisions (Schendan and Kutas, 2007; Schendan and the VPP/N170 but enhances later ERPs. Such enhancements of
Maher, 2009; Schendan and Lucia, 2010). These same processes are later ERPs are thought to reflect greater (not less) recruitment
triggered by strategic top-down signals from the prefrontal cortex of the underlying processes (Schendan and Kutas, 2003). Hence,
during mental imagery (Ganis and Kosslyn, 2007) and so mental early adaptation of categorical perception during the VPP/N170
imagery time courses like those here can define when and how impedes later cognitive processing (i.e., due to failure of an ear-
mental simulation grounds cognition. Previously, semantic prim- lier critical process), thereby recruiting additional top-down pro-
ing has revealed the most about automatic mental simulation and cessing resources to accomplish these later cognitive functions
its brain basis, especially with words (e.g., Marinkovic et al., 2003; (Kosslyn et al., 1994; Ganis et al., 2007).
Rossell et al., 2003; Kutas and Federmeier, 2011). This is because The direction of the object perception findings tells a different
automatic spreading activation across a semantic memory net- story, which further supports a facilitation interpretation of the
work, which explains such priming, is thought to operate via the mental imagery findings because the direction of the effect is the
same automatic and reflexive, top-down processes that have been same as for imagery. Specifically, under rapid, immediate serial
implicated in automatic simulation (e.g., Collins and Loftus, 1975; presentation, the perception of an object picture primes (facili-
Franklin et al., 2007; Kutas and Federmeier, 2011). The similarity tates) a subset of the ERPs to a target object picture that mental
between the results here using a validated mental imagery task imagery also primes. For congruous relative to incongruous con-
and ERP findings related to semantic congruity and immediate ditions, the VPP is more positive (and N170 more negative), the
repetition priming supports this conclusion. N3 less negative, and LPC less positive, whereas the N400 shows
minimal or no priming. Unlike imagery, though, perceptual rep-
PERCEIVED PICTURE IDENTIFICATION ADAPTS FACES BUT PRIMES etition shows no P3 modulation, but this is presumably due to
OBJECTS no overt or implied task on the perceived target picture. Surpris-
Overall, perception control results confirm that common top- ingly, therefore, priming of whatever processes are shared among
down processes underlie similarities between imagery and percep- a set of real objects from different basic categories can facilitate
tion, while bottom-up processes for perception (but not imagery) processing of each other (i.e., congruous perception), in con-
underlie their differences (Ganis and Schendan, 2008). The time trast to the cross domain case of perceiving a face and then a
precision of ERPs offers advantages over fMRI and behavior non-face object (i.e., incongruous perception). The resemblance
for characterizing such similarities and differences between per- between these perception (and the mental imagery) results and
ception and imagery. Specifically, perception results dissociate those for certain kinds of rapid perceptual and semantic prim-
between categories, consistent with the domain-specificity of ing supports a facilitation interpretation. After all, the N3, N400,
object and face processing (Downing et al., 2006): Perceptual and LPC effects of object perception congruity resemble a sub-
repetition adapts processing of perceived faces from categorical set of effects for mental imagery and immediate repetition and
perception onward, as predicted, but instead unexpectedly primes semantic priming. Moreover, the waveforms resemble those asso-
processing of objects during categorical perception, visual knowl- ciated with priming under the most rapid, serial presentation of
edge processing, and strategic semantic analysis. Critically, iden- pictures (SOA < 500 ms) when the prime is either not masked
tification time for the first picture is similar for objects and faces and semantically related pictures repeat (Holcomb and McPher-
and so cannot explain differences in congruity effects. In fact, the son, 1994; McPherson and Holcomb, 1999; Kiefer et al., 2011) or
timing was chosen to replicate classic face adaptation effects on the masked and identical pictures repeat (Eddy and Holcomb, 2010).
VPP/N170(Jeffreys, 1996) obtained with an 1,100 ms SOA, 800 ms In masked repetition priming, the VPP (labeled N190 in these
duration, and 300 ms ISI (i.e., like the 1,070 ms identification RT studies), N3, and N400 are more negative [and occipitotempo-
and 200 ms ISI here), as was achieved (Ganis and Schendan, 2008). ral N170 (labeled P190) and N3 counterpart are more positive]
To understand the perception control findings, it is necessary to for unrelated than repeated (identical) pictures of objects (Eddy
highlight that both prime and target were always different pic- et al., 2006; Eddy and Holcomb, 2010). The waveform similarity
tures, even in the congruous condition. Thus, perception results between object perception here and masked priming must be due
show how perceiving a picture of a face (object) is affected by to the very short 200 ms ISI used here for perception, causing the
having previously identified a perceived picture of a different face ERPs to the prime and target to overlap temporally, as they do
(object), compared to having previously identified the opposite in the masked priming work, which uses short ISIs of 100 ms or
category picture [i.e., of an object (face)]. less. The short ISI is probably also responsible for some of the
For faces, seeing two different people sequentially adapts the effects resembling effects for rapid repetition and masked seman-
ERPs, thereby producing the opposite congruity effect from that tic priming (with ISIs of 200 ms or less) more than for longer lag,
for mental imagery. For congruous relative to incongruous percep- immediate repetition priming (with SOAs of 1200–3500 ms). In
tion, the VPP is less positive (and N170 less negative), as reported turn, the longer SOA here may explain why the later perceptual
previously (Ganis and Schendan, 2008), the N3 and N400 are congruity effects also resemble some longer lag, immediate rep-
more negative, and the LPC is more positive. Altogether this find- etition priming effects on the N3 and LPC at SOAs of about 1
ing and the mental imagery finding indicate that early congruity to 1.5 s, which show the same pattern of congruity effects on the

N3, N400, and a posterior P3/LPC (Barrett and Rugg, 1990; Hol- (Grill-Spector et al., 2006), and top-down processes from the
comb and McPherson, 1994; McPherson and Holcomb, 1999; Bach frontal lobe, which support mental imagery, have been implicated
et al., 2009). Thus, for objects (but not faces) perception-driven in perceptual repetition priming (Schacter et al., 2007). Consider
priming and mental imagery congruity effects differ quantitatively that different faces share more lower-level features, such as spa-
and qualitatively but nonetheless all follow a direction that indi- tial frequency spectra, and so have less interstimulus variance
cates priming facilitation. The similarity between the perception than different objects (Costen et al., 1996; Thierry et al., 2007;
and imagery ERP congruity effects with objects further bolsters Ganis et al., 2012). Hence, different faces can mutually adapt
the idea that mental imagery mimics perception: Mental imagery not only face processing but also lower-level feature processing
of objects, primes (facilitates) object processing like repeating a more than different objects. For example, the N170 face-specificity
perceived picture. effect overlaps spatiotemporally with an N1(00) component (i.e.,
N170 face-specificity modulates the N1), which reflects wide-
CAVEATS spread bottom-up and reflexive feedback processing along the
Future work will need to investigate why, despite identical tim- visual pathways from lower to higher level areas (e.g., Vogel and
ing, perception of faces and objects produce opposite congruity Luck, 2000; Bullier, 2001). Adaptation of both the face-specific and
(repetition) effects on the ERPs to the probe picture. N170 rapid the low-level visual processes results in a large decrease, as both
adaptation evidence (Nemrodov and Itier, 2012) suggests that N170 face and N1 visual components are affected. In contrast,
faces are stronger adaptors than some non-face categories. Face mental imagery of face-specific processes increases N170 ampli-
(and car) primes reduce (i.e., adapt) the N170 more than chair tude, but this is a smaller change (than the decrease for adaptation)
and house primes for all categories of test objects (i.e., faces, cars, because only the N170 face component is affected. This explana-
chairs, houses). Prime category and ISIs are the key factors deter- tion needs to be considered to resolve whether perception and
mining how much, if at all, the prime adapts the test stimulus; after mental imagery share properties (Freyd and Finke, 1984) or not
all, the ISIs of 232–268 ms and test stimulus duration of 200 ms (Craver-Lemley and Reeves, 1992).
resemble the present timing, but the faces were of unknown peo- Finally, given the novelty of this mental imagery probe par-
ple, whereas here they were famous, and the prime was much adigm (Ganis and Schendan, 2008), several issues remain to
briefer (i.e., 200 ms) than here, suggesting neither knowledge nor be resolved. For example, future work will need to evaluate
prime duration can explain the findings. The reason that the cate- how individual differences in mental imagery and other abili-
gory of the prime matters is unclear but generally consistent with ties (Kozhevnikov et al., 2005) affect mental imagery processes,
transfer appropriate processing and encoding specificity accounts as well as how congruity effects differ as a function of the vivid-
of memory (Tulving and Thomson, 1973; Morris et al., 1977). ness of the perceived stimulus and subjective vividness of mental
Other proposals include interference from ongoing late process- imagery (Herholz et al., 2012). Also, mental imagery will need to be
ing of the adaptor due to the short ISI and neural fatigue that compared directly with cross modal priming from a word to a pic-
is not category specific but selectively tuned to adaptor proper- ture and potential processing differences (e.g., depth of semantic
ties (Nemrodov and Itier, 2012). Regardless, faces can adapt more network activation) will need to be addressed.
than other object categories. The present results suggest an addi-
tional twist: Non-face objects can prime better than faces. One CONCLUSION
may speculate that, here (and in related work), face perception The ERP results described in this report define the neurophysio-
shows a substantial adaptation pattern because faces (congruent) logical characteristics and time course of top-down processes for
adapt better than other objects (incongruent), while objects prime mental imagery of the visual shape of faces and objects that can
better, and object perception shows a substantial priming pattern ground cognition in these modal processes. These findings provide
because objects (congruent) prime better than faces, while faces striking direct neural evidence that top-down feedback processes
(incongruent) adapt better; both adaptation and priming influ- of mental imagery sustain an imagistic representation that mim-
ences could affect any result, making congruity effects larger than ics perception well enough to prime subsequent perception and
either influence alone. The relative strengths of adaptation and cognition like an actual picture. By manipulating congruity by
priming explain the pattern observed, which will be important for switching between face and object categories, which involve dif-
future research to tease apart. ferent modal processes along the ventral visual stream, the ERPs
This caveat also speaks to an alternative hypothesis to explain reveal the largest set of top-down processes for mental imagery
why both perception and mental imagery prime, but perception of these shape categories. The subset of these mental imagery
also adapts a stimulus: Perception adapts not only face (object) processes that correspond to the reflexive top-down inputs from
processing but also adapts visual sensory processing, and the lat- higher to lower level areas along the posterior ventral face (object)
ter results in reduced sensitivity in lower-level areas, including processing pathway also constitute the automatic mental simula-
occipital cortex (Wilson and Humanski, 1993; Anderson and Wil- tion processes that can ground cognition of faces (objects). The
son, 2005; Loffler et al., 2005). This is another way of stating ERP congruity effects here therefore provide direct neurophysi-
that both perception and mental imagery can recruit top-down ological markers for these visual shape mental simulations that
processes, while perception is also driven by bottom-up processes can be used to determine precisely when, how, and how much
(Ganis and Schendan, 2008) but adds the idea that perception also these cortical mental simulation mechanisms ground cognition.
adapts lower-level occipital areas. After all, both perceptual prim- Together, the robust frontal N3 and minimal centroparietal N400
ing and adaptation effects on perception are well-documented mental imagery congruity findings confirm the visual imagistic

(non-linguistic) nature of shape mental imagery, and further In addition, for both imagery and perception, functional dis-
implicate the N3 as an index of visual knowledge (Schendan and sociations and spatial distribution differences between faces and
Maher, 2009) and the N400 as an index of linguistic knowledge. objects provide further evidence for domain-specificity and the
This is consistent with grounded cognition distinctions between modality-specific processing posited for grounded cognition. In
non-linguistic (“experiential” modal sensorimotor and mental contrast to mental imagery, perception of a face (instead of an
state information) and linguistic systems (word-related associ- object) adapts categorical perception, which consequently impairs
ations) for semantic memory (Kousta et al., 2011; Paivio and later processing of a different face, whereas perception of an object
Sadoski, 2011). Thus, future work on the cortical dynamics of (instead of a face) primes categorical perception, activation of
the contribution of mental simulation of visual shape to seman- visual knowledge, and later categorization of a different object. The
tic memory should focus on the frontal N3. Further, finding both differences between mental imagery and perception are consistent
mid-latency (N3, N400) and LPC congruity effects suggests that, with the strategic top-down processes required to construct and
to ground cognition in modal processing, two types of mental maintain mental imagery versus the bottom-up input required for
simulation can operate at two distinct times. Automatic simula- perception, whereas the similarities are consistent with the com-
tions of visual shape (due to reflexive top-down processes) and mon, automatic, top-down, modal processing supporting both,
linguistic processes operate between 200 and 500 ms during the and further support the imagistic nature of mental imagery.
N3 and N400, respectively, and effortful simulation (due to strate-
gic top-down processes) operates between 400 and 700 ms during AUTHOR NOTE
the LPC. Altogether, these markers, and others defined using the Preparation of this manuscript was supported by the University of
methods developed here, can be used to characterize and probe Plymouth and funded by an International Research and Collabo-
these mental simulation processes in future research on grounded ration grant to Haline E. Schendan. The authors are grateful for
cognition theory, especially for discovering the neural mechanisms the assistance of Roderick Elias, B. S., Dr. Stephen M. Maher, and
of how mental simulation works. Lisa C. Lucia, M.Sc. for recording the EEG data.
REFERENCES semantic matching of pictures. Brain representations of familiar faces. Br. modeling of evoked responses in
Allison, T., Puce, A., Spencer, D. D., Cogn. 14, 201–212. J. Psychol. 102, 943–958. EEG and MEG. Neuroimage 30,
and Mccarthy, G. (1999). Electro- Barsalou, L. W. (2008). Grounded Cabeza, R., Burton, A. M., Kelly, S. W., 1255–1272.
physiological studies of human face cognition. Annu. Rev. Psychol. 59, and Akamatsu, S. (1997). Investi- Dien, J., Spencer, K. M., and Donchin,
perception I: potentials generated in 617–645. gating the relation between imagery E. (2004). Parsing the late positive
occipitotemporal cortex by face and Barsalou, L. W. (2009). Simulation, situ- and perception: evidence from face complex: mental chronometry and
non-face stimuli. Cereb. Cortex 9, ated conceptualization, and predic- priming. Q. J. Exp. Psychol. A 50, the ERP components that inhabit
415–430. tion. Philos. Trans. R. Soc. Lond. B 274–289. the neighborhood of the P300. Psy-
Anderson, N. D., and Wilson, H. R. Biol. Sci. 364, 1281–1289. Carr, T. H., Mccauley, C., Sperber, R. D., chophysiology 41, 665–678.
(2005). The nature of synthetic Bentin, S., and McCarthy, G. (1994). and Parmelee, C. M. (1982). Words, Dobel, C., Junghofer, M., Breitenstein,
face adaptation. Vision Res. 45, The effects of immediate stimu- pictures, and priming: on seman- C., Klauke, B., Knecht, S., Pantev,
1815–1828. lus repetition on reaction time and tic activation, conscious identifica- C., and Zwitserlood, P. (2010). New
Anderson, S. E., Chiu, E., Huette, S., and event-related potentials in tasks of tion, and the automaticity of infor- names for known things: on the
Spivey, M. J. (2011). On the tempo- different complexity. J. Exp. Psychol. mation processing. J. Exp. Psychol. association of novel word forms
ral dynamics of language-mediated Learn. Mem. Cogn. 20, 130–149. Hum. Percept. Perform. 8, 757–777. with existing semantic information.
vision and vision-mediated lan- Biggs, T. C., and Marmurek, H. H. Chatterjee, A. (2010). Disembodying J. Cogn. Neurosci. 22, 1251–1261.
guage. Acta Psychol. (Amst.) 137, C. (1990). Picture and word nam- cognition. Lang. Cogn. 2, 79–116. Doniger, G. M., Foxe, J. J., Murray,
181–189. ing – is facilitation due to processing Clarke, A., Taylor, K. I., and Tyler, L. K. M. M., Higgins, B. A., Snodgrass,
Auchterlonie, S., Phillips, N. A., and overlap. Am. J. Psychol. 103, 81–100. (2011). The evolution of meaning: J. G., Schroeder, C. E., and Javitt,
Chertkow, H. (2002). Behavioral and Blackford, T., Holcomb, P. J., Grainger, spatio-temporal dynamics of visual D. C. (2000). Activation timecourse
electrical brain measures of semantic J., and Kuperberg, G. R. (2012). object recognition. J. Cogn. Neurosci. of ventral visual stream object-
priming in patients with Alzheimer’s A funny thing happened on the 23, 1887–1899. recognition areas: high density elec-
disease: implications for access fail- way to articulation: N400 attenua- Collins, A. M., and Loftus, E. F. (1975). trical mapping of perceptual clo-
ure versus deterioration hypotheses. tion despite behavioral interference A spreading activation theory of sure processes. J. Cogn. Neurosci. 12,
Brain Cogn. 48, 264–267. in picture naming. Cognition 123, semantic processing. Psychol. Rev. 615–621.
Bach, P., Gunter, T. C., Knoblich, G., 84–99. 82, 407–428. Doniger, G. M., Foxe, J. J., Schroeder,
Prinz, W., and Friederici, A. D. Brown, A. S., Neblett, D. R., Jones, T. Costen, N. P., Parker, D. M., and Craw, I. C. E., Murray, M. M., Higgins,
(2009). N400-like negativities in C., and Mitchell, D. B. (1991). Trans- (1996). Effects of high-pass and low- B. A., and Javitt, D. C. (2001).
action perception reflect the acti- fer of processing in repetition prim- pass spatial filtering on face iden- Visual perceptual learning in human
vation of two components of an ing: some inappropriate findings. J. tification. Percept. Psychophys. 58, object recognition areas: a repetition
action representation. Soc. Neurosci. Exp. Psychol. Learn. Mem. Cogn. 17, 602–612. priming study using high-density
4, 212–232. 514–525. Craver-Lemley, C., and Reeves, A. electrical mapping. Neuroimage 13,
Bajo, M. T. (1988). Semantic facili- Bub, D. N., and Masson, M. E. (2010). (1992). How visual imagery inter- 305–313.
tation with pictures and words. J. On the nature of hand-action rep- feres with vision. Psychol. Rev. 99, Downing, P. E., Chan, A. W., Peelen, M.
Exp. Psychol. Learn. Mem. Cogn. 14, resentations evoked during written 633–649. V., Dodds, C. M., and Kanwisher, N.
579–589. sentence comprehension. Cognition David, O., Harrison, L., and Friston, (2006). Domain specificity in visual
Barrett, S. E., and Rugg, M. D. (1989). 116, 394–408. K. J. (2005). Modeling event-related cortex. Cereb. Cortex 16, 1453–1461.
Event-related potentials and the Bullier, J. (2001). Integrated model of responses in the brain. Neuroimage Eddy, M. D., and Holcomb, P. J. (2009).
semantic matching of faces. Neu- visual processing. Brain Res. Brain 25, 756–770. Electrophysiological evidence for
ropsychologia 27, 913–922. Res. Rev. 36, 96–107. David, O., Kiebel, S. J., Harrison, L. size invariance in masked picture
Barrett, S. E., and Rugg, M. D. (1990). Burton, A. M., Jenkins, R., and M., Mattout, J., Kilner, J. M., and repetition priming. Brain Cogn. 71,
Event-related potentials and the Schweinberger, S. R. (2011). Mental Friston, K. J. (2006). Dynamic causal 397–409.

Eddy, M. D., and Holcomb, P. J. processing in vision,” in Represen- memory for familiar tunes. J. Cogn. viewpoints. A PET investigation.
(2010). The temporal dynam- tation and Brain, ed. S. Funahashi Neurosci. 24, 1382–1397. Brain 117, 1055–1071.
ics of masked repetition picture (Tokyo: Springer Verlag), 21–45. Hirschfeld, G., Feldker, K., and Zwit- Kosslyn, S. M., Ganis, G., and Thomp-
priming effects: manipulations of Ganis, G., and Kutas, M. (2003). An serlood, P. (2012). Listening to “fly- son, W. L. (2001). Neural founda-
stimulus-onset asynchrony (SOA) electrophysiological study of scene ing ducks”: individual differences in tions of imagery. Nat. Rev. Neurosci.
and prime duration. Brain Res. 1340, effects on object identification. Brain sentence-picture verification inves- 2, 635–642.
24–39. Res. Cogn. Brain Res. 16, 123–144. tigated with ERPs. Psychophysiology Kosslyn, S. M., Thompson, W. L., and
Eddy, M. D., and Holcomb, P. J. (2011). Ganis, G., Kutas, M., and Sereno, M. 49, 312–321. Ganis, G. (2006). The Case for Men-
Invariance to rotation in depth mea- I. (1996). The search for common Hirschfeld, G., and Zwitserlood, P. tal Imagery. New York, NY: Oxford
sured by masked repetition prim- sense: an electrophysiological study (2011). How vision is shaped by lan- University Press.
ing is dependent on prime duration. of the comprehension of words and guage comprehension – top-down Kousta, S. T., Vigliocco, G., Vinson, D.
Brain Res. 1424, 38–52. pictures in reading. J. Cogn. Neu- feedback based on low-spatial fre- P., Andrews, M., and Del Campo,
Eddy, M. D., Schmid, A., and Holcomb, rosci. 8, 89–106. quencies. Brain Res. 1377, 78–83. E. (2011). The representation of
P. J. (2006). Masked repetition prim- Ganis, G., and Schendan, H. E. (2008). Holcomb, P. J. (2003). “ERPSystem,” abstract words: why emotion mat-
ing and event-related brain poten- Visual mental imagery and percep- 2.0a ed. Medford: Tufts University. ters. J. Exp. Psychol. Gen. 140, 14–34.
tials: a new approach for tracking the tion produce opposite adaptation Holcomb, P. J., and McPherson, W. B. Kozhevnikov, M., Kosslyn, S., and Shep-
time-course of object perception. effects on early brain potentials. (1994). Event-related brain poten- hard, J. (2005). Spatial versus object
Psychophysiology 43, 564–568. Neuroimage 42, 1714–1727. tials reflect semantic priming in an visualizers: a new characterization of
Eddy, M. D., Schnyer, D., Schmid, A., Ganis, G., and Schendan, H. E. (2011). object decision task. Brain Cogn. 24, visual cognitive style. Mem. Cognit.
and Holcomb, P. J. (2007). Spa- Mental imagery. Wiley Interdiscip. 259–276. 33, 710–726.
tial dynamics of masked picture Rev. Cogn. Sci. 2, 239–252. Jeffreys, D. A. (1996). Evoked potential Kutas, M., and Federmeier, K. D. (2011).
repetition effects. Neuroimage 34, Ganis, G., Schendan, H. E., and Koss- studies of face and object processing. Thirty years and counting: finding
1723–1732. lyn, S. M. (2007). Neuroimaging Vis. Cogn. 3, 1–38. meaning in the N400 component
Farah, M. J., Peronnet, F., Gonon, M. A., evidence for object model verifica- Johnson, J. S., and Olshausen, B. A. of the event-related brain poten-
and Giard, M. H. (1988). Electro- tion theory: role of prefrontal con- (2003). Time course of neural sig- tial (ERP). Annu. Rev. Psychol. 62,
physiological evidence for a shared trol in visual object categorization. natures of object recognition. J. Vis. 621–647.
representational medium for visual Neuroimage 34, 384–398. 3, 499–512. Lau, E. F., Phillips, C., and Poeppel,
images and visual percepts. J. Exp. Ganis, G., Smith, D., and Schendan, H. Johnson, J. S., and Olshausen, B. A. D. (2008). A cortical network
Psychol. Gen. 117, 248–257. E. (2012). The N170, not the P1, (2005). The earliest EEG signatures for semantics: (de)constructing
Federmeier, K. D., and Kutas, M. (2001). indexes the earliest time for categor- of object recognition in a cued- the N400. Nat. Rev. Neurosci. 9,
Meaning and modality: influences of ical perception of faces, regardless of target task are postsensory. J. Vis. 5, 920–933.
context, semantic memory organi- interstimulus variance. Neuroimage 299–312. Loffler, G., Yourganov, G., Wilkinson, F.,
zation, and perceptual predictability 62, 1563–1574. Johnson-Laird, P. N. (1983). Mental and Wilson, H. R. (2005). fMRI evi-
on picture processing. J. Exp. Psy- Ganis, G., Thompson, W. L., Mast, F. Models. Cambridge, MA: Harvard dence for the neural representation
chol. Learn. Mem. Cogn. 27, 202–224. W., and Kosslyn, S. M. (2003). Visual University Press. of faces. Nat. Neurosci. 8, 1386–1390.
Federmeier, K. D., and Kutas, M. (2002). imagery in cerebral visual dysfunc- Kazmerski, V. A., and Friedman, D. Luck, S. J. (2005). An Introduction to the
Picture the difference: electrophys- tion. Neurol. Clin. 21, 631–646. (1997). Old/new differences in direct Event-Related Potential Technique.
iological investigations of picture Grill-Spector, K., Henson, R., and Mar- and indirect memory tests using Cambridge, MA: MIT Press.
processing in the two cerebral tin, A. (2006). Repetition and the pictures and words in within- and Mahon, B. Z., and Caramazza,A. (2009).
hemispheres. Neuropsychologia 40, brain: neural models of stimulus- cross-form conditions: event-related Concepts and categories: a cogni-
730–747. specific effects. Trends Cogn. Sci. potential and behavioral measures. tive neuropsychological perspective.
Feldman, L. B. (2000). Are morpholog- (Regul. Ed.) 10, 14–23. Brain Res. Cogn. Brain Res. 5, Annu. Rev. Psychol. 60, 27–51.
ical effects distinguishable from the Hasson, U., Harel, M., Levy, I., 255–272. Marinkovic, K., Dhond, R. P., Dale,
effects of shared meaning and shared and Malach, R. (2003). Large- Kemmerer, D., Castillo, J. G., Talavage, A. M., Glessner, M., Carr, V., and
form? J. Exp. Psychol. Learn. Mem. scale mirror-symmetry organization T., Patterson, S., and Wiley, C. Halgren, E. (2003). Spatiotemporal
Cogn. 26, 1431–1444. of human occipito-temporal object (2008). Neuroanatomical distribu- dynamics of modality-specific and
Fischer, M. H., and Zwaan, R. A. (2008). areas. Neuron 37, 1027–1041. tion of five semantic components supramodal word processing. Neu-
Embodied language: a review of the Heil, M. (2002). The functional signif- of verbs: evidence from fMRI. Brain ron 38, 487–497.
role of the motor system in language icance of ERP effects during men- Lang. 107, 16–43. McDermott, K. B., and Roediger, H. L.
comprehension. Q. J. Exp. Psychol. tal rotation. Psychophysiology 39, Kiefer, M., Sim, E. J., Helbig, H., and III. (1994). Effects of imagery on
(Colchester) 61, 825–850. 535–545. Graf, M. (2011). Tracking the time perceptual implicit memory tests. J.
Fodor, J. A. (1983). The Modularity of Henson, R. N., Goshen-Gottstein, Y., course of action priming on object Exp. Psychol. Learn. Mem. Cogn. 20,
Mind. Cambridge, MA: MIT Press. Ganel, T., Otten, L. J., Quayle, A., and recognition: evidence for fast and 1379–1390.
Forster, K. I., and Davis, C. (1984). Repe- Rugg, M. D. (2003). Electrophys- slow influences of action on per- McPherson, W. B., and Holcomb, P.
tition priming and frequency atten- iological and haemodynamic cor- ception. J. Cogn. Neurosci. 23, 1864– J. (1999). An electrophysiological
uation in lexical access. J. Exp. Psy- relates of face perception, recogni- 1874. investigation of semantic priming
chol. Learn. Mem. Cogn. 10, 680–698. tion and priming. Cereb. Cortex 13, Koester, D., and Schiller, N. O. (2008). with pictures of real objects. Psy-
Franklin, M. S., Dien, J., Neely, J. 793–805. Morphological priming in overt lan- chophysiology 36, 53–65.
H., Huber, E., and Waterson, L. D. Henson, R. N., Rylands, A., Ross, guage production: electrophysiolog- Meteyard, L., Cuadrado, S. R., Bahrami,
(2007). Semantic priming modu- E., Vuilleumeir, P., and Rugg, M. ical evidence from Dutch. Neuroim- B., and Vigliocco, G. (2011). Com-
lates the N400, N300, and N400RP. D. (2004). The effect of repe- age 42, 1622–1630. ing of age: a review of embodiment
Clin. Neurophysiol. 118, 1053–1068. tition lag on electrophysiological Kosslyn, S. M. (1994). Image and Brain: and the neuroscience of semantics.
Freyd, J. J., and Finke, R. A. (1984). and haemodynamic correlates of The Resolution of the Imagery Debate. Cortex 48, 788–804.
Facilitation of length discrimina- visual object priming. Neuroimage Cambridge, MA: MIT Press. Michelon, P., and Koenig, O. (2002). On
tion using real and imaged context 21, 1674–1689. Kosslyn, S. M., Alpert, N. M., Thomp- the relationship between imagery
frames. Am. J. Psychol. 97, 323–341. Herholz, S. C., Halpern, A. R., and son, W. L., Chabris, C. F., Rauch, S. and visual perception: evidence from
Ganis, G., and Kosslyn, S. M. (2007). Zatorre, R. J. (2012). Neuronal cor- L., and Anderson, A. K. (1994). Iden- priming studies. Eur. J. Cogn. Psy-
“Multiple mechanisms of top-down relates of perception, imagery, and tifying objects seen from different chol. 14, 161–184.

Michelon, P., Snyder, A. Z., Buck- visual imagery. J. Exp. Psychol. 103, Schendan, H. E., and Kutas, M. (2002). Sitnikova, T., Perrone, C., Goff, D.,
ner, R. L., Mcavoy, M., and Zacks, 509–514. Neurophysiological evidence for two and Kuperberg, G. R. (2010). Neu-
J. M. (2003). Neural correlates of Philiastides, M. G., Ratcliff, R., and processing times for visual object rocognitive mechanisms of concep-
incongruous visual information. An Sajda, P. (2006). Neural represen- identification. Neuropsychologia 40, tual processing in healthy adults and
event-related fMRI study. Neuroim- tation of task difficulty and deci- 931–945. patients with schizophrenia. Int. J.
age 19, 1612–1626. sion making during perceptual cat- Schendan, H. E., and Kutas, M. (2003). Psychophysiol. 75, 86–99.
Michelon, P., and Zacks, J. M. (2003). egorization: a timing diagram. J. Time course of processes and repre- Spironelli, C., Galfano, G., Umilta,
What is primed in priming from Neurosci. 26, 8965–8975. sentations supporting visual object C., and Angrilli, A. (2011). Word
imagery? Psychol. Res. 67, 71–79. Philiastides, M. G., and Sajda, P. (2006). identification and memory. J. Cogn. position affects stimulus recogni-
Morris, C. D., Bransford, J. D., and Temporal characterization of the Neurosci. 15, 111–135. tion: evidence for early ERP short-
Franks, J. J. (1977). Levels of process- neural correlates of perceptual deci- Schendan, H. E., and Kutas, M. (2007). term plastic modulation. Int. J. Psy-
ing versus transfer appropriate pro- sion making in the human brain. Neurophysiological evidence for the chophysiol. 82, 217–224.
cessing. J. Verbal Learn. Verbal Behav. Cereb. Cortex 16, 509–518. time course of activation of global Stanfield, R. A., and Zwaan, R. A. (2001).
16, 519–533. Philiastides, M. G., and Sajda, P. (2007). shape, part, and local contour repre- The effect of implied orientation
Morton, J. (1979). “Facilitation in word EEG-informed fMRI reveals spa- sentations during visual object cat- derived from verbal context on pic-
recognition: experiments causing tiotemporal characteristics of per- egorization and memory. J. Cogn. ture recognition. Psychol. Sci. 12,
change in the Logogen model,” in ceptual decision making. J. Neurosci. Neurosci. 19, 734–749. 153–156.
Proceedings of the Conference on the 27, 13082–13091. Schendan, H. E., and Lucia, L. C. Thierry, G., Martin, C. D., Downing, P.,
Processing of Visible Language, eds P. Puce, A., Allison, T., and Mccarthy, G. (2009). Visual object cognition pre- and Pegna, A. J. (2007). Controlling
A. Kolers, M. E. Wrolstad, and H. (1999). Electrophysiological studies cedes but also temporally overlaps for interstimulus perceptual vari-
Bouma (New York: Plenum), 259– of human face perception III: effects mental rotation. Brain Res. 1294, ance abolishes N170 face selectivity.
268. of top-down processing on face- 91–105. Nat. Neurosci. 10, 505–511.
Moulton, S. T., and Kosslyn, S. M. specific potentials. Cereb. Cortex 9, Schendan, H. E., and Lucia, L. C. (2010). Tsao, D. Y., Freiwald, W. A., Tootell, R.
(2009). Imagining predictions: men- 445–458. Object-sensitive activity reflects ear- B., and Livingstone, M. S. (2006).
tal imagery as mental emulation. Puce, A., and Perrett, D. (2003). Elec- lier perceptual and later cognitive A cortical region consisting entirely
Philos. Trans. R. Soc. Lond. B Biol. trophysiology and brain imaging of processing of visual objects between of face-selective cells. Science 311,
Sci. 364, 1273–1280. biological motion. Philos. Trans. R. 95 and 500 ms. Brain Res. 1329, 670–674.
Nemrodov, D., and Itier, R. J. (2012). Soc. Lond. B Biol. Sci. 358, 435–445. 124–141. Tucker, M., and Ellis, R. (1998). On the
Is the rapid adaptation paradigm Pulvermuller, F. (1999). Words in the Schendan, H. E., and Maher, S. M. relations between seen objects and
too rapid? Implications for face and brain’s language. Behav. Brain Sci. 22, (2009). Object knowledge during components of potential actions. J.
object processing. Neuroimage 61, 253–279; discussion 280–336. entry-level categorization is acti- Exp. Psychol. Hum. Percept. Perform.
812–822. Pylyshyn, Z. (2003). Return of the men- vated and modified by implicit 24, 830–846.
Nessler, D., Mecklinger, A., and tal image: are there really pictures in memory after 200 ms. Neuroimage Tulving, E., and Thomson, D. M. (1973).
Penney, T. B. (2005). Perceptual the brain? Trends Cogn. Sci. (Regul. 44, 1423–1438. Encoding specificity and retrieval
fluency, semantic familiarity and Ed.) 7, 113–118. Schendan, H. E., and Stern, C. E. processes in episodic memory. Psy-
recognition-related familiarity: an Rossell, S. L., Price, C. J., and Nobre, (2008). Where vision meets mem- chol. Rev. 80, 359–380.
electrophysiological exploration. A. C. (2003). The anatomy and time ory: prefrontal-posterior networks Vigliocco, G., Warren, J., Siri, S., Arci-
Brain Res. Cogn. Brain Res. 22, course of semantic priming investi- for visual object constancy dur- uli, J., Scott, S., and Wise, R. (2006).
265–288. gated by fMRI and ERPs. Neuropsy- ing categorization and recognition. The role of semantics and gram-
Neumann, M. F., and Schweinberger, chologia 41, 550–564. Cereb. Cortex 18, 1695–1711. matical class in the neural represen-
S. R. (2008). N250r and N400 Rugg, M. D., and Curran, T. (2007). Scott, L. S., Tanaka, J. W., Sheinberg, D. tation of words. Cereb. Cortex 16,
ERP correlates of immediate famous Event-related potentials and recog- L., and Curran, T. (2006). A reevalu- 1790–1796.
face repetition are independent of nition memory. Trends Cogn. Sci. ation of the electrophysiological cor- Vogel, E. K., and Luck, S. J. (2000).
perceptual load. Brain Res. 1239, (Regul. Ed.) 11, 251–257. relates of expert object processing. J. The visual N1 component as an
181–190. Rumiati, R. I., Papeo, L., and Corradi- Cogn. Neurosci. 18, 1453–1465. index of a discrimination process.
Nielsen-Bohlman, L., and Knight, R. Dell’acqua, C. (2010). Higher-level Sehatpour, P., Molholm, S., Javitt, D. C., Psychophysiology 37, 190–203.
T. (1994). Electrophysiological dis- motor processes. Ann. N. Y. Acad. Sci. and Foxe, J. J. (2006). Spatiotem- Voss, J. L., Schendan, H. E., and Paller, K.
sociation of rapid memory mech- 1191, 219–241. poral dynamics of human object A. (2010). Finding meaning in novel
anisms in humans. Neuroreport 5, Schacter, D. L., Addis, D. R., and Buck- recognition processing: an inte- geometric shapes influences electro-
1517–1521. ner, R. L. (2008). Episodic simula- grated high-density electrical map- physiological correlates of repeti-
O’Hare, A. J., Dien, J., Waterson, L. D., tion of future events: concepts, data, ping and functional imaging study tion and dissociates perceptual and
and Savage, C. R. (2008). Activa- and applications. Ann. N. Y. Acad. of “closure” processes. Neuroimage conceptual priming. Neuroimage 49,
tion of the posterior cingulate by Sci. 1124, 39–60. 29, 605–618. 2879–2889.
semantic priming: a co-registered Schacter, D. L., Wig, G. S., and Stevens, Sehatpour, P., Molholm, S., Schwartz, Warren, C., and Morton, J. (1982). The
ERP/fMRI study. Brain Res. 1189, W. D. (2007). Reductions in cortical T. H., Mahoney, J. R., Mehta, A. effects of priming on picture recog-
97–114. activity during priming. Curr. Opin. D., Javitt, D. C., Stanton, P. K., and nition. Br. J. Psychol. 73, 117–129.
Paivio, A., and Sadoski, M. (2011). Lex- Neurobiol. 17, 171–176. Foxe, J. J. (2008). A human intracra- Wassenburg, S. I., and Zwaan, R. A.
icons, contexts, events, and images: Schendan, H. E., Ganis, G., and Kutas, nial study of long-range oscillatory (2010). Readers routinely represent
commentary on Elman (2009) from M. (1998). Neurophysiological evi- coherence across a frontal-occipital- implied object rotation: the role of
the perspective of dual coding the- dence for visual perceptual cat- hippocampal brain network during visual experience. Q. J. Exp. Psychol.
ory. Cogn. Sci. 35, 198–209. egorization of words and faces visual object processing. Proc. Natl. (Colchester) 63, 1665–1670.
Pascual-Marqui, R. D. (2002). Stan- within 150 ms. Psychophysiology 35, Acad. Sci. U.S.A. 105, 4399–4404. Wilson, H. R., and Humanski, R. A.
dardized low resolution electromag- 240–251. Sitnikova, T., Goff, D., and Kuperberg, (1993). Spatial frequency adaptation
netic tomography (sLORETA): tech- Schendan, H. E., Kanwisher, N. G., and G. R. (2009). Neurocognitive abnor- and contrast gain control. Vision Res.
nical details. Methods Find. Exp. Kutas, M. (1997). Early brain poten- malities during comprehension of 33, 1133–1149.
Clin. Pharmacol. 24, 5–12. tials link repetition blindness, prim- real-world goal-directed behaviors Wilson, M. (2002). Six views of embod-
Peterson, M. J., and Graham, S. ing and novelty detection. Neurore- in schizophrenia. J. Abnorm. Psychol. ied cognition. Psychon. Bull. Rev. 9,
E. (1974). Visual detection and port 8, 1943–1948. 118, 256–277. 625–636.

Zhang, X. L., Begleiter, H., Porjesz, Conflict of Interest Statement: The Citation: Schendan HE and Ganis Copyright © 2012 Schendan and Ganis.
B., Wang, W. Y., and Litke, A. authors declare that the research was G (2012) Electrophysiological poten- This is an open-access article distrib-
(1995). Event related potentials dur- conducted in the absence of any com- tials reveal cortical mechanisms for uted under the terms of the Creative
ing object recognition tasks. Brain mercial or financial relationships that mental imagery, mental simulation, Commons Attribution License, which
Res. Bull. 38, 531–538. could be construed as a potential con- and grounded (embodied) cogni- permits use, distribution and reproduc-
Zwaan, R. A., Stanfield, R. A., and Yax- flict of interest. tion. Front. Psychology 3:329. doi: tion in other forums, provided the orig-
ley, R. H. (2002). Language com- 10.3389/fpsyg.2012.00329 inal authors and source are credited
prehenders mentally represent the Received: 07 May 2012; accepted: 17 This article was submitted to Frontiers in and subject to any copyright notices
shapes of objects. Psychol. Sci. 13, August 2012; published online: 14 Sep- Perception Science, a specialty of Frontiers concerning any third-party graphics
168–171. tember 2012. in Psychology. etc.

published: 18 January 2013
doi: 10.3389/fpsyg.2012.00561
A cross-modal perspective on the relationships between

imagery and working memory
Lora T. Likova*
The Smith-Kettlewell Eye Research Institute, San Francisco, CA, USA
Edited by: Mapping the distinctions and interrelationships between imagery and working memory
(WM) remains challenging. Although each of these major cognitive constructs is defined
and treated in various ways across studies, most accept that both imagery andWM involve a
Reviewed by:
Holly Bridge, University of Oxford, UK form of internal representation available to our awareness. In WM, there is a further empha-
Axel Kohler, University of Münster, sis on goal-oriented, active maintenance, and use of this conscious representation to guide
Germany voluntary action. Multicomponent WM models incorporate representational buffers, such
*Correspondence: as the visuo-spatial sketchpad, plus central executive functions. If there is a visuo-spatial
Lora T. Likova, The Smith-Kettlewell
“sketchpad” for WM, does imagery involve the same representational buffer? Alternatively,
Eye Research Institute, 2318 Fillmore
Street, San Francisco, CA, USA. does WM employ an imagery-specific representational mechanism to occupy our aware-
e-mail: lora@ski.org ness? Or do both constructs utilize a more generic “projection screen” of an amodal nature?
To address these issues, in a cross-modal fMRI study, I introduce a novel Drawing-Based
Memory Paradigm, and conceptualize drawing as a complex behavior that is readily adapt-
able from the visual to non-visual modalities (such as the tactile modality), which opens
intriguing possibilities for investigating cross-modal learning and plasticity. Blindfolded par-
ticipants were trained through our Cognitive-Kinesthetic Method (Likova, 2010a, 2012) to
draw complex objects guided purely by the memory of felt tactile images. If this WM task
had been mediated by transfer of the felt spatial configuration to the visual imagery mech-
anism, the response-profile in visual cortex would be predicted to have the “top-down”
signature of propagation of the imagery signal downward through the visual hierarchy.
Remarkably, the pattern of cross-modal occipital activation generated by the non-visual
memory drawing was essentially the inverse of this typical imagery signature. The sole
visual hierarchy activation was isolated to the primary visual area (V1), and accompanied
by deactivation of the entire extrastriate cortex, thus ’cutting-off’ any signal propagation
from/to V1 through the visual hierarchy. The implications of these findings for the debate
on the interrelationships between the core cognitive constructs of WM and imagery and
the nature of internal representations are evaluated.
Keywords: visual imagery, visuo-spatial sketchpad, working memory, primary visual cortex V1, drawing, fMRI
INTRODUCTION for the temporarily storage and manipulation of visuo-spatial

Mapping the distinctions and interrelationships between imagery material, the phonological loop is posited to provide a similar
and working memory (WM) remains challenging. Although each function for verbal material. An enhanced version of the mul-
of these major cognitive constructs is defined and treated in var- ticomponent WM model added an episodic buffer (Baddeley,
ious ways across studies, most accept that both imagery and WM 2003).
involve a type of internal representation available to our awareness; Interestingly, the involvement of the early visual cortex, and
in WM, however, there is a further emphasis on goal-oriented, area V1 in particular, has been a critical issue in discussions both
active maintenance and use of this conscious representation to on the neural substrate of the putative visuo-spatial sketchpad and
guide voluntary action. on the nature of imagery. The V1-substrate propositions are based
WM refers to the temporary storage and manipulation of infor- mainly on the fact that, although most areas in occipital cortex are
mation, and is invoked as the mechanism for information pro- topographically organized, area V1 has the unique status of being
cessing during the performance of a wide range of everyday tasks the largest topographic map in the brain, with the highest spatial res-
(e.g., Baddeley and Hitch, 1974; Baddeley, 1986, 1992, 2000, 2003; olution, in addition to parallel processing of the information from
Logie et al., 1989; Logie and Marchetti, 1991; Logie, 1995; Badde- the whole map surface (in contrast to the sequential processing in
ley and Andrade, 2000). Initially, the proposed structure included some other modalities).
the central executive component and two active storage buffers – All of these features are critically important for a successful
the visuo-spatial sketchpad and the articulatory/phonological loop. “sketchpad” implementation, which is why previous theoretical as
While the visuo-spatial sketchpad is considered to be responsible well as neurophysiological studies in non-human primates (e.g.,
www.frontiersin.org January 2013 | Volume 3 | Article 561 | 99

Likova Cross-modal perspective on imagery and memory
Mumford, 1991, 1996; Lee et al., 1998; Super et al., 2001a,b; Lee and (non-pictorial propositional imagery coding), and Methodologi-
Mumford, 2003; Super, 2003) had suggested V1 as the source of cal Factors (determining factors, such as low neuroimaging reso-
the high-resolution visuo-spatial “sketchpad” function:“instead of lution or differential resting activation in postulated imagery loci,
being the first stage in a feedforward pipeline,V1 is better described that may need to be controlled in order to resolve imagery-specific
as the unique high-resolution buffer in the visual system” (Lee and activation). Their analysis identified three variables that optimally
Mumford, 2003). predicted the differences in the probability of activation across
However, the same characteristics of this region are also key imagery studies. Notably, two of the variables were task-dependent
requirements for the existence of a pictorial-code form of visual requirements (the requirement to note high-resolution details in
imagery. Thus, the issue of V1 involvement has been central in the the stimuli and the requirement to visualize shapes rather than
long-standing debate about the nature of visual imagery, which abstract spatial relations), while the third was purely technical
relates to the question of whether the imagery “code” is pictor- (sufficiently high sensitivity of the technique).
ial or propositional (Kosslyn et al., 2001; Kosslyn and Thompson, Thus, the operation of the (pictorial) visual imagery and of the
2003). visuo-spatial sketchpad concepts share (i) similar task require-
Neuroimaging is a valuable tool that can help to resolve these ments, and (ii) similar need of V1 usage (although note that,
issues, but while there has been much neuroimaging work on in contrast to the putative sketchpad, the top-down theories of
imagery, this is not the case with the putative WM sketchpad. imagery are not restricted to V1 but require activation of the whole
The neural substrate for visual imagery has been found to largely visual hierarchy). A number of behavioral studies have addressed
overlap with that for visual perception (e.g., Ishai and Sagi, 1995; possible interactions between WM and visual imagery (e.g., Bruyer
Kosslyn et al., 1999; Kreiman et al., 2000; O’Craven and Kanwisher, and Scailquin, 1998; Baddeley, 2000), with the most recent by
2000; Kosslyn et al., 2001; Kosslyn and Thompson, 2003; Mechelli Keogh and Pearson (2011) suggesting an imagery-dependent
et al., 2004), with the activation pattern implying that the signal dichotomy in cognitive strategies for visual WM.
propagates from higher cortical regions in a top-down manner
through the visual hierarchy toward V1. The resultant top-down QUESTIONS
gradient of activation provides a notable signature of the visual Despite the array of studies on the issue, the above review indi-
imagery activation pattern; consistent with this employment of cates that the neural substrates as well as the interactions between
the visual pathway for imagery, there is no significant negative these two cognitive constructs are still far from being definitively
signal (i.e., no flow interruption) for imagery in occipital cortex resolved. In particular, the visuo-spatial sketchpad remains an
(Ganis et al., 2004). almost entirely theoretical construct. If there is a visuo-spatial
While a substantial activation in the higher areas has been con- “sketchpad” for WM, does imagery use the same representational
sistently found across the imagery studies, this has not been the buffer? Alternatively, does WM employ an imagery-specific rep-
case with V1. Although some level of V1 activation during imagery resentational mechanism to occupy our awareness? Or do both
has been reported in several studies (e.g., Kosslyn et al., 1993; Le constructs utilize a more generic “projection screen” of an amodal
Bihan et al., 1993; Sabbah et al., 1995; Kosslyn et al., 1996; Chen nature?
et al., 1998; Shin et al., 1999; Thompson et al., 2001; Ishai et al.,
2002; Lambert et al., 2002; Ganis et al., 2004), a larger number of A NOVEL APPROACH
studies did not find any V1 activation at all (e.g., Goldenberg et al., The drawing-based memory paradigm
1991; Charlot et al., 1992; Mellet et al., 1995, 1998a,b; D’Esposito Likova (2010a, 2012) recently conceptualized the drawing task as
et al., 1997; Ishai et al., 2000; Knauff et al., 2000; Trojano et al., the basis for a novel memory paradigm to address these questions.
2000; Wheeler et al., 2000; Formisano et al., 2002; Sack et al., 2002; Drawing, and in particular memory-guided drawing, challenges
Mazard et al., 2004; Kaas et al., 2010). Importantly, even when both the encoding of detailed spatial representations and their
V1 was activated during imagery, the signal there was significantly explicit retrieval from memory for “projection” back onto a mental
weaker than in the extrastriate visual areas. Thus, the level of V1 high-resolution “screen” to guide the movements of the draw-
activation is of great importance for the imagery debate, as imagery ing hand with the requisite precision. A cortical region such as
activation in V1 implies the usage of a pictorial code. Kaas et al. V1 would be an ideal neural implementation of the required
(2010), however, whose primary goal was “to eliminate the effects “screen”; thus, the putative V1 visuo-spatial sketchpad is a plau-
of (short- or long-term) memory in the investigation of the effects sible theoretical construct that provides for memory retrieval for
of mental imagery,” conclude that their results “suggest that the just the kinds of spatial representations involved in the drawing
activation in early visual areas observed in previous imagery stud- task, allowing for the active maintenance of information about
ies might be related to short- or long-term memory retrieval of stimuli no longer in view. (It is not by chance that this ubiquitous
specific sensory experiences.” tool of the real drawing process – the use of a disposable sketch-
To analyze the disparate results on the V1 involvement in the pad – was the metaphor employed for the memory module in
imagery literature, Kosslyn and Thompson (2003) defined sets question.)
of variables associated with each of three theories, which were Importantly, the drawing-from-tactile-memory task effectively
then fit to the observed results using logistic regression analy- transcends simple “recognition memory.” An easy demonstration
sis to discover how well each theory predicted when early visual makes the point: Close your eyes and try to imagine the objects
cortex was activated. The three theories were Perceptual Antici- on your desk, the face of your close friend, or even your own face;
patory Theory (pictorial imagery coding), Propositional Theory in particular, try to “see” the detailed shapes as though you are
Frontiers in Psychology | Perception Science January 2013 | Volume 3 | Article 561| 100
going to draw them. It is amazing how misleading the feeling is motor loop, and consequently it demands “high-resolution”
that we “know” any of those very well. Despite the fact that we internal representations to be communicated back through the
would effortlessly recognize them (recognition memory), when we drawing act.
try to retrieve the details, they fade or somehow escape our grasp, 5. Studies on memory would highly benefit from tasks providing
although we do not feel these information gaps when recognizing “direct” memory-control. Drawing -from-memory incorporates
immediately the object as a whole. Many early representational such direct control by providing direct memory “readout,” as it
details seem to be lost along the passage through the visual pathway ensures an explicit expression of the remembered information
before they have been integrated into the internal reconstruction by externalization of the mental representation that guides the
of the face/object that they represent. In other words, recognition drawing hand.
memory seems to operate at the higher-level of object category
processing and does not need to retain the more “local” level These considerations led me to the choice of non-visual draw-
of detail. In contrast, a highly detailed kind of spatial memory ing, which incorporates all of the above principles, as a paradigm
is engaged to meet the needs of the efferent drawing task; this for both training and studying cross-modal memory. The role of
“memory-for-drawing ” preserves and recalls details sufficient to WM and imagery in mediating the training effects were evaluated
enable the complex spatiomotor act of producing an accurate by functional Magnetic Resonance Imaging (fMRI).
drawing.
The cognitive-kinesthetic drawing method
Conceptual framework To employ this novel memory paradigm for studying learning-
Both the novel Drawing-Based Memory Paradigm and the based plasticity, a method is needed to train non-sighted people
Cognitive-Kinesthetic Training Method, are based on a framework to draw not simply without visual feedback, but guided solely by
of principles (Likova, 2012), including: non-visual memory. Recently I have developed a novel technique,
the Cognitive-Kinesthetic Drawing Method, which proved to be
1. Space transcends any specific sensory modality. As emphasized very effective in the successful training of people under total visual
by the phenomenon of drawing by the blind (e.g., Kennedy, deprivation. Congenitally blind, late-onset blind and blindfolded
1993, 2000; Kennedy and Igor, 2003; Kennedy and Juricevic, were successfully trained in only a week of 1.5 h/day sessions to
2006; Ponchillia, 2008; Likova, 2010a, 2012), space, and spatial draw complex face and object structures (as opposed to simple
structure are not represented solely by the visual modality. The geometric or grid structures), guided solely by tactile-memory
visual system is best suited to process spatial information, but (Likova, 2010a,b, 2012).
it is not the only one. Thus, when deprived of visual input, In contrast to standard imaging studies of tactile activation, the
the brain is capable of employing the “free” visual resources fMRI evaluation that was run before and after the training was
in the most relevant way. (As there is an ambiguity in the use specifically designed to probe the memory involvement by record-
of the term “spatial,” particularly in the WM and imagery lit- ing the brain activity while drawing-from-memory in the absence
erature, note that when used in this paper, “spatial” refers to of any visual or tactile input from the learned raised-line drawing
the perception of any spatial structure – 2D or 3D, static or templates (see Materials and Methods). This novel Drawing-Based
dynamic – independently of the sensory modality exploring it. Memory Paradigm has the unique advantage of providing an
For example, a face can be recognized by exploring its spatial explicit memory “readout” of the specific memory representa-
structure with the hands, or a geometric function can be “seen” tion that guides it. Importantly, the Cognitive-Kinesthetic training
by audio-graphics, etc.) My view is that drawing deals with spa- allows subjects to learn to draw from memory the specific mem-
tial structures in this general sense, and consequently it has the orized objects and faces that they had explored, not just some
advantage that can readily be “translated” from a visual into a long-standing “clichés,” thus showing that the particular memory-
tactile form. representations generated during the tactile exploration phase were
2. Closing the perception-cognition-action loop is a powerful ampli- guiding their drawing.
fier for learning, so the task selection is critical. Drawing is To provide for comparative pre/post-training analyses of brain
a complex task precisely orchestrating multiple brain mech- activation, an innovative platform was developed including the
anisms, and consequently, it provides for an integrative, first MRI-compatible multisensory drawing tablet, with a stylus
perception-cognition-action paradigm. incorporating a fiber-optic motion-capture system to record the
3. Training in highly engaging unfamiliar tasks that provide fun drawing movements in the scanner for off-line analyses.
and inspiring outcomes is a fruitful paradigm for driving brain These unique capabilities allow for testing between the follow-
reorganization and assessing its earliest stages. Drawing inher- ing hypotheses:
ently embodies all of these components, particularly when
studied under the circumstances of visual deprivation (which Hypothesis I. If the memory-based drawing task is mediated by
are unusual for a “visual” art); the characteristic for draw- a transfer of the tactually felt spatial configuration to the visual
ing sense of completion, creativity, and fulfillment amplify the imagery mechanism, the predicted response-profile in the visual
experience-based plasticity. cortex would have the top-down “imagery signature” of propaga-
4. Tasks demanding detailed re-expression of memory-representations tion of the imagery signal downward through the visual hierarchy,
force the development of precise and robust memory. Drawing - with activation significantly decreasing from the higher extrastriate
from-memory demands such explicit re-expression through the areas toward V1.
www.frontiersin.org January 2013 | Volume 3 | Article 561| 101

Hypothesis II. If V1 was being employed as a WM sketchpad inde- of the stylus tip was recorded with high precision. Scribble was a
pendently from the visual imagery process, it would be activated control for both movement-specific activation (due to hand or eye
by a separate pathway external to the visual hierarchy, together movements) and absence of a memory involvement; the subjects
with activation of WM-related sites beyond the occipital lobe. had to move the stylus with the right hand in a random trajectory
Moreover, if V1 activation is found in non-sighted drawing, it will over the right slot of the tablet matching the extent and rate of the
also confirm a re-conceptualization of the sketchpad buffer from drawing movements, but under instructions to avoid planning or
being visuo-spatial to being independent from sensory modality, imagining any particular trajectory or cognitive content.
or amodal as previously proposed on the basis of a study in the
congenitally blind (Likova, 2012). Rational of the drawing requirements
(i) The requirement that the models were always explored with
MATERIALS AND METHODS the left hand but drawn by the right hand was an advanced aspect
EXPERIMENTAL DESIGN of the experimental design ensuring that in the MD task the cor-
A battery of raised-line models of faces and objects was developed tex controlling the right (drawing) hand did not have any direct
as the drawing targets in a three-task block fMRI paradigm, with “haptic knowledge” of the image. This design enforces the devel-
interleaved baseline conditions (Figure 1A). The three tasks were opment of a detailed memory representation in order to transfer
as follows: Explore/Memorize (E/M ) – perceptual exploration and the information later to the opposite (drawing) hand. (ii) Further-
memorization of the model to be drawn; MemoryDraw (MD) – more, the left hand was not allowed to follow the contour being
a memory-guided non-visual drawing task; and Scribble (S) – a drawn by the right hand, ensuring that the subjects learned to
motor-control and negative memory-control task. Each task dura- draw from memory without relying on specific tactile feedback of
tion was 20 s, with a 20 s baseline condition [NullInterval (NI )] the raised-line configuration. Moreover, the reliance on detailed
intervening between the tasks. Importantly, as opposed to the enough memory was further guaranteed by the fact that the virtual
usual null periods, the subjects not only rested motionless but were stylus left no tactually perceivable trace, eliminating any possibility
instructed and practiced to clear any memory or image structure for tactile tracing during drawing in the scanner. Together, these
from awareness (“mind-blank”). The start of each task or null design features enforce the encoding of a robust spatial memory
interval was prompted by an auditory cue. The whole three-task representation needed to guide the drawing trajectory. The quality
sequence with interleaved null intervals (NI, E/M, NI, MD, NI, S, of the reproductions was assessed by a masked rating procedure,
NI ) was repeated 12 times in each 1-hour fMRI session using a based on recognition and similarity to the templates.
new image for each repeat.
In Explore/Memorize, using the left hand only, the subjects had TACTILE STIMULUS PRESENTATION AND HAND MOVEMENT CONTROL
to tactually explore a raised-line drawing model on the left slot of Multisensory MRI-compatible drawing system
the drawing tablet, and to develop a full memory representation of To run drawing studies in the scanner is not a conventional
the image in preparation for the MD task. Then the model image protocol and faced a lot of challenging technological problems.
was removed, and the subjects rested motionless for 20 s with no We developed a multisensory drawing-system that: (1) is MRI-
image in mind (NI ). In the following MD phase the fiber-optic compatible, (2) is ergonomically adaptable to the small space
stylus was used to draw the image (from tactile-memory) on the available inside the scanner bore, (3) allows multiple tactile images
right slot of the tablet using only the right hand; the trajectory to be presented sequentially in the scanner without the need
FIGURE 1 | Experimental design. (A) Drawing was investigated in a between the tasks, the whole 140 s trial sequence being repeated 12 times in
three-phase paradigm consisting of a memory-guided drawing task, each scanning session using a new image for each repeat. (B) Raised-line
abbreviated as “MemoryDraw” (MD), plus two control tasks: a motor-control drawings of realistic faces and objects were presented as templates to be
and negative memory-control task “Scribble” (S), and a task of perceptual explored by the subject using her left hand. The quality of the reproductions
exploration and memorization of the model to be drawn “Explore/Memorize” was assessed by a masked rating procedure, based on recognition and
(E/M). Each task’s duration was 20 s, with 20 s null intervals interposed similarity to the templates (examples of reproduction are shown in Figure 3).
Frontiers in Psychology | Perception Science January 2013 | Volume 3 | Article 561 | 102
of any operator assistance, (4) captures and records the draw- were collected with EPI acquisition from the whole head coil.
ing trajectory with high precision, and (5) provides a real-time There were 34 axial slices at 2 s TR, with TE of 28 ms and flip angle
visual feedback when appropriate (Likova, 2010a). It incorporates of 80˚, providing 3.0 mm × 3.0 mm × 3.5 mm voxels throughout
a dual-slot drawing tablet that is height/distance adjustable for the the brain. The functional activations were processed for slice-time
subject’s arm length and a specially adapted version of a fiber- correction and motion correction. An anatomical segmentation
optic device for motion capture of the drawing movements with algorithm (mrGray) was applied to the T1 scan, ensuring local-
high-resolution (Figure 2). This unique drawing-system supports ization of the signal within the cortical gray matter close to the
the fMRI investigation of both tactilely and visually guided draw- activated neurons and greatly reducing the blood drain artifacts
ing. It also allows us to record relevant behavioral and feedback of BOLD signals displaced from the neural activation sites, which
events and to correlate them to the brain activation for full off-line afflict studies in which cortical segmentation is not used.
analysis.
FMRI time-course analyses
Auditory cue presentation The analysis software was Stanford Vision and Imaging Science and
The auditory stimuli were presented through Resonance Tech- Technology (VISTA) software. The data were analyzed to estimate
nologies Serene Sound earphones (Resonance Technologies, the effective neural activation amplitudes (for each task across the
Salem, MA, USA). To reduce scanner noise, this equipment 12 repeats of the three-task sequence in the 1-h scan) by the fol-
employs external ear protectors with perforated earplugs that con- lowing procedure. A General Linear Model (GLM) consisting of a
duct the auditory cues directly into the auditory passage while (3 + 1)-parameter boxcar neural activation model convolved with
blocking much of the scanner noise. an estimated hemodynamic response function (HRF) was fitted
to the BOLD responses (i.e., a 1-parameter boxcar for each of
MRI DATA COLLECTION, ANALYSIS, AND VISUALIZATION the three tasks, plus a 1-parameter 8-boxcar sequence to model
Subjects the auditory cue presentations). An additive fourth-order poly-
The study was conducted on a group of six subjects with normal nomial was applied to capture low-frequency drift in the BOLD
vision who were blindfolded during the experiments. The subjects signal. (The HRF parameters were determined once per session
ranged in age from 25 to 59 and were four females and two males. by optimizing this model to a subset of gray matter voxels iden-
All subjects gave informed consent for the experimental proto- tified as most responsive to the task/null alternation frequency in
col approved by the local research ethics committee, Institutional this experiment.) Thus, the parameters of the activation model
Review Board. consisted of the boxcar activation amplitudes for the three-task
periods, relative to the remainder of the 140 s scan duration.
FMRI acquisition
MR data were collected on a Siemens Trio 3T scanner equipped Voxel-wise parametric maps
with eight-channel EXCITE capability, a visual stimulus presen- For each task statistical parametric maps were generated, based on
tation system, response buttons. A high-resolution anatomical the estimated activation amplitudes from the above GLM in each
(T 1-weighted) volume scan of the entire brain was obtained voxel. As is standard for GLM, the boxcar neural activation model
for each subject (voxel size = 0.8 mm × 0.8 mm × 0.8 mm). The for each 20 s task period was contrasted with the entire remainder
fMRI blood-oxygenation-level-dependent (BOLD) responses of the 140 s scan duration. All three task-models were optimized
FIGURE 2 | A subject on the scanner bed operating our novel multimodal movements with high precision. The motion-capture information synchronized
MRI-compatible drawing system. The plexiglass gantry supports a drawing with the fMRI allows the effect of behavioral events to be analyzed to high
tablet while a fiber-optic drawing stylus captures and records the drawing precision.
www.frontiersin.org January 2013 | Volume 3 | Article 561| 103

jointly to the detrended BOLD waveform. Also, for the pre-post areas V3A, V3B, hV4, and V7 were specified in accordance with
comparison, voxel-wise maps of the change in activation follow- Tyler et al. (2005). The retinotopic mapping was done by using
ing the training were generated, scaled in terms of z-score of the standard retinotopic stimuli – expanding rings and traveling
pre-post difference signals. These maps could be viewed in the wedges. An innovative 14-step procedure (Likova, 2010a) allowed
3D volume or projected onto 3D views of the inflated cortex or us to warp the brains to the same MNI brain coordinates, within
flat-maps of cortical regions of particular interest. which other localizers could also be specified on the basis of prior
studies.
ROI activation analysis
The effective neural activation amplitudes (bar-graphs) for each
RESULTS
condition in each region of interest (ROI) were estimated by the
The subjects were all able to improve their drawing skills so as to
same GLM procedure but now applied to the average signal across
complete each drawing with its particular characteristics within
all voxels within the ROI. This procedure also provided high-
the 20 s allotted for the drawing tasks (which took many times
quality time courses for evaluation of the response dynamics and
longer before training). Examples of drawing recorded in the post-
its comparison across tasks and stages of training.
training scanning session are shown in Figure 3, illustrating the
The confidence intervals in the bar-graphs were defined by the
level of detail required to complete each drawing.
amplitude variability of the 12 repeats of the three-task sequence
in each 1-h scan. The dashed lines and the error bars represent con-
fidence intervals for two different forms of statistical comparison BOLD RESPONSES TO DRAWING GUIDED BY TACTILE-MEMORY
of the activation levels (i.e., of the beta weights for the event types The initial analysis gives an overview of the averaged cortical
in the GLM): (1) The dashed lines represent the 99% “zero” confi- activation for the MD task in the brains of the group of blind-
dence interval (p < 0.01, uncorrected) within which the activation folded subjects. Figure 4 shows the post-training map, projected
amplitudes are not significantly different from zero (i.e., relative on inflated representations of the lateral and medial views of the
to the noise variance for no stimulus-related activation defined two hemispheres. As expected, there is strong activation in the
as the residual variance after the GLM model fit of the FMRI pre-motor, motor, and somatosensory cortex in the dorsal regions
time-course analyses section described above); thus this statistical of the brain (the pre-central and post-central sulci, PreCS and
criterion is designed to indicate the significance of each individual PostCS) predominantly for the left hemisphere controlling the
activation (at p < 0.05, corrected for multiple applications within right hand that was performing the drawing task (Figures 4A,B).
each figure); (2) The error bars are “difference” confidence intervals The anterior and posterior dorsal regions also showed enhanced
designed to illustrate the t-test for the significance of differences activation bilaterally, implying enhanced kinesthetic processing for
between activation levels in each figure (i.e., the differences are not the drawing movements. The supplementary motor area (SMA, on
significant unless they exceed the confidence intervals for both the medial surface), which plays a role in the planning of complex
compared activations). In the text, all ROI-comparisons are speci- coordinated movements is also activated bilaterally, although again
fied as significant by the t-test using a statistical criterion threshold the left hemisphere responds more strongly. The dorsolateral-
of p < 0.05 corrected for multiple comparisons. prefrontal cortex, known to be of key importance in WM, decision-
making, executive control, etc. is activated in both hemispheres.
Topographic mapping Temporal lobe activation can be also seen in the LOtv region sug-
The boundaries of the retinotopic projection areas V1, V2d, V2v, gested to be involved in tactile object processing. The involvement
V3d, and V3v were established as described in Sereno et al. (1995); of many of these regions would be very much predicted on the basis
Tootell et al. (1996); Engel et al. (1997). Retinotopic projection of prior studies. There is also an extensive network of deactivation
FIGURE 3 | Examples of blindfolded drawings of the vase with a flower, recorded in the scanner by the motion-capture system show a lot of specific
the face profile, and the boot (the corresponding templates shown in detail, which makes them readily recognizable as specific examples of their
Figure 1B are easy to recognize: the first and the second in the top row, category, although they were drawn without visual or tactile input (i.e., with
and the first in the bottom row). Remarkably, the post-training drawings, eye-hand coordination eliminated), but were guided solely by tactile-memory.
FIGURE 4 | BOLD activation and deactivation in non-visual memory network that corresponds broadly to the default-mode network (except for
drawing in the blindfolded. Post-training group responses from the MD the occipital lobe portion). (C,D) Both medial views show massive
task are derived according to the GLM described in Section “Materials and activation along the calcarine sulcus (CaS) corresponding to V1 surrounded
Methods,” and projected on inflated representations of the lateral left (A) by deactivation, which extends throughout the lateral regions of the visual
and right (B), and medial left (C) and right (D) hemispheres in MNI brain cortex. Activation is shown down to −1 < z < 1; the scale bar indicates the
coordinates. Dark-gray, sulci; light gray, gyri. (A,B) A non-occipital network color-coding for the respective z-score levels. Note that, interestingly, the
of temporal, parietal, and frontal regions is activated (yellow-orange medial CaS activation spreads to the same eccentricity in both
coloration), together with strong deactivation (blue-cyan coloration) in a hemispheres.
that, beyond the occipital lobe regions, largely overlaps with the significances, the color-coding provides explicit information as to
default-mode network (e.g., Raichle and Snyder, 2007). the activation pattern for any preferred threshold level.
What is surprising within the traditional view of brain archi- To better evaluate the activation/deactivation pattern in the con-
tecture, however, is the massive activation in the occipital region text of the functional “geography” of the visual cortex, Figure 5A
along the calcarine sulcus (CaS) for this non-visually and not shows a flat-map representation of the occipital cortex of a repre-
even sensory (neither visually nor tactually) guided task (in the sentative blindfolded subject who had all of the visual-hierarchy
sense that, as described in Methods, the task has been accom- areas mapped, as well as the motion complex hMT+ and LOC.
plished with no visual input at any stage, and the drawing phase The flat-maps are centered on the occipital pole and oriented as
has been accomplished with guidance only by tactile-memory, i.e., if viewed from the back of the head, with activation/deactivation
with no concurrent sensory input of any form about the image to designated as in Figure 4. The retinotopic boundaries were deter-
be drawn). This region corresponds to the location of area V1 and mined in a separate scan using a 20˚ circular field. Note that the
is the key focus of the present analysis. The strong V1 activation peripheral boundaries of these regions thus correspond to about
(orange-yellow coloration) can be seen on the medial surface in 10˚ of eccentricity, which corresponds approximately to the max-
both hemispheres (Figures 4C,D). imum extent of the BOLD activation (yellow-orange coloration).
Activation is shown down to −1 < z < 1 because the lower Area V1 is outlined by a red contour. It is remarkable to see that the
amplitudes appear to form a consistent fringe around the cluster tactile-memory drawing not only generates activation specific to
regions of high significance. Indeed, such a visualization approach V1 (although no sensory visual or even tactile information about
is not often taken; however, in cases where not only the posi- the drawing templates was available), while the entire extrastriate
tive but the negative clusters are also of interest, this visualization hierarchy that surrounds V1 is massively deactivated. The extended
approach allows the following to be shown: (i) whether the sur- non-visual response in what traditionally is considered as primary
rounding regions similarly show activation but just below the “visual” cortex was accompanied by activation in a region of cortex
threshold level, or (ii) whether there is an extended cluster of deac- at the occipitotemporal border known to be involved in tactile
tivation. Higher-threshold presentations mask such differences; object recognition, LOtv (e.g., Amedi et al., 2001, 2002; Reed et al.,
thus, it seems worthwhile to provide maximum information in the 2004); activation is also seen dorsally in the caudal intraparietal
figures. For restricting to the individual voxel (rather than cluster) sulcus (cIPS).

FIGURE 5 | (A) MD flat-maps centered on the occipital pole. ROIs for the the caudal intraparietal sulcus (cIPS) dorsally and an additional locus at the
retinotopic hierarchy are indicated by colored outlines, with hMT+ and LOC occipitotemporal border (LOtv). (B) Average response amplitude with
based on functional localizers. The post-training MD map shows a “triad” of standard errors for blindfolded memory-guided drawing in a group of six
three activation regions (orange-yellow coloration). Note in particular that the subjects, showing positive signal in the triad of areas – primary visual area V1,
(non-stimulated visually) primary visual cortex, V1, forms an unusual isolated cIPS, and LOtv; these three “islands” of positive activation are separated by
“island” of activation surrounded by a “sea” of suppression in the adjacent strong deactivation in both the ventral and the dorsal extrastriate areas. Error
retinotopic areas. The other two activated regions seen on the flat map are bars represent 1 standard error of the means.
Average amplitudes across the ROIs shown in Figure 5A are (white bars); the four dark-gray bars indicate the 20 s null inter-
quantified for a group of six normally sighted subjects performing vals separating E/M, MD, and S tasks. The bar-graphs in Figure 7B
the drawing task under blindfolded conditions in Figure 5B. As show the estimated activation in each hemisphere for each task.
in the example subject, the only three regions in this part of the Importantly, the MD task (red color), which requires retrieval of
cortex showing significant activation are V1, and the two regions detailed tactile-memory representation is the one that produces
beyond the visual retinotopic hierarchy, LOtv, and cIPS. All the the strongest activation in V1; note that eliminating the memory
other regions show significant reductions in the BOLD below component (in the “non-memory” drawing task S, green color)
baseline. correspondingly eliminates any response in this purely motor form
A difference MD map, which represents voxel-wise compar- of the task. Although the E/M task (blue color), which repre-
ison of the post-training BOLD activation relative to the presents the memory encoding phase, also seems to employ V1 to
training level is presented in the CaS region in Figure 6. It some degree, significantly less activation is observed than for the
shows a pronounced bilateral increase of activation after the MD task. Correspondingly, Figures 7C,D show the average time-
Cognitive-Kinesthetic training. courses and bar-graphs for the deactivated extrastriate regions.
The MD response for these regions is the most prominent and
CROSS-TASK COMPARISON OF V1 ACTIVATION highly significantly negative. E/M and S, on the other hand, have
Figure 7A below shows the average time-courses of BOLD activ- marginally significant responses, but are not significantly differ-
ity (black lines) in V1 for the sequence of the three-task intervals ent from each other. All cross-task ROI-comparisons in the text
FIGURE 6 | The MD task shows the predominant training right (RH) hemispheres, with orange-yellow coloration showing
effect in the primary visual cortex. A voxel-wise comparison, the average pre/post increase in BOLD activation for MD in the
projected on inflated representations of the posterior left (LH) and CaS (V1) region.
FIGURE 7 | Comparison of the activation pattern across the three tasks. represent the 99% “zero” confidence interval, within which the activations
(A) Average time-courses of BOLD activity (black curve) in V1 for the are not significantly different from zero. Error bars are 99% “difference”
sequence of the E/M, MD, and S task intervals (white bars); the colored confidence intervals designed to illustrate the t-test to assess the
curves are the best fits of the model predictions for the three tasks to the significance of the differences between pairs of activation levels in each
time course; the four dark-gray bars indicate the 20 s null intervals separating figure, i.e., amplitude differences are not significant unless they exceed the
the three tasks. (B) Bar-graphs for the estimated V1 activation for each task; confidence intervals for both compared activations. (C) Average time-courses
the activation levels refer to the beta weights for the event types in the GLM. of BOLD activity in the deactivated regions surrounding V1. (D) Bar-graphs of
Dotted lines and the error bars represent confidence intervals for two estimated activation for each task for the deactivated regions presented in
different forms of statistical comparison of the activation levels. Dotted lines (C). Conventions in (C,D) are as in (A,B), respectively.

FIGURE 8 | The proposed re-conceptualization of the (after Baddeley, 2003), where the added “Amodal-Spatial
visuo-spatial sketchpad as an amodal-spatial sketchpad. Sketchpad” block depicts our re-conceptualization of the
Modified schematic of the main modules of Baddeley’s classic visuo-spatial sketchpad as being accessible to any sensory modality
model of working memory including the visuo-spatial sketchpad (from Likova, 2012).
are specified as significant by the t-test at a statistical criterion That is, in principle, it is possible that the memory of the complex
threshold of p < 0.05, corrected for multiple comparisons. spatial structures could in some way be transferred to higher-order
Although the time-course plots show that there was no strong visual imagery “processors,” which in turn may have mediated
activation during the “mind-blank” null periods between the task the corresponding conscious experience via a top-down propa-
epochs, to formally assess the degree of memory involvement in gation through the visual hierarchy to a high-resolution memory
V1 following the E/M phase, we ran regressors for all null periods, representation in V1.
and compared the one after E/M with the average for the remain- However, this hypothesis seems not to be supported by the data
der of the null periods. As expected based on the “mind-blank” because, although we find strong activation in the iconic visual
design of these periods, there were no significant differences in the area V1, it is implausible that the signal is “delivered” through the
V1 ROI (p 0.1), implying that there was no evidence for mem- visual hierarchy, as this hierarchy is not only not activated but is
ory engagement of V1 during this null period. The corresponding massively deactivated. This implies that V1 is “cut-off ” from the
analysis was also run on the deactivated extrastriate ROI with the higher-level cortical regions that could generate and propagate the
same result. imagery signals. It is important to emphasize that this pattern of
V1-activation/extrastriate-deactivation is quite distinct, almost the
DISCUSSION inverse of the known hierarchical pattern for visual imagery (which
Employing a novel memory paradigm based on drawing in nor- is a top-down process strongest in the higher extrastriate areas and
mally sighted subjects while blindfolded revealed that the primary decreasing toward the lower areas, often not reaching V1 at all; see
“visual” area V1 can be strongly activated in a non-visual WM Introduction). The implications of the deactivation cutting-off V1
task: the task of blindfolded drawing guided solely by memory from receiving signal through the extrastriate visual pathway go
acquired during haptic exploration of complex spatial structures, further, beyond simple judgment of “similarity” or “difference” of
such as raised-line objects and faces. Furthermore, the pattern of activation patterns, to imply functional incompatibility with the
response showed a number of unique characteristics: main principle of visual imagery as a process propagating through
the visual pathway downward to V1.
(i) The occipital activation was largely restricted to area V1. Consequently, the unique pattern of response in the blindfolded
(ii) The strong V1 activation was remarkably well structured, is not compatible with an explicit role for visual imagery in this
ceasing rapidly at a specific eccentricity. form of WM. Instead, the strong but isolated V1 activation seems
(iii) Surprisingly, the V1 activation was surrounded by massive to be more consistent with Hypothesis II that V1 is operating as a
deactivation of the entire extrastriate visual hierarchy. WM component, such as the spatial memory-buffer/sketchpad of
the composite WM model.
In contrast to the parallel nature of typical visual processing, The training paradigm of the current study also provides a
haptic exploration operates in a sequential manner. As the blind- causal manipulation that links the memory enhancement to the
folded are subjects who have an intact visual system, and presum- increase in V1 activation as a result of the (non-visual) Cognitive-
ably a well developed visual imagery mechanism, one possibility Kinesthetic training, consistent with the memory-buffer interpre-
it is that once the spatiotemporal integration of the sequentially tation of the role of V1. Moreover an important twist for this
explored template-objects was completed, the memory retrieval interpretation is the lack of any visual stimulation under the
was implemented by the visual imagery mechanism (Hypothesis I ). blindfolded conditions, implying that activation of the V1 buffer
should be independent of the input modality. In other words, the putative WM sketchpad in the MD task, and not as a visual
the present results imply that the nature of the V1-buffer is not imagery component, they have further significance beyond this
“visuo-spatial” but “amodal-spatial”. specific dichotomy.
Importantly, the results in the blindfolded reinforce the pre- In a more general sense, the current results provide a strong
vious implication from a parallel study of memory drawing in a demonstration of massive employment of V1 in a higher-order
congenitally blind subject (Likova, 2012) that V1 was operating cognitive task that involves no visual (or even tactile) sensory stim-
as an amodal memory-buffer because the subject had had a com- ulation. Thus, in principle, these results do not exclude a broader
plete absence of visual experience and visual memory throughout hypothesis, specifically that V1 may play the role of a more generic
life and had performed the task entirely based on the memory “projection screen” of an amodal nature, which could be utilized
from the tactile input. The strong post-training activation of V1 by each of the two main cognitive constructs discussed here, as
in that study thus could not meaningfully be attributed to visual well as by visual WM and by other forms of cognitive functions
imagery, but is more consistent with the hypothesis that V1 uses requiring such a high-resolution “projection screen.” Depending
an amodal-spatial representation in its operation as the puta- on the specific task needs, it may be utilized in either a cross-modal
tive memory-buffer. Our re-conceptualization of the visuo-spatial or intramodal manner.
sketchpad as being amodal-spatial is depicted by the yellowish
block in Figure 8. POSSIBLE MECHANISMS
It is important to note, however, that such an interpretation The general field of visuotactile interactions, especially with
does not mean that V1 is employed for storage of the memory respect to primary visual cortex in non-visual sensory stimulation,
trace. Moreover, in contrast to the usual format of a baseline con- began from studies in the blind (e.g., Sadato et al., 1996; Cohen
dition, we instructed and practiced the subjects to eliminate any et al., 1997; Zangaladze et al., 1999; Pascual-Leone and Hamilton,
rehearsal of either the just-explored templates or of any other 2001; Block, 2003). Although further analyses are needed to inves-
memory images for the full 20 s duration of each null interval. tigate what underlying mechanisms may mediate the cross-modal
This duration is also too long to account for the known reten- activation observed in V1 in the current tactile-memory task, there
tion time of any iconic image of a memory trace, that is of the is a range of general theoretical possibilities, such as unmasking of
order of a second or less (Sperling, 1963; Di Lollo, 1980). Since the pre-existing inter-region connections, changes in synaptic weights,
drawings were not experienced as spatial images during the null modulation of long-range intercortical influences, up-regulation
interval, they were evidently held in some other, non-conscious of non-local transmitter sources, or a variety of subcortical mech-
storage location until it was needed for the subsequent drawing anisms (e.g., Florence and Kaas, 1995; Jones, 2000; Pascual-Leone
task. and Hamilton, 2001; Raineteau and Schwab, 2001; Block, 2003;
Further support for the general idea of early sensory areas in Merabet et al., 2008; Van Brussel et al., 2011).
human being involved in some form in WM comes from the semi- Indeed, the extent of V1 connectivity is currently undergo-
nal study of Harrison and Tong (2009), which “demonstrated that ing an extensive re-evaluation. Recent electrophysiological and
early visual areas can retain specific information about visual fea- anatomical studies in non-human primates reveal a picture of mul-
tures held in WM.” However, similarly to visual imagery (and in tiple reciprocal connections at lower hierarchical levels, including
contrast to our data), the whole sequence of early visual areas – the primary areas. In addition to the well-known direct feedback
V1, V2, V3, V3A–V4 was activated in that study. Thus, as should projections to V1 originating from the visual hierarchy (Perkel
be expected for a visual process, visual WM did not suppress but et al., 1986; Ungerleider and Desimone, 1986a,b; Shipp and Zeki,
activated the extrastriate areas, i.e., it did not “cut-off ” the V1 1989; Rockland, 1994; Budd, 1998; Barone et al., 2000; Suzuki
signal propagation through the visual pathways. Their result of et al., 2000), there are direct feedforward projections to V1 orig-
“widespread activity throughout the early visual system” makes it inating from a variety of subcortical structures, including the
clear that visually driven WM uses pathways different from those pulvinar, LGNd, claustrum, nucleus paracentral, raphe system,
in our non-visually driven WM task, which had deactivated all of locus coeruleus, and the nucleus basalis of Meynert (Ogren and
the extrastriate visual areas. Hendrickson, 1976; Rezak and Benevento, 1979; Graham, 1982;
Thus, visual WM can not account for our data, while on the Blasdel and Lund, 1983; Doty, 1983; Perkel et al., 1986; Lachica
other hand, our concept of an amodal (modality-independent) and Casagrande, 1992; Hendry and Yoshioka, 1994; Adams et al.,
spatial-sketchpad in V1 is consistent with the V1 involvement in 2000; Schmolesky, 2000). In principle, any of these subcortical
both visual and non-visual WM, as its amodal nature implies structures could be involved in the processes of memory storage
availability through the visual as through the tactile modality. and retrieval for the performance of the high-resolution drawing
Therefore, the present results encourage an expanded view of the task.
V1 functionality to a cross-modal involvement in WM processing Additionally, Clavagnier et al. (2004), examined feedback pro-
(the access to which, however, would require different pathways jections to area V1 using retrograde tracer injections. Notably,
for different modalities). in addition to well-known areas and a number of long-
distance feedback connections originating from auditory (A1) and
GENERALIZATION BEYOND THE SKETCHPAD: V1 AS A GENERIC multisensory (STP) cortices, they also found connections from
“PROJECTION SCREEN” OF AN AMODAL NATURE? a perirhinal area. The perirhinal-to-V1 connections appear of
Although the present results are consistent with our Hypothe- particular interest (Likova, 2012) in the context of our finding
sis II, namely that V1 was operating as the neural substrate for of a memory-related role for V1, as the perirhinal area is adjacent

to the hippocampus and has a well-established role in memory from a congenitally blind study (Likova, 2012) that the spatial
storage and retrieval. WM sketchpad may operate in an amodal (rather than exclu-
These connections therefore could represent another potential sively visual) fashion. The converse hypothesis that the memory
pathway for the involvement of V1 in WM and the active pro- retrieval would activate the visual hierarchy in a top-down fash-
cessing of stored spatial information. Nevertheless, none of the ion (as would be expected if this task were mediated through
multiple connections above directly predicts the way that V1 is the imagery network), was not supported by the data. In a more
cut-off from the surrounding visual hierarchy by the deactivation. general sense, the results are a strong demonstration of a mas-
Definitive studies on these issues remain to be conducted. sive cross-modal activation in V1 in a high-level cognitive func-
tion. In combination with our previous work with the Drawing-
CONCLUSION Based Memory Paradigm and training effects of the Cognitive-
This novel experimental approach, showing how a WM task Kinesthetic protocol, these studies further propel the emerging
accesses the highest resolution topographic map in the brain re-conceptualization of brain architecture as highly interactive and
(V1) even under non-visual conditions, provides a “real-life” capable of plastic reorganization even after short-term sensory
yet tractable paradigm for addressing the role of such high- deprivation.
order cognitive processes in a cross-modal manner. V1 was
activated in an isolated fashion in the drawing-from-memory ACKNOWLEDGMENTS
task, supporting our Hypothesis II that V1 operates as the This research is supported by NSF/SLC Grant to Lora T. Likova.
active spatial “sketchpad” underlying the accurate drawing per- The author thanks Spero Nicholas for his help in data pre-
formance under non-visual conditions. The blindfolded draw- processing and analysis tools, and Christopher W. Tyler for fruitful
ing results were also consistent with the previous conclusion discussions.
REFERENCES distribution of neurons in extrastri- Cogn. Affect. Behav. Neurosci. 4, Goldenberg, G., Podreka, I., Steiner, M.,
Adams, M. M., Hof, P. R., Gattass, R., ate areas projecting to visual areas V1 117–126. Franzen, P., and Deecke, L. (1991).
Webster, M. J., and Ungerleider, L. and V4 correlates with the hierarchi- Cohen, L. G., Celnik, P., Pascual- Contributions of occipital and tem-
G. (2000). Visual cortical projections cal rank and indicates the operation Leone, A., Corwell, B., Falz, L., poral brain regions to visual and
and chemoarchitecture of macaque of a distance rule. J. Neurosci. 20, Dambrosia, J., et al. (1997). Func- acoustic imagery-A SPECT study.
monkey pulvinar. J. Comp. Neurol. 3263–3281. tional relevance of cross-modal plas- Neuropsychologia 29, 695–702.
419, 377–393. Blasdel, G. G., and Lund, J. S. (1983). ticity in blind humans. Nature 389, Graham, J. (1982). Some topographi-
Amedi, A., Jacobson, G., Hendler, T., Termination of afferent axons in 180–183. cal connections of the striate cor-
Malach, R., and Zohary, E. (2002). macaque striate cortex. J. Neurosci. D’Esposito, M., Detre, J. A., Aguirre, G. tex with subcortical structures in
Convergence of visual and tactile 3, 1389–1413. K., Stallcup, M., Alsop, D. C., Tippet, Macaca fascicularis. Exp. Brain Res.
shape processing in the human lat- Block, N. (2003). Tactile sensation via L. J., et al. (1997). A functional MRI 47, 1–14.
eral occipital complex. Cereb. Cortex spatial perception. Trends Cogn. Sci. study of mental image generation. Harrison, S. A., and Tong, F. (2009).
12, 1202–1212. (Regul. Ed.) 7, 285–286. Neuropsychologia 35, 725–730. Decoding reveals the contents of
Amedi, A., Malach, R., Hendler, T., Bruyer, R., and Scailquin, J. C. (1998). Di Lollo, V. (1980). Temporal integra- visual working memory in early
Peled, S., Zohary, E. (2001). Visuo- The visuospatial sketchpad for men- tion in visual memory. J. Exp. Psy- visual areas. Nature 458, 632–635.
haptic object-related activation in tal images: Testing the multicom- chol. Gen. 109, 75–97. Hendry, S. H., and Yoshioka, T. (1994).
the ventral visual pathway. Nat. Neu- ponent model of working mem- Doty, R. W. (1983). Nongeniculate affer- A neurochemically distinct third
rosci. 4, 324–330. ory. Acta Psychol. (Amst.) 98, ents to striate cortex in macaques. J. channel in the macaque dorsal lat-
Baddeley, A. D. (1986). Working Mem- 17–36. Comp. Neurol. 218, 159–173. eral geniculate nucleus. Science 264,
ory. New York, NY: Oxford Univer- Budd, M. L. (1998). Extrastriate feed- Engel, S. A., Glover, G. H., and Wandell, 575–577.
sity Press. back to primary visual cortex in B. A. (1997). Retinotopic organiza- Ishai, A., Haxby, J. V., and Ungerleider, L.
Baddeley, A. D. (1992). Working mem- primates: a quantitative analysis of tion in human visual cortex and the G. (2002). Visual imagery of famous
ory. Science 255, 556–559. connectivity. Proc. Biol. Sci. 265, spatial precision of functional MRI. faces: Effects of memory and atten-
Baddeley, A. D. (2000). The episodic 1037–1044. Cereb. Cortex 7, 181–192. tion revealed by fMRI. Neuroimage
buffer: a new component of working Charlot, V., Tzourio, N., Zilbovicius, M., Florence, S. L., and Kaas, J. H. (1995). 17, 1729–1741.
memory? Trends Cogn. Sci. (Regul. Mazoyer, B., and Denis, M. (1992). Large-scale reorganization at multi- Ishai, A., and Sagi, D. (1995). Common
Ed.) 4, 417–423. Different mental imagery abilities ple levels of the somatosensory path- mechanisms of visual imagery and
Baddeley, A. D. (2003). Working mem- result in different regional cerebral way follows therapeutic amputation perception. Science 268, 1771–1974.
ory: looking back and looking blood flow activation patterns dur- of the hand in monkeys. J. Neurosci. Ishai, A., Ungerleider, L., and Haxby, J. V.
forward. Nat. Rev. Neurosci. 4, ing cognitive tests. Neuropsychologia 15, 8083–8095. (2000). Distributed neural systems
829–839. 30, 565–580. Formisano, E., Linden, D. J., Di Salle, for the generation of visual images.
Baddeley, A. D., and Andrade, J. (2000). Chen, W., Kato, T., Zhu, X.-H., Ogawa, F., Trojano, L., Esposito, F., Sack, Neuron 28, 979–990.
Working memory and the vividness S., Tank, D. W., and Ugurbil, K. A. T., et al. (2002). Tracking the Jones, E. G. (2000). Plasticity and neuro-
of imagery. J. Exp. Psychol. Gen. 129, (1998). Human primary visual cor- mind’s image in the brain: I. plasticity. J. Hist. Neurosci. 9, 37–39.
126–145. tex and lateral geniculate nucleus Timeresolved fMRI during visu- Kaas, A., Weigelt, S., Roebroeck, A.,
Baddeley, A. D., and Hitch, G. J. (1974). activation during visual imagery. ospatial mental imagery. Neuron 35, Kohler, A., and Muckli, L. (2010).
“Working memory,” in The Psychol- Neuroreport 9, 3669–3674. 185–194. Imagery of a moving object: the
ogy of Learning and Motivation, Vol. Clavagnier, S., Falchier, A., and Kennedy, Ganis, G., Thompson, W. L., and Koss- role of occipital cortex and human
8, ed. G. Bower (San Diego, CA: H. (2004). Long-distance feedback lyn, S. M. (2004). Brain areas under- MT/V5+. Neuroimage 49, 794–804.
AcademicPress), 47–90. projections to area V1: implications lying visual mental imagery and Kennedy, J. (1993). Drawing and the
Barone, P., Batardiere, A., Knoblauch, for multisensory integration, spatial visual perception: An fMRI study. Blind. New Haven, CT: Yale Univer-
K., and Kennedy, H. (2000). Laminar awareness, and visual consciousness. Cogn. Brain Res. 20, 226–241. sity Press.
Kennedy, J. M. (2000). “Recognizing human primary visual cortex during emission tomography study of visual 17) in the macaque monkey. Brain
outline pictures via touch: Align- visual recall: a magnetic resonance and mental spatial exploration. J. Res. 167, 19–40.
ment theory,” in Touch, Representa- imaging study. Proc. Natl. Acad. Sci. Cogn. Neurosci. 7, 433–445. Rockland, K. S. (1994). “The organiza-
tion and Blindness, ed. M. A. Heller U.S.A. 90, 11802–11805. Merabet, L. B., Hamilton, R., Schlaug, tion of feedback connections from
(Oxford: Oxford University Press), Lee, T. S., and Mumford, D. (2003). G., Swisher, J. D., Kiriakopou- areaV2 (18) to V1 (17),” in Cerebral
67–98. Hierarchical Bayesian inference in los, E. T., Pitskel, N. B., et Cortex, Vol. 10: Primary Visual Cor-
Kennedy, J. M., and Igor, J. (2003). the visual cortex. J. Opt. Soc. Am. A al. (2008). Rapid and reversible tex in Primates, eds A. Peters and K. S.
Haptics and projection: drawings by Opt. Image Sci. Vis. 20, 1434–1448. recruitment of early visual cor- Rockland (NewYork, NY: Plenum),
Tracy, a blind adult. Perception 32, Lee, T. S., Mumford, D., Romero, R., tex for touch. PLoS ONE 3:e3046. 261–299.
1059–1071. and Lamme, V. A. F. (1998). The doi:10.1371/journal.pone.0003046 Sabbah, P., Simond, G., Levrier, O.,
Kennedy, J. M., and Juricevic, I. (2006). role of the primary visual cortex in Mumford, D. (1991). On the computa- Habib, M., Trabaud, V., Murayama,
Foreshortening, convergence and higher level vision. Vision Res. 38, tional architecture of the neocortex N., et al. (1995). Functional mag-
drawings from a blind adult. Percep- 2429–2454. I: the role of the thalamo-cortical netic resonance imaging at 1.5 T dur-
tion 35, 847–851. Likova, L. T. (2010a). “Drawing in the loop. Biol. Cyber. 65, 135–145. ing sensorimotor and cognitive task.
Keogh, R., and Pearson, J. (2011). blind and the sighted as a probe Mumford, D. (1996). “Commentary on Eur. Neurol. 35, 131–136.
Mental imagery and visual work- of cortical reorganization,” in Pro- banishing the homunculus by H. Sack, A. T., Sperling, J. M., Prvulovic, D.,
ing memory. PLoS ONE 6:e29221. ceedings of the SPIE7527 Human Barlow,” in Perception as Bayesian Formisano, E., Goebel, R., Di Salle,
doi:10.1371/journal.pone.0029221 Vision and Electronic Imaging, eds Inference, eds D. C. Knill and W. F., et al. (2002). Tracking the mind’s
Knauff, M., Kassubek, J., Mulack, T., B. E. Rogowitz and T. N. Pappas, Richards (Cambridge: Cambridge image in the brain: II. Transcra-
and Greenlee, M. W. (2000). Corti- 752708–752714. University Press), 501–504. nial magnetic stimulation reveals
cal activation evoked by visual men- Likova, L. T. (2010b). The pri- O’Craven, K. M., and Kanwisher, N. parietal asymmetry in visuospatial
tal imagery as measured by fMRI. mary visual cortex as a modality- (2000). Mental imagery of faces imagery. Neuron 35, 195–204.
Neuroreport 11, 3957–3962. independent ‘screen’ for working and places activates corresponding Sadato, N., Pascual-Leone, A., Grafman,
Kosslyn, S. M., Alpert, N. M., Thomp- memory. J. Vision 10:776. stimulus-specific brain regions. J. J., Ibañez, V., Deiber, M. P., Dold, G.,
son, W. L., Maljkovic, V., Weise, Likova, L. T. (2012). Drawing enhances Cogn. Neurosci. 12, 1013–1023. et al. (1996). Activation of the pri-
S. B., Chabris, C. F., et al. (1993). cross-modal memory plastic- Ogren, M., and Hendrickson, A. (1976). mary visual cortex by Braille read-
Visual mental-imagery activates ity in the human brain: A case Pathways between striate cortex ing in blind subjects. Nature 380,
topographically organized visual study in a totally blind adult. and subcortical regions in Macaca 526–528.
cortex – pet investigations. J. Cogn. Front. Hum. Neurosci. 6:44. mulatta and Saimiri sciureus: evi- Schmolesky, M. (2000). The Pri-
Neurosci. 5, 263–287. doi:10.3389/fnhum.2012.00044. dence for a reciprocal pulvinar con- mary Visual Cortex. Available at:
Kosslyn, S. M., Ganis, G., and Thomp- Logie, R. (1995). Visio-spatial Working nection. Exp. Neurol. 53, 780–800. http://webvision.med.utah.edu/Visual
son, W. L. (2001). Neural founda- Memory. Hove: Lawrence Erlbaum Pascual-Leone, A., and Hamilton, R. Cortex.html.
tions of imagery. Nat. Rev. Neurosci. Associates Ltd. (2001). The metamodal organiza- Sereno, M. I., Dale, A. M., Reppas, J. B.,
2, 635–642. Logie, R. H., Baddeley, A. D., Mane, A., tion of the brain. Prog. Brain Res. Kwong, K. K., Belliveau, J. W., Brady,
Kosslyn, S. M., Pascual-Leone, A., Feli- Donchin, E., and Sheptak, R. (1989). 134, 427–445. T. J., et al. (1995). Borders of multi-
cian, O., Camposano, S., Keenan, J. Working memory and the analy- Perkel, D. J., Bullier, J., and Kennedy, ple visual areas in humans revealed
P., Thompson, W. L., et al. (1999). sis of a complex skill by secondary H. (1986). Topography of the affer- by functional magnetic resonance
The role of area 17 in visual imagery: task methodology. Acta Psychol. 71, ent connectivity of area 17 of the imaging. Science 268, 889–893.
convergent evidence from PET and 53–87. macaque monkey: a double labeling Shin, L. M., McNally, R. J., Kosslyn,
rTMS. Science 284, 167–170. Logie, R. H., and Marchetti, C. (1991). study. J. Comp. Neurol. 253, 374–402. S. M., Thompson, W. L., Rauch,
Kosslyn, S. M., Shin, L. M., Thompson, “Visio-soatial working memory: Ponchillia, P. E. (2008). “Non-visual S. L., Alpert, N. M., et al. (1999).
W. L., McNally, R. J., Rauch, S. L., visual, spatial or central executive?” sports and arts: fertile substrates Regional cerebral blood flow during
Pitman, R. K., et al. (1996). Neural in Mental Images in Human Cogni- for the growth of knowledge about script-driven imagery in childhood
effects of visualizing and perceiving tion, eds R. H. Logie and M. Denis brain plasticity in people who are sexual abuse-related PTSD: a PET
aversive stimuli: A PET investiga- (Amsterdam: Elsevier), 105–115. blind or have low vision,” in Blind- investigation. Am. J. Psychiatry 156,
tion. Neuroreport 7, 1569–1576. Mazard, A., Tzourio-Mazoyer, N., Criv- ness and Brain Plasticity in Naviga- 575–584.
Kosslyn, S. M., and Thompson, W. L. ello, F., Mazoyer, B., and Mellet, tion and Object Perception, eds J. J. Shipp, S., and Zeki, S. M. (1989).
(2003). When is early visual cor- E. (2004). A PET meta-analysis of Rieser, D. H. Ashmead, F. F. Ebner, The organization of connections
tex activated during visual mental object and spatial mental imagery. and A. L. Corn (London: Lawrence between areas V5 and V1 in macaque
imagery? Psychol. Bull. 129, 723–746. Eur. J. Cogn. Psychol. 16, 673–695. Erlbaum Associates), 283–313. monkey visual cortex. Eur. J. Neu-
Kreiman, G., Koch, C., and Fried, I. Mechelli, A., Price, C. J., Friston, K. Raichle, M. E., and Snyder, A. Z. (2007). rosci. 1, 309–332.
(2000). Imagery neurons in the J., and Ishai, A. (2004). Where A default mode of brain function: Sperling, G. (1963). A model for visual
human brain. Nature 408, 357–361. bottom-up meets top-down: neu- A brief history of an evolving idea. memory tasks. Hum. Factors 5,
Lachica, E. A., and Casagrande, V. ronal interactions during percep- Neuroimage 37, 1083–1090. 19–31.
A. (1992). Direct W-like genicu- tion and imagery. Cereb. Cortex 14, Raineteau, O., and Schwab, M. E. (2001). Super, H. (2003). Working memory
late projections to the cytochrome- 1256–1265. Plasticity of motor systems after in the primary visual cortex. Arch.
oxidase (CO) blobs in primate visual Mellet, E., Petit, L., Mazoyer, B., incomplete spinal cord injury. Nat. Neurol. 60, 809–812.
cortex: axon morphology. J. Comp. Denis, M., and Tzourio, N. (1998a). Rev. Neurosci. 2, 263–273. Super, H., Spekreijse, H., and Lamme,
Neurol. 319, 141–158. Reopening the mental imagery Reed, C. L., Shoham, S., and Halgren, V. A. F. (2001a). A neural correlate
Lambert, S., Sampaio, E., Scheiber, C., debate: Lessons from functional E. (2004). neural substrates of tactile of working memory in the monkey
and Mauss, Y. (2002). Neural sub- anatomy. Neuroimage 8, 129–139. object recognition: An fMRI study. primary visual cortex. Science 293,
strates of animal mental imagery: Mellet, E., Tzourio, N., Denis, M., Hum. Brain Mapp. 21, 236–246. 120–124.
Calcarine sulcus and dorsal pathway and Mazoyer, B. (1998b). Cortical Rezak, M., and Benevento, L. A. (1979). Super, H., Spekreijse, H., and Lamme,
involvement – an fMRI study. Brain anatomy of mental imagery of con- A comparison of the organization V. A. F. (2001b). Two distinct
Res. 924, 176–183. crete nouns based on their dictionary of the projections of the dorsal lat- models of sensory processing
Le Bihan, D., Turner, R., Zeffiro, T. definition. Neuroreport 9,803–808. eral geniculate nucleus, the inferior observed in monkey primary visual
A., Cue´nod, C. A., Jezzard, P., and Mellet, E., Tzourio, N., Denis, M., pulvinar and adjacent lateral pulv- cortex (V1). Nat. Neurosci. 4,
Bonnerot, V. (1993). Activation of and Mazoyer, B. (1995). A positron inar to primary visual cortex (area 304–310.

Suzuki, W., Saleem, K. S., and Tanaka, K. visual stimulation. Cereb. Cortex 10, plasticity in adult mouse visual cor- Received: 15 June 2012; accepted: 28
(2000). Divergent backward projec- 473–481. tex following monocular enucle- November 2012; published online: 18
tions from the anterior part of the Tyler, C. W., Likova, L. T., Chen, ation. Cereb. Cortex 21, 2133–2146. January 2013.
inferotemporal cortex (area TE) in C. C., Kontsevich, L. L., Schira, Wheeler, M. E., Petersen, S. E., and Citation: Likova LT (2013) A cross-
the macaque. J. Comp. Neurol. 422, M. M., and Wade, A. R. (2005). Buckner, R. L. (2000). Memory’s modal perspective on the relation-
206–228. Extended concepts of occipital echo: Vivid remembering reactivates ships between imagery and working
Thompson, W. L., Kosslyn, S. M., Sukel, retinotopy. Curr. Med. Imag. Rev. 1, sensory-specific cortex. Proc. Natl. memory. Front. Psychology 3:561. doi:
K. E., and Alpert, N. M. (2001). 319–329. Acad. Sci. U.S.A. 97, 11125–11129. 10.3389/fpsyg.2012.00561
Mental imagery of high- and low- Ungerleider, L. G., and Desimone, R. Zangaladze, A., Epstein, C. M., Grafton, This article was submitted to Frontiers in
resolution gratings activates area 17. (1986a). Projections to the superior S. T., and Sathian, K. (1999). Involve- Perception Science, a specialty of Frontiers
Neuroimage 14, 454–464. temporal sulcus from the central and ment of visual cortex in tactile dis- in Psychology.
Tootell, R. B., Dale, A. M., Sereno, M. I., peripheral field representations of crimination of orientation. Nature Copyright © 2013 Likova. This is an
and Malach, R. (1996). New images Vl and V2. J. Comp. Neurol. 248, 401, 587–590. open-access article distributed under the
from human visual cortex. Trends 147–163. terms of the Creative Commons Attribu-
Neurosci. 95, 818–824. Ungerleider, L. G., and Desimone, R. Conflict of Interest Statement: The tion License, which permits use, distrib-
Trojano, L., Grossi, D., Linden, D. E. J., (1986b). Cortical connections of author declares that the research was ution and reproduction in other forums,
Formisano, E., Hacker, H., Zanella, F. visual area MT in the macaque. J. conducted in the absence of any com- provided the original authors and source
E., et al. (2000). Matching two imag- Comp. Neurol. 248, 190–222. mercial or financial relationships that are credited and subject to any copy-
ined clocks: The functional anatomy Van Brussel, L., Gerits, A., and Arckens, could be construed as a potential con- right notices concerning any third-party
of spatial analysis in the absence of L. (2011). Evidence for cross-modal flict of interest. graphics etc.
HUMAN NEUROSCIENCE published: 20 February 2012
doi: 10.3389/fnhum.2012.00024
Verbal to visual code switching improves working memory

in older adults: an fMRI study
Mariko Osaka 1 , Yuki Otsuka 2 and Naoyuki Osaka 2*
1
Department of Psychology, Graduate School of Human Sciences, Osaka University, Osaka, Japan
2
Department of Psychology, Graduate School of Letters, Kyoto University, Kyoto, Japan
Edited by: The effect of verbal to visual code switching training on working memory performance
Joel Pearson, The University of was investigated in individuals aged 63 and older. During verbal working memory task
performance, the training group (n = 25) was introduced to a verbal to visual code
Reviewed by:
switching strategy while the control group (n = 25) was not exposed to such a strategy.
Claude Alain, Rotman Research
Institute, Canada Working memory recognition accuracy was enhanced only in the training group. To explore
Axel Kohler, University of Münster, the neural substrates underlying these strategy effects, fMRI was used to measure brain
Germany activity in both groups during working memory task performance before and after an
*Correspondence: attention training period. In a comparison between pre- and post-training sessions, results
Naoyuki Osaka, Department of
showed increased activation in the anterior cingulate cortex (ACC). Relative to the control
Psychology, Graduate School of
Letters, Kyoto University, group, the post-training group exhibited increased activation in the left and right inferior
Yoshida-Honmachi, Sakyo-ku, parietal lobules (IPLs) and right superior parietal lobule (SPL). These findings suggest
Kyoto 606-8501, Japan. that use of a verbal to visual code switching strategy may assist older individuals in the
e-mail: nosaka@bun.kyoto-u.ac.jp
maintenance of information in working memory.
Keywords: working memory, older adults, fMRI, strategy training, attention switching
INTRODUCTION argued that these results provide evidence of brain plasticity in

Working memory serves simultaneous information, storage, and older individuals. Other studies indicated that training in the
processing functions (Baddeley, 1986). According to the multi- n-back task, which requires executive control of working memory,
component model proposed by Baddeley (Baddeley, 1986, 2003; is associated with improvements in the working memory per-
Baddeley and Logie, 1999), working memory is composed of formance of older adults (Dahlin et al., 2008a; Li et al., 2008).
two storage buffers: the phonological loop for storing verbal Dahlin et al. (2008b) showed that training in updating infor-
information and the visuo-spatial sketchpad for the storage of mation, which requires executive function in working memory,
visual information. Because working memory resources are lim- improved significantly in young and older adults relative to con-
ited, an executive system assigns and coordinates these limited trols. Based on these reports, it is likely that the dual-task deficits
resources for storing and manipulating information (Daneman typically observed in older individuals are caused not only by a
and Carpenter, 1980; Baddeley and Logie, 1999; Engle et al., decreased ability to maintain information in working memory,
1999). but also by deficits in the executive processing system.
Working memory impairments are commonly found in older The neural basis of the working memory system, particu-
adults. In particular, older individuals often have difficulty per- larly executive functioning, is thought to be located in the PFC;
forming dual tasks that require both storage and processing recent neuroimaging studies have suggested that the executive
(Salthouse, 1996; West, 1996; Craik and Salthouse, 2000). It has system is located in the dorsolateral PFC (DLPFC, BA9/46) and
been proposed that vulnerability of the frontal lobes accounts anterior cingulate cortex (ACC) (D’Esposito et al., 1995, 1999;
for the age-related impairments in memory functioning found in Smith et al., 1996; Cohen et al., 1997; Owen et al., 1998; Smith
older adults (Raz et al., 1997; Raz, 2000; Madden et al., 2004). and Jonides, 1999; Bunge et al., 2000; Osaka et al., 2003, 2004;
In accordance with this hypothesis, decreased prefrontal cortex Linden, 2007). The brain regions underlying working memory
(PFC) activation has been found in older adults through neu- appear to drive different executive functions. The DLPFC sup-
roimaging studies (Bunge et al., 2000; Cabeza, 2001). Age-related ports the active maintenance of attention on task goals, whereas
reductions in the ability to suspend non-task-related activity the ACC serves as an attentional coordinator, which enables indi-
and engage areas for carrying out memory tasks have also been viduals to inhibit inappropriate stimuli while monitoring task
reported (Grady et al., 2006). performance (MacDonald et al., 2000). ACC activation appears to
Several studies have examined whether training improves increase when cognitive tasks require greater attentional demands
working memory in older adults (Nyberg et al., 2003; Erickson (Petersen et al., 1988; Posner and Petersen, 1990).
et al., 2007). Erickson et al. (2007) found that dual-task training Osaka et al. (2003, 2004) used the reading span test (RST) to
induced performance improvements which were correlated with compare PFC activation in high-performing normal young adults
an increase in hemispheric asymmetry and a reduction in age dif- with those of low performers, and found that ACC and DLPFC
ferences in ventral and dorsal PFC activation. The investigators activation was greater in high versus low performers. The RST is
Frontiers in Human Neuroscience www.frontiersin.org February 2012 | Volume 6 | Article 24 | 113

Osaka et al. Code switching in older adults
a commonly used dual task developed to measure working mem- as the highest level at which a participant could recall all words
ory resources available for reading sentences and storing a target from the list in three of the five trials. Overall performance on the
word associated with each sentence (Daneman and Carpenter, RST was assessed in terms of the total number of words recalled.
1980; Just and Carpenter, 1992). Individual differences in work- RST scores of the participants ranged from 1.5 to 4.0.
ing memory capacity are likely derived from recruited activity Both the training and control groups were matched for age,
shared in the DLPFC and ACC (Osaka et al., 2003). In addition educational level, and RST estimates in the screening test. Pre-
to these results for young adults, our previous research in older test, training, and post-test sessions all occurred on the same day.
adults showed less ACC activation relative to younger individuals Twenty-five participants (17 males and eight females, age range =
(Otsuka et al., 2006, 2008). Based on these findings, it is plausi- 63–73 years, mean = 68.08, SD = 2.7, educational level range =
ble to assume that working memory impairments in older adults 9–16, mean = 12.0, SD = 2.3) participated in the strategy training
are due to ineffective attentional coordination, as supported by a (training group), whereas the other 25 (15 males and 10 females,
network including the ACC and DLPFC. age range = 63–75 years, mean = 68.50, SD = 3.4, educational
In the present study, we investigated the effect of attentional level range = 9–16, mean = 11.8, SD = 2.0) did not receive train-
coordination training on dual-task performance in older adults. ing (control group). The initial RST scores of both groups fell
Strategic training is considered to be an effective method for within the same range, from 1.5 to 4.0 (mean = 2.22 in the train-
increasing working memory performance (McNamara and Scott, ing group, mean = 2.19 in the control group). Informed consent
2001). For example, strategic training using visual imagery pro- was obtained from each participant, in accordance with the pro-
duces increased correct recall performance when older individ- tocol approved by the Advanced Telecommunications Research
uals perform a paired association task (Yesavage, 1983; Carretti Institute International Brain Imaging Center Review Board. All
et al., 2007). In our previous study (Osaka and Nishizaki, participants were paid a modest honorarium for their participa-
2000) we found that high-RST performers tended to use a tion in the study.
mental imagery strategy more often than low-RST perform-
ers, who appear to restrict themselves to verbal rehearsal. Based VISUAL IMAGERY TRAINING PROCEDURE
upon Baddeley’s multi-component model of working memory Specific training was provided in the form of a modified RST that
(Baddeley, 2003), we introduced for the first time a unique train- required participants to read three sentences and remember three
ing task that required participants to switch attention from verbal target words. During the training period, participants practiced
to visual images, (i.e., switching from phonological informa- imagining a target word by drawing a figure that represented the
tion to visuo-spatial information), with the intention that this word. Drawing the figures required participants to switch their
approach would strengthen the executive control function of focus of attention from verbal to visual images. For example, as
working memory in older adults. To our knowledge, there have the sample protocol in Figure 1 shows, participants were asked
been no published reports on the use of visual image forma- to draw the figure of the jet plane using a black pencil when the
tion as a training strategy to improve working memory in older target word indicated “travel.”
adults. We hypothesized that attention switching training through In the first part of the training, as Figure 1 shows, participants
the use of image formation would improve elder participants’ read aloud one sentence and then drew a figure representing the
working memory performance. target word on a piece of paper within 60 s. Although producing
To investigate the neural substrates underlying such training these drawings was initially challenging, the task became easier
effects, we measured fMRI activation while participants per- with practice. In the second part of the training, participants were
formed a working memory task (the RST) before and after the
training period. We hypothesized that this training of attention
switching from verbal to visual images activated brain regions
related to attentional control. Moreover, it is predicted that an
activation increase would be found in both the left and right
hemispheres according to the engagement of phonological and
visuo-spatial information processing simultaneously.
MATERIALS AND METHODS

PARTICIPANTS
Fifty volunteers participated in the experiment. All participants
reported that they had no history of neuropsychological or psy-
chiatric disease and all were right-handed. Their educational
levels ranged from 9 to 16 years (mean = 12.76, SD = 1.9).
Participants performed the RST in a preliminary behavioral
screening using a paper version. In the RST, participants were
required to read aloud sets of one to five sentences presented one
at a time and to remember a target word for each sentence. Each
FIGURE 1 | Examples of participant’s figure drawings (right) of the
sentence condition included five trials, and all participants com- target word (left).
pleted five total trials in the experiment. Span score was calculated

asked to form a mental image of a figure representing the target concurrently remembering three target words in each of the sen-
word, instead of drawing the image on paper. Both parts of the tences. Each sentence appeared within 6 s, and participants were
practice training consisted of 20 trials composed of three sen- required to push a button after they finished reading each sen-
tences each and took approximately one hour. Participants in the tence. After three sentences were presented for a given trial, three
control group read the same sentences and remembered the target probe stimuli appeared at 6 s intervals in the same order as the
words without being instructed to draw or imagine any figures. sentences. Each probe stimulus comprised one word. When par-
The number of trials was the same as the training group. Time ticipants identified the target in the probe array, they pushed the
was limited to 30 min in the first training session and 25 min in left key to identify a word as “in set” and the right key to identify
the second training session. The control group spent the same a word as “out of set.” Half of the probe words were targets while
amount of time with the material as the training group. None of the other half were not.
the participants in the training or control groups exceeded the Figure 2 shows an RST experimental block (modified RST
time allotted. fitted to an fMRI experiment). One experimental RST block com-
prised three sentences in the reading phase (each lasting 18 s) and
EXPERIMENTAL PROCEDURE an 18 s recognition phase. RST blocks were repeated 16 times in
Each participant was tested on the RST task during both pre- one session. A control condition lasting 18 s was inserted between
and post-training experimental sessions. The post-training RST experimental RST blocks. During the control condition, partic-
experimental session was performed after the training (control ipants pushed the right and left keys when the words “right” or
training for the control group). The post-training session began “left” appeared on the screen. Stimulus sentences ranged from 18
approximately 2 h after the end of the pre-training session for to 24 character lengths (mean = 21.5) and were presented on the
both groups. In the experimental sessions, only three sentence tri- screen within a visual angle of 45◦ , using a mirror attached to
als were performed, and participants read each sentence silently. a head coil. The target word that the participants were instructed
In the preliminary screening RST, most participants could not to memorize was underlined in each sentence. The position of the
remember three target words while they read three sentences target word in the sentence was counterbalanced across trials. RST
aloud. Therefore, we adapted a recognition method to make task blocks were repeated 16 times in post-training session, which was
performance easier. Participants were instructed to make recogni- identical to the pre-training session. The RST and control con-
tion judgments about probe stimuli (i.e., deciding whether each ditions were repeated in the same order during the post-training
stimulus was a target word or not) rather than recall the words session, but the stimulus sentences changed so that participants
orally. They were required to read three sentences silently while never read the same sentence twice.
FIGURE 2 | Experimental time course of the RST. One experimental block comprised three sentences. After three sentences were presented, three probe
stimuli appeared at 6 s intervals. Reading and recognition phases each took 18 s, followed by an 18 s baseline condition.

fMRI DATA ACQUISITION AND ANALYSIS such that a significant increase in the post-training session was
Whole brain imaging data were acquired on a 1.5-T MRI scan- found only in the training group (Tukey’s HSD post-hoc analy-
ner (Shimazu-Marconi Magnex Eclipse), using a standard head sis, p < 0.01). Mean response times for the training group were
coil. Head movements were minimized using a forehead strap. 1454 ms (SD = 418.29) in the pre-training session and 1350 ms
For functional imaging, a gradient-echo echo-planar imaging (SD = 339.38) in the post-training session. The control group
sequence with the following parameters was used: Repetition time response times were 1543 ms (SD = 482.00) in the pre-training
(TR), 2000 ms; echo time (TE), 48 ms; flip angle, 80◦ ; field of view session and 1436 ms (SD = 409.00) in the post-training session.
(FOV), 22 × 22 cm; and matrix, 64 × 64 pixels. In one experi- A Two-Way ANOVA showed a main effect of training, F(1,48) =
mental session, 376 contiguous images and 20 slices each with a 19.60, p < 0.01. There was a non-significant tendency toward
6 mm thickness were obtained in the axial plane for each par- an interaction between group and session, F(1,48) = 3.06, p <
ticipant. After image collection, T1 anatomical images using a 0.08. However, there was a significant post-training RT decrease
conventional spin echo pulse sequence (TR = 12 ms, TE = 4.5 ms, in the training group (Tukey’s HSD post-hoc analysis, p < 0.01).
flip angle = 20◦ , FOV = 25.6 × 25.6 cm, and pixel matrix = After the post-training session, all participants in the training
256 × 256) were collected for anatomical co-registration at the group reported that they had used an imagery-based strategy to
same locations as the functional images. Scanner sequences were perform the post-training task. However, in the control group,
synchronized with stimulus presentation using Presentation stim- most of the participants (20/25) used verbal rehearsal and five
ulus software (Neurobehavioral System, Inc.). Data were analyzed participants used both verbal rehearsal and visual imagery. These
using SPM2 (Wellcome Department of Cognitive Neurology, five participants did not show any performance increase.
London, UK) on Matlab (MathWorks, Sherborn, MA).
Analysis of fMRI data was initially performed for each indi- fMRI RESULTS
vidual participant for each pre- and post-training session. Six Figure 3A shows areas of increased activation after training, com-
initial images from each scanning session were discarded from pared using paired t-tests between the pre- and post-training
analysis in order to eliminate non-equilibrium effects of magneti- sessions (voxel-level threshold uncorrected for multiple compar-
zation, leaving 369 total images for analysis. All functional images isons, p < 0.0001). We applied an uncorrected criterion to focus
were realigned to correct for head movement. We selected images
with less than 1 mm movement between scans. After realignment,
the anatomical images were co-registered to the mean func-
tional images. Functional images were then normalized with the
anatomical image and spatially smoothed using a Gaussian filter
(7 mm full width-half maximum). The box-car reference func-
tion was adopted for individual analysis to identify voxels under
each task condition. Global activity for each scan was corrected
using grand mean scaling. Low-frequency noise was modeled
using hemodynamic response functions and the corresponding
derivative. Single participant data were analyzed using a fixed-
effects model, whereas group data from pre- and post-training
sessions were analyzed using a random-effects model. Extraction
of ROI data and computation of percent signal change of each
ROI was performed using MarsBaR (Brett et al., 2002). As a com-
parison between post- and pre-training sessions, we performed
a paired t-test to examine the effect of training (post-training
vs. pre-training) in each ROI. A statistical threshold of p < 0.05
was used.
RESULTS
BEHAVIORAL PERFORMANCE
Recognition accuracy and response times during the experimen-
tal RST were calculated for both pre- and post-training phases. In
the training group, the mean percentage recognition accuracies
were higher in the post- than in the pre-training session (pre-,
90.33%, SD = 11.70; post-, 97.60%, SD = 5.80). However, in the
FIGURE 3 | (A) Areas of significantly increased activation after the training,
control group, there was little apparent difference between pre- as determined by paired t-test between the pre- and post-training session.
training and post-training sessions (pre-, 90.43%, SD = 10.70; The figures on the left side show activations for the training participants,
post-, 91.20%, SD = 9.13). A Two-Way analysis of variance and those on the right show those for the control participants.
(ANOVA) of group × session showed a significant main effect (B) Significant ACC activation (upper), left IPL activation (middle), and right
IPL activation (lower) by paired t-test between the pre- and post-training
of training, F(1,48) = 6.26, p < 0.05. The interaction between
session.
group and session was also significant, F(1,48) = 9.59, p < 0.01,

upon specific regions where increased activation had previously

been reported for younger adults performing the RST (Bunge
et al., 2000; Osaka et al., 2003, 2004). The figures on the left side
show activations for training participants, and those on the right
show activations for control participants.
Figure 3B displays areas for which there was signifi-
cantly increased activation after training (voxel-level threshold
uncorrected for multiple comparisons, p < 0.0001). Figure 3B
shows significantly increased activation in the ACC (x, y, z =
−12, 22, 28, based on MNI coordinates, Z-score = 3.33,
t-value = 3.81, p < 0.0001, uncorrected), which was observed
only in the training group. Increased activation during the post-
training session was also found in the left (x, y, z = −60, −24, 48,
Z-score = 4.22, t-value = 5.22) and right inferior parietal lob-
ule (IPL) (x, y, z = 40, −32, 68, Z-score = 3.49, t-value = 4.03).
An activation increase was also found in the right superior pari-
etal lobule (SPL) (x, y, z = 30, −66, 56, Z-score = 3.14, t-value =
3.53; not shown in the Figure 3). In the control participants, we
could not find any areas showing significant differences between
pre- and post-training sessions. Furthermore, we could not find
any areas for which there was significantly increased activation
after training in the recognition phase both for experimental and
control groups.
SIGNAL INTENSITY
The ACC, bilateral DLPFC, bilateral IPL, and bilateral SPL were
specified as the regions of interest (ROI) for detecting signal
changes. Following the identification of activated areas, percent
signal changes in each ROI were obtained for both training FIGURE 4 | Mean percent signal changes and SDs under training and
and control groups. Time-course data were obtained for the control groups in each region of interest (ACC, bilateral DLPFC,
local-maxima of each ROI for each participant. Mean percent bilateral IPL, bilateral SPL) during pre- and post-training RST
(∗ p < 0.05).
signal change was calculated by subtracting the mean blood
oxygenation-level-dependent (BOLD) signal at baseline from sig-
nals observed during the pre- and post-training sessions.
Figure 4 shows mean signal changes in each ROI during the DISCUSSION
pre- and post-training sessions in the training and control groups. INCREASING ACC RECRUITMENT
Two-Way ANOVAs of group × session were performed for the In the present experiment, we found relatively scarce ACC
ROIs. Analyses were performed using the STATISTICA software activation during the pre-training RST, and these results were
package (StatSoft). A significant interaction between session and in accordance with previous findings in older adults (Otsuka
group was found for the ACC region, F(1,48) = 6.93, p < 0.01. et al., 2006). Interestingly enough, ACC activation significantly
Training effects were found in the training group (Tukey’s HSD increased in the post-training RST, likely due to the effects
post-hoc analysis, p < 0.05). A significant main effect of session of training. The ACC has an attention coordination function,
[F(1,48) = 5.09, p < 0.05] and an interaction between session including inhibiting attention directed toward inappropriate
and group [F(1,48) = 5.80, p < 0.05] were also found for the stimuli, whereas the DLPFC more generally supports sustain-
left and right IPL. For the right IPL, training effects in sig- ing attention on task goals (MacDonald et al., 2000). Increased
nal intensity increases were found in the training group (by ACC activity is also reported to occur with errors on high-
Tukey’s HSD post-hoc analysis, p < 0.05) but not the control conflict trials such as go/no-go, oddball, and two-alternative
group. Similarly, training effects were also found in the left IPL forced-choice selections (Bush et al., 1998; Carter et al., 1998;
in the training group (p < 0.05) but not the control group. A Braver et al., 2001), suggesting that the ACC is active in conflict
significant main effect of session [F(1,48) = 7.04, p < 0.05] and monitoring.
an interaction between session and group [F(1,48) = 4.80, p < During the encoding phase of the RST in the scanner, par-
0.05] were also found for the right SPL, with the signal change ticipants needed to focus attention on the target words while
actually increasing after training (Tukey’s HSD post-hoc analy- inhibiting other irrelevant words in the sentence. Because the
sis, p < 0.05). However, training effects in the recognition phase ACC potentially plays a role in perceiving conflicts between
were not significant in these three regions for either the train- target and non-target words during the dual task, ACC impair-
ing group [F(1,48) = 1.13 ∼ 2.23, p = 0.26 ∼ p = 0.13] or the ment is associated with some deficits in exact executive control.
control group [F(1,48) = 0.81 ∼ 2.24, p = 0.37 ∼ p = 0.14]. Our participants likely did not have sufficient ACC activation in

the pre-training session, leading to difficulties perceiving conflict Right IPL and SPL activation is thought to be involved in the
between target and irrelevant words even if there was suffi- basic attentional process of executive function. A recent study
cient DLPFC activation to sustain attention to the target words. demonstrated that right IPL activity is associated with disengage-
Therefore, without perceiving conflict during the encoding phase, ment and reorienting of attention to a relevant word presented
participants face difficulty detecting differences between the tar- outside the current focus of attention, whereas the intraparietal
get and non-target word during the recognition phase. sulcus (IPS) contributes to an attentional set for the task goal
However, while participants performed the training task, they (Corbetta et al., 2008). Furthermore, the right IPL (along with
had to read one sentence at a time and then draw a figure repre- the SPL) likely plays a role in attentional filtering, which is a pro-
senting the target word for each sentence. While they were draw- cess whereby irrelevant information is screened out and relevant
ing the figures (see Figure 1), participants could learn to switch information prioritized (Minamoto et al., 2010).
attention from the target to its associated visual image code. Osaka et al. (2007) showed that increased left SPL activation
Following such attention switching training (i.e., transferring coincided with attentional demands while performing a difficult
from verbal to visual imagery), participant’s likely experienced RST requiring switching attention. While participants performed
increased ACC activity as a function of increased attentional coor- the difficult RST, the SPL appeared to play a role in reorient-
dination. After the training, participants could more easily attend ing attention and driving cooperative activity between the ACC
to targets and have more working memory resources available and DLPFC. The SPL (including the lateral IPS area) is gen-
for both reading sentences and maintaining target words. With erally related to attentional processes (Culham and Kanwisher,
sufficient working memory resources, our participants could 2001) and is specifically responsible for visuo-spatial attention
more easily assign the limited resources available for storage and and working memory (Wager and Smith, 2003). Activation in the
processing during the RST. posterior region supports the limited capacity of working mem-
Training appeared to promote ACC activation in the present ory (Linden, 2007). Thus, the SPL may potentially play a shared
study. With adequate coordination of attention, participants did role in focusing attention along with DLPFC and ACC (Osaka and
not face a difficult conflict between the target and irrelevant filler Osaka, 2007).
words when they were required to recognize targets. Behavioral In the present study, bilateral activation was found in the IPL,
data also indicated better performance compared with perfor- which is often regarded as a mechanism that compensates for
mance during a pre-training RST. Moreover, response times declining functional activity while performing working memory
decreased, which is seemingly indicative of increased confidence tasks (Cabeza et al., 2002; Reuter-Lorenz, 2002; Wager and Smith,
in correct responses after training. 2003). It has also been suggested that reduced asymmetry in
the PFC is compensatory when the homologous region can sub-
INCREASED ACTIVATION IN THE RIGHT IPL AND SPL serve a complementary role in task performance (Colcombe et al.,
The present fMRI data showed that right IPL and SPL activa- 2005). When the attentional control system weakens, another
tion was more apparent during the post-training RST compared system develops to compensate for any functional impairment.
to the pre-training RST. Retention of verbal information in the Bilateral IPL activation observed in the current study may serve
phonological loop itself activates the left IPL, while spatial infor- as a compensatory system for impairments in working memory.
mation activates the right homologues (Smith et al., 1996). In In accord with previous suggestions (Wager and Smith, 2003;
post-training, all participants in the training group reported per- Colcombe et al., 2005), the observed right IPL activation possibly
forming the task using verbal to visual code switching strategies. A compensated for the declines in executive functioning with the
similar result was reported by Carretti et al. (2007), who showed help of the visuo-spatial inner scribe. This suggestion indicates
that strategic training using imagery to improve the memoriza- that when one store becomes vulnerable to age-related decline,
tion of word lists improved working memory performance in use of an alternative store, such as the visuo-spatial sketchpad,
young and older individuals. According to Logie (2003), the is a potentially reliable method to improve task performance in
visuo-spatial sketchpad can be subdivided into a visual cache, older adults. In the present experiment, we did not control the
involved in storing visual images, and an inner scribe, involved strategy use of the non-training group. If participants in the con-
in the retention of spatial information via active rehearsal (i.e., by trol group had adopted a visual imagery strategy, they might
drawing in the current study). The increased right IPL activation have shown a performance increase in behavioral and neural
observed during the post-training RST may have been evoked effects without training. Further investigation is needed to clar-
by a visuo-spatial inner scribe that was used to sustain images ify the effects of strategy on working memory performance when
of the target words (Logie and Pearson, 1997). Using the visuo- only imagery instruction is provided. Moreover, the long-term
spatial inner scribe, participants can share the working memory effects of visual strategy training on working memory perfor-
load across two subsystems, providing them with a more efficient mance should be investigated in elderly populations. In addition,
attentional coordination system. Grady et al. (2006) suggested that there is a gradual, age-related
It is interesting that providing a visually based imagery strat- reduction in the ability to suspend non-task-related or default-
egy improved performance on a verbal task in the present study. mode activity. Further study will be required to test default-mode
These findings suggest the intriguing possibility that when a ver- activity in older individuals.
bal store within a multi-component model shows age-related In sum, we found that training in a specific verbal to visual
performance declines, an alternative visual store can possibly code switching strategy improved complex working memory task
serve to compensate for the deficits. performance in older adults. After training, fMRI data showed

an increase in ACC, bilateral IPL, and right SPL activation in the switching, could be one potentially effective approach for coping
encoding phase. with working memory deficits in aging.
Based on these findings, we suggest that age-related impair-
ment in working memory performance is partially caused by ACKNOWLEDGMENTS
ineffective attention coordination, as supported by the ACC This work was supported by grants from the Japan Society for the
in connection with the IPL and SPL. Training in dual task Promotion of Science (#23240036) and (#22220003) to Mariko
paradigms, which require compensatory strategies such as code Osaka and Naoyuki Osaka, respectively.
REFERENCES training improve the working mem- D’Esposito, M., Postle, B. R., Provenzale, J. M. (2004). Diffusion
Baddeley, A. D. (1986). Working ory performance of young and older Ballard, D., and Lease, J. (1999). tensor imaging of adult age differ-
Memory. Oxford: Oxford University adults? Exp. Psychol. 54, 311–320. Maintenance versus manipulation ences in cerebral white matter: rela-
Press. Carter, C. S., Braver, T. S., Barch, D. of information held in working tion to response time. Neuroimage
Baddeley, A. D. (2003). Working mem- M., Botvinick, M. M., Noll, D., and memory: an event-related fMRI 21, 1174–1181.
ory: looking back and looking Cohen, J. D. (1998). Anterior cingu- study. Brain Cogn. 41, 66–86. McNamara, D. S., and Scott, J. L.
forward. Nat. Rev. Neurosci. 4, late cortex, error detection, and the Engle, R. W., Tuholski, S. W., Laughlin, (2001). Working memory capacity
829–839. online monitoring of performance. J. E., and Conway, A. R. (1999). and strategy use. Mem. Cognit. 29,
Baddeley, A. D., and Logie, R. H. (1999). Science 280, 747–749. Working memory, short-term 10–17.
“Working memory: the multiple Cohen, J. D., Perlstein, W. M., Braver, memory, and general fluid intelli- Minamoto, T., Osaka, M., and Osaka,
component model,” in Models of T. S., Nystrom, L. E., Noll, D. C., gence: a latent-variable approach. N. (2010). Individual differences in
Working Memory: Mechanisms of Jonides, J., and Smith, E. E. (1997). J. Exp. Psychol. Gen. 128, 309–331. working memory capacity and dis-
Active Maintenance and Executive Temporal dynamics of brain activa- Erickson, K. I., Colcombe, S. J., tractor processing: possible contri-
Control, eds A. Miyake and P. tion during a working memory task. Wadhwa, R., Bherer, L., Peterson, bution of top-down inhibitory con-
Shah (Cambridge, MA: Cambridge Nature 386, 604–608. M. S., Scalf, P. E., Kim, J. S., trol. Brain Res. 1335, 63–73.
University Press), 28–61. Colcombe, S. J., Kramer, A. F., Erickson, Alvarado, M., and Kramer, A. F. Nyberg, L., Sandblom, J., Jones, S.,
Braver, T. S., Barch, D. M., Gray, J. K. I., and Scalf, P. (2005). The (2007). Training-induced plasticity Neely, A. S., Petersson, K. M.,
R., Molfese, D. L., and Snyder, implications of cortical recruitment in older adults: effects of training on Ingvar, M., and Backman, L. (2003).
A. (2001). Anterior cingulate and brain morphology for individ- hemispheric asymmetry. Neurobiol. Neural correlates of training-related
cortex and response conflict: ual differences in inhibitory func- Aging 28, 272–283. memory improvement in adulthood
effects of frequency, inhibition tion in aging humans. Psychol. Aging Grady, C. L., Springer, M. V., and aging. Proc. Natl. Acad. Sci.
and errors. Cereb. Cortex 11, 20, 363–375. Hongwanishkul, D., Mcintosh, U.S.A. 100, 13728–13733.
825–836. Corbetta, M., Patel, G., and Shulman, A. R., and Winocur, G. (2006). Age- Osaka, M., Komori, M., Morishita,
Brett, M., Anton, J. L., Valabregue, R., G. L. (2008). The reorienting system related changes in brain activity M., and Osaka, N. (2007). Neural
and Poline, J. B. (2002). “Region of the human brain: from environ- across the adult lifespan. J. Cogn. bases of focusing attention in
of interest analysis using an SPM ment to theory of mind. Neuron 58, Neurosci. 18, 227–241. working memory: an fMRI study
toolbox,” The 8th International 306–324. Just, M. A., and Carpenter, P. A. based on group differences.
Conference on Functional Mapping Craik, F. I. M., and Salthouse, T. (1992). A capacity theory of com- Cogn. Affect. Behav. Neurosci. 7,
of the Human Brain. June 2–6, A. (eds). (2000). The Handbook of prehension: individual differences 130–139.
Sendai, Japan. Available on Aging and Cognition. New Jersey: in working memory. Psychol. Rev. Osaka, M., and Nishizaki, Y. (2000).
CD-ROM in Neuroimage 16, Lawrence Erlbaum. 99, 122–149. “How working memory works in
Abstract 974. Culham, J. C., and Kanwisher, N. G. Li, S. C., Schmiedek, F., Huxhold, the central executive,” in The Brain
Bunge, S. A., Klingberg, T., Jacobsen, (2001). Neuroimaging of cognitive O., Rocke, C., Smith, J., and and Working Memory, ed N. Osaka
R. B., and Gabrieli, J. D. (2000). functions in human parietal cortex. Lindenberger, U. (2008). Working (Kyoto: Kyoto University Press),
A resource model of the neural Curr. Opin. Neurobiol. 11, 157–163. memory plasticity in old age: prac- 203–223.
basis of executive working mem- Dahlin, E., Neely, A. S., Larsson, tice gain, transfer, and maintenance. Osaka, M., and Osaka, N. (2007).
ory. Proc. Natl. Acad. Sci. U.S.A. 97, A., Backman, L., and Nyberg, L. Psychol. Aging 23, 731–742. “Neural bases of focusing attention
3573–3578. (2008a). Transfer of learning after Linden, D. E. (2007). The working in working memory: an fMRI study
Bush, G., Whalen, P. J., Rosen, B. R., updating training mediated by the memory networks of the human based on individual differences,”
Jenike, M. A., Mcinerney, S. C., and striatum. Science 320, 1510–1512. brain. Neuroscientist 13, 257–267. in The Cognitive Neuroscience of
Rauch, S. L. (1998). The counting Dahlin, E., Nyberg, L., Backman, L., Logie, R. H. (2003). Spatial and Working Memory, eds N. Osaka,
Stroop: an interference task special- and Neely, A. S. (2008b). Plasticity visual working memory: a mental R. H. Logie, and M. D’Esposito
ized for functional neuroimaging– of executive functioning in young workspace. Psychol. Learn. Motiv. (Oxford: Oxford University Press),
validation study with functional and older adults: immediate train- 42, 37–78. 99–118.
MRI. Hum. Brain Mapp. 6, ing gains, transfer, and long-term Logie, R. H., and Pearson, D. G. (1997). Osaka, M., Osaka, N., Kondo, H.,
270–282. maintenance. Psychol. Aging 23, The inner eye and the inner scribe Morishita, M., Fukuyama, H., Aso,
Cabeza, R. (2001). Cognitive neuro- 720–730. of visuo-spatial working memory: T., and Shibasaki, H. (2003). The
science of aging: contributions of Daneman, M., and Carpenter, P. A. evidence from developmental frac- neural basis of individual differ-
functional neuroimaging. Scand. J. (1980). Individual differences in tionation. Eur. J. Cogn. Psychol. 9, ences in working memory capac-
Psychol. 42, 277–286. working memory and reading. 241–257. ity: an fMRI study. Neuroimage 18,
Cabeza, R., Anderson, N. D., Locantore, J. Verb. Learn. Verb. Behav. 19, MacDonald, A. W. 3rd, Cohen, J. D., 789–797.
J. K., and Mcintosh, A. R. (2002). 450–466. Stenger, V. A., and Carter, C. S. Osaka, N., Osaka, M., Kondo, H.,
Aging gracefully: compensatory D’Esposito, M., Detre, J. A., Alsop, (2000). Dissociating the role of the Morishita, M., Fukuyama, H., and
brain activity in high-performing D. C., Shin, R. K., Atlas, S., and dorsolateral prefrontal and anterior Shibasaki, H. (2004). The neu-
older adults. Neuroimage 17, Grossman, M. (1995). The neural cingulate cortex in cognitive con- ral basis of executive function in
1394–1402. basis of the central executive system trol. Science 288, 1835–1838. working memory: an fMRI study
Carretti, B., Borella, E., and De Beni, R. of working memory. Nature 378, Madden, D. J., Whiting, W. L., Huettel, based on individual differences.
D. (2007). Does strategic memory 279–281. S. A., White, L. E., Macfall, J. R., and Neuroimage 21, 623–631.

Otsuka, Y., Osaka, N., Morishita, M., word processing. Nature 331, Salthouse, T. A. (1996). The processing- Conflict of Interest Statement: The
Kondo, H., and Osaka, M. (2006). 585–589. speed theory of adult age differ- authors declare that the research
Decreased activation of anterior Posner, M. I., and Petersen, S. E. ences in cognition. Psychol. Rev. 103, was conducted in the absence of any
cingulate cortex in the working (1990). The attention system of the 403–428. commercial or financial relationships
memory of the elderly. Neuroreport human brain. Annu. Rev. Neurosci. Smith, E. E., and Jonides, J. (1999). that could be construed as a potential
17, 1479–1482. 13, 25–42. Storage and executive processes conflict of interest.
Otsuka, Y., Osaka, N., and Osaka, M. Raz, N. (2000). “Aging of the brain in the frontal lobes. Science 283,
(2008). Functional asymmetry of and its impact on cognitive perfor- 1657–1661. Received: 26 October 2011; accepted:
superior parietal lobule for working mance: integration of structural and Smith, E. E., Jonides, J., and Koeppe, R. 05 February 2012; published online: 20
memory in the elderly. Neuroreport functional findings,” in The Hand- A. (1996). Dissociating verbal and February 2012.
19, 1355–1359. book of Aging and Cognition, eds F. I. spatial working memory using PET. Citation: Osaka M, Otsuka Y and Osaka
Owen, A. M., Stern, C. E., Look, M. Craik and T. A. Salthouse (New Cereb. Cortex 6, 11–20. N (2012) Verbal to visual code switch-
R. B., Tracey, I., Rosen, B. R., Jersey: Lawrence Erlbaum), 1–90. Wager, T. D., and Smith, E. E. (2003). ing improves working memory in older
and Petrides, M. (1998). Functional Raz, N., Gunning, F. M., Head, D., Neuroimaging studies of work- adults: an fMRI study. Front. Hum.
organization of spatial and non- Dupuis, J. H., Mcquain, J., Briggs, ing memory: a meta-analysis. Neurosci. 6:24. doi: 10.3389/fnhum.
spatial working memory process- S. D., Loken, W. J., Thornton, A. Cogn. Affect. Behav. Neurosci. 3, 2012.00024
ing within the human lateral frontal E., and Acker, J. D. (1997). Selective 255–274. Copyright © 2012 Osaka, Otsuka and
cortex. Proc. Natl. Acad. Sci. U.S.A. aging of the human cerebral cortex West, R. L. (1996). An application of Osaka. This is an open-access article dis-
95, 7721–7726. observed in vivo: differential vulner- prefrontal cortex function theory to tributed under the terms of the Creative
Petersen, S. E., Fox, P. T., Posner, ability of the prefrontal gray matter. cognitive aging. Psychol. Bull. 120, Commons Attribution Non Commercial
M. I., Mintun, M., and Raichle, Cereb. Cortex 7, 268–282. 272–292. License, which permits non-commercial
M. E. (1988). Positron emis- Reuter-Lorenz, P. (2002). New visions Yesavage, J. A. (1983). Imagery pre- use, distribution, and reproduction in
sion tomographic studies of of the aging mind and brain. Trends training and memory training in the other forums, provided the original
the cortical anatomy of single- Cogn. Sci. 6, 394. elderly. Gerontology 29, 271–275. authors and source are credited.

published: 03 December 2012
doi: 10.3389/fpsyg.2012.00525
Mental imagery for musical changes in loudness

Freya Bailes*, Laura Bishop, Catherine J. Stevens and Roger T. Dean
MARCS Institute, University of Western Sydney, Sydney, NSW, Australia
Edited by: Musicians imagine music during mental rehearsal, when reading from a score, and while
composing. An important characteristic of music is its temporality. Among the parame-
ters that vary through time is sound intensity, perceived as patterns of loudness. Studies
Reviewed by:
Holly Bridge, University of Oxford, UK of mental imagery for melodies (i.e., pitch and rhythm) show interference from concur-
Joel Pearson, The University of New rent musical pitch and verbal tasks, but how we represent musical changes in loudness
South Wales, Australia is unclear. Theories suggest that our perceptions of loudness change relate to our percep-
*Correspondence: tions of force or effort, implying a motor representation. An experiment was conducted
Freya Bailes, Department of Drama
to investigate the modalities that contribute to imagery for loudness change. Musicians
and Music, University of Hull,
Cottingham Road, Hull HU6 7RX, UK. performed a within-subjects loudness change recall task, comprising 48 trials. First, par-
e-mail: f.bailes@hull.ac.uk ticipants heard a musical scale played with varying patterns of loudness, which they were
asked to remember. There followed an empty interval of 8 s (nil distractor control), or the
presentation of a series of four sine tones, or four visual letters or three conductor gestures,
also to be remembered. Participants then saw an unfolding score of the notes of the scale,
during which they were to imagine the corresponding scale in their mind while adjusting
a slider to indicate the imagined changes in loudness. Finally, participants performed a
recognition task of the tone, letter, or gesture sequence. Based on the motor hypothesis,
we predicted that observing and remembering conductor gestures would impair loudness
change scale recall, while observing and remembering tone or letter string stimuli would
not. Results support this prediction, with loudness change recalled less accurately in the
gestures condition than in the control condition. An effect of musical training suggests that
auditory and motor imagery ability may be closely related to domain expertise.
Keywords: mental imagery, loudness, music, motor processing, melody, working memory
INTRODUCTION and listener, an appreciation of the Effort (E) involved in this

Musicians imagine music during mental rehearsal (Holmes, 2005), contributes to the perception of Loudness (L) and Arousal (A;
when reading from a score (Brodsky et al., 2003), and while Dean and Bailes, 2008, 2010). This proposal (FEELA) was based
composing (Covington, 2005; Bailes, 2009; Bailes and Bishop, on computational analyses of patterns of acoustic intensity, and
2012). An important characteristic of music is its temporality, and it closely relates to ideas of Todd (1992). Some empirical evi-
among the parameters that vary through time is sound intensity, dence to corroborate this theory is provided by Eitan and Granot
perceived as patterns of loudness. Desired increases in loudness (2006). In an experiment in which participants were asked to imag-
can be notated in a score as “crescendi,” while decreases can be ine a figure moving to musical stimuli, an association was found
notated as “decrescendi.” There is anecdotal evidence that imag- between crescendi (increase in sound intensity) and the energy of
ined music can vary in its overall “loudness” level (Trusheim, 1991; the movements of the imagined figure. For example, figures were
Sacks, 2007), but empirical evidence of imagery for loudness is imagined moving from a walk to a run as the loudness increased.
inconclusive (Intons-Peterson, 1992; Pitt and Crowder, 1992; Wu Following from this theoretical and empirical work, we hypothe-
et al., 2010). Moreover, imagery for changes in loudness remains a size that motor representations contribute to imagery for musical
relatively neglected topic. changes in loudness.
Mental representations of pitch and melody have been shown to The role of motor representations in auditory imagery is
involve auditory (Deutsch, 1970; Keller et al., 1995), verbal (Keller critical to distinctions that have been made between an “inner
et al., 1995), and motor processing (Mikumo, 1994; Finney and ear” (acoustic imagery) and an “inner voice” (subvocal rehearsal)
Palmer, 2003). Yet, how we represent musical changes in loudness (Smith et al., 1992). Both have been associated with the working
is unclear. In the current study, the modalities that contribute to memory subsystem known as the “phonological loop,” involved
imagery for loudness change were investigated. Theories suggest in processing and memorizing verbal material (Kalakoski, 2001).
that our perceptions of loudness change relate to our perceptions Articulatory suppression has a negative impact on performance in
of force or effort, implying that a motor representation is involved. tasks likely to elicit auditory mental rehearsal, suggesting that sub-
For example, we have proposed a causal chain whereby the literal vocalization can contribute to the generation of auditory imagery
Force (F) with which a player activates an instrument is trans- (Smith et al., 1995). Brodsky et al. (2003) used a musical task that
mitted as the physical Energy (E) of the sound. For both player is commonly performed by musicians, namely reading musical
www.frontiersin.org December 2012 | Volume 3 | Article 525 | 121

Bailes et al. Mental imagery for loudness change
notation in silence, to examine the impact of concurrently per- (rather than recognition) task. Accordingly, we used a continuous
forming auditory versus phonatory interference tasks on the suc- response paradigm that encouraged participants to imagine loud-
cessful imagining of a notated melody. Phonatory interference by ness change stimuli. Participants moved a volume slider to indicate
way of concurrently singing or humming a different melody to increases and decreases in the “loudness” of the imagined stimuli.
that in the score proved the most distracting, pointing to acoustic The advantages of such an approach are twofold. First, enacting
imagery, and subvocal rehearsal in imagery for melodies. How- the response in time is more likely to recruit a mental image of
ever, this research is limited in its focus on imagery for sequences the stimulus than performing a stimulus recognition task. Sec-
of discrete events (notes), rather than on the dynamic properties of ond, movement is integral to this response mode, respecting our
music. It is also shaped by the original concern of working memory hypothesized link between representations of intensity and motor
models to describe verbal processing, with motor considerations in effort.
auditory imagery restricted to vocal production (see also Hubbard, The principal hypothesis was that imagining changes in loud-
2010). ness would be disrupted by concurrently remembering movement
In the current study, an experiment was conducted to investi- sequences (presented visually). However, the loudness change
gate the modalities that contribute to imagery for loudness change. stimuli in the current experiment comprised loudness changes
Investigations of visual imagery and working memory have used in the sounding of ascending and descending scales, and pitch
an interference paradigm as the means to disrupt different types of (the patterns of note ascent and descent) is integral to a repre-
processing. For example, in a study of movement imagery in rock sentation of such stimuli, so it was possible that tone sequences
climbing, Smyth and Waller (1998) trained participants on two would also impair mental imagery. Finally, if participants chose
routes, one vertical and the other horizontal. After training, partic- a strategy of labeling increases and decreases of intensity as “up”
ipants imagined climbing the routes under control conditions and and “down” respectively in order to remember the loudness change
with one of three secondary tasks – dynamic visual noise, spatial scale stimuli, then a concurrent verbal task of remembering letters
tapping, or kinesthetic suppression. The secondary or interference could be expected to interfere with the task of recreating the loud-
tasks affected differentially the duration of horizontal and verti- ness change stimuli, perhaps suggestive of a verbal representation
cal routes leading the authors to conclude that there are multiple rather than a mental image. In line with previous research (e.g.,
and complex forms of processing action and imaging movement. Williamson et al., 2010), we describe letters as verbal stimuli due
An investigation of memory span for ballet movements by pro- to their possible encoding in word form.
fessional dancers showed no effect of dynamic visual noise as a In experiments on working memory for actions that use an
form of visual suppression on span and, by contrast, a significant interference paradigm (e.g., Smyth and Pendleton, 1989) there is a
effect of a motor interference task on span (Rossi-Arnaud et al., problem, rarely discussed, of the similarity of intervening material
2004). Pearson et al. (2008) manipulated background luminance with to-be-remembered material. For example, greater interfer-
during an imagery or a feature-based attention task. Differential ence for recalling configurational movements of the body has been
effects of background luminance on the two tasks were used to observed when intervening material consists of configurational
distinguish effects attributable to imagery from those attributable movements of the body than spatial locations. The conclusion is
to task instructions (Experiment 4). Following the tradition of then drawn that configurational movements are coded in work-
an interference paradigm to probe working memory processes, ing memory by a spatial plus kinesthetic system. However, there
we devised an interference paradigm in which a trial comprises is also much greater similarity between the intervening and the
two interleaved memory tasks, designed to test the interference of to-be-remembered material in the configurational interference
remembering material from one on the other. Rehearsal is gener- condition than in the spatial interference condition. This problem
ally required for maintenance of material in short-term memory of similarity between to-be-remembered and interference material
(Berz, 1995), and the rationale of the current experiment is that is addressed in the present experiment by having distractor stimuli
such rehearsal will be variously disturbed by material of different that are all dissimilar from the to-be-remembered loudness change
modalities. Distractor tasks were designed to differentially place scale material.
loads on verbal, auditory, and visuo-motor processing. The design Conductor gestures were used as visuo-spatial distractor stim-
requires the concurrent rehearsal of unfamiliar musical (melodies uli of relevance to the communication and understanding of
and loudness change scales) and distractor stimuli in working musical intensity, and to represent a motor sequence. Action-
memory. observation theories would suggest that observing a sequence of
One of the challenges presented by ubiquitous real-world stim- conductor gestures necessarily activates motor representations.
uli such as music is that it is time-varying. Prior studies of mental Simulation theory (see Berthoz, 1996; Grush, 2004) also argues
imagery have investigated more static material such as pictures, that we observe and understand the actions of others by covertly
objects, or alphanumeric characters. Thus there is a need to eval- simulating them. Accordingly, observing a visual sequence of
uate contemporary accounts of imagery in the context of sequen- movements with a view to recalling them would involve simulating
tial and temporally structured and varied material. In turn, this their production.
requires the development of new methods of: stimulus presenta- It also was important in the current study to separately deter-
tion, on-line generative responding, and analysis of the resulting mine whether these auditory, verbal, and motor distractor tasks
production (time-series) data. In short, investigation of imagery in would impact on imagery for melodic material, as suggested by past
music demands a method of responding that captures its temporal research. We hypothesized that imagining melodies would be dis-
unfolding, and this may be best achieved by way of a production rupted by concurrently remembering tone sequences (presented
Frontiers in Psychology | Perception Science December 2012 | Volume 3 | Article 525 | 122
aurally). However, we expected that imagining melody would produced. Eight loudness patterns were superimposed on an
involve motor processes too, such that having to remember a ascending/descending (in pitch) one octave major scale, while
sequence of movements while performing a test of imagery for eight were superimposed on a descending/ascending (in pitch)
melody would also interfere. As when imagining changes in loud- scale. The audio files were generated and recorded through the
ness, remembering visually presented letters while attempting Disklavier, controlled by Max/MSP. Each note in the scale was
to imagine melody could interfere if the letters were encoded 500 ms, so that all the scales spanned 8 s. Half of each scale
sonically rather than visually. type (e.g., ascending/descending) began with a crescendo, while
the other half began with a decrescendo. Each stimulus com-
MATERIALS AND METHODS prised between two and four loudness changes (crescendo or
A within-subjects design comprised two different imagery tasks decrescendo), lasting between three and eight notes each. Loud-
(melodies, loudness change scales), each with four different dis- ness changes were implemented by manipulating the MIDI signal
tractor conditions (control, letter sequence, tone sequence, move- sent from Max/MSP to the Disklavier for the velocity at which each
ment sequence), generating eight different experiment conditions. note should be played. The minimum and maximum note veloc-
ities were the same for all loudness changes (MIDI note velocity
PARTICIPANTS
range from 20 to 60). No more than two consecutive notes shared
Participants (N = 32, 17 female, 15 male) able to read musical
the same note velocity.
notation were recruited from universities and community music
For use in the test phase, visual scores of the scales, without
societies in greater Sydney. They received a small travel reim-
loudness change markings, were written in Sibelius. Powerpoint
bursement (15 AU$). Ages ranged from 22 to 71 years (M = 41.6,
and the screen capture software Capture Me were then used to
SD = 16). Participants had a mean Ollen Musical Sophistication
record videos of each scale being gradually revealed at the rate of
Index (OMSI; Ollen, 2006) of 595 (range 119–993, where a score
one note per 500 ms.
>500 classifies the participant as “more musically sophisticated”
and a score <500 as “less musically sophisticated”), with a mean
Distractor stimuli
of 8.7 years of musical training (range 1–16).
Letter sequence. The same total set of six letters as used by
STIMULI Williamson et al. (2010) was used to construct four-letter visual
Melodies sequences. Three letters from this set rhyme (B, D, G) and were
For the melody imagery task, 28 melodies were selected from expected to be easily confused in phonological memory, while
the Australian Music Examinations Board (AMEB) aural test syl- three do not (M, Q, R). “Different” trials at test replaced a letter
labus for grades 2–3 (AMEB, 2002). In the current experiment, from the presentation sequence with either a rhyming letter (half)
these melodies were designed to be retained in memory and to be or a non-rhyming letter (half). Letter sequences were created in
related to a visual score. The melodies were monophonic, between Powerpoint and recorded as videos using Capture Me. Each letter
eight and 12 notes in length, and written in a variety of different remained on the screen for 2 s with no gap in-between.
major and minor keys. All melodies ended on the tonic of the key.
The audio files were generated and recorded through a Yamaha Tone sequence. Four-tone sequences were generated in Audacity.
Disklavier 3 MIDI (Musical Instrument Digital Interfance) grand Pure sine tones were used, each being 2 s long, presented sequen-
piano, controlled by Max/MSP. The velocity of each note was held tially with no gap in-between. For each trial, the four tones were
constant, and each melody was made to span 8 s. Visual scores of selected from outside the key1 of the corresponding melody or
the melodies were written in Sibelius. loudness change scale stimulus. A set of possible tone sequences
Half of the melodies were altered to produce the “different” was constructed for each key prior to the experiment, from which
test stimuli, while half were unaltered for the “same” test stimuli. one tone sequence was randomly selected once a melody or scale
Three types of alteration were made: (1) the order of two con- from the corresponding key was presented. “Different” trials at test
secutive pitches was reversed, as in the “Exchange” comparison of replaced a tone from the presentation sequence with a tone also
Mikumo (1994; four melodies), (2) a “step” was exchanged for a from outside the key of the melody or scale stimulus.
“leap” (four melodies), or (3) a “leap” was exchanged for a “step”
(four melodies). As in Dowling (1978), a step was defined as an Movement sequence. A set of 10 clips of musical conducting
interval of three semitones or fewer, and a leap was an interval of were selected from the videos provided in“Expressive Conducting”
four semitones or greater. Within the constraints of each alteration (Wiens, 2002). The clips were selected to represent varied conduct-
type, changes were designed to be visually non-obvious (e.g., no ing gestures that ranged from slow to fast, and from small to large.
new accidentals or repeated notes) and avoid introducing disso- No attempt was made to control the relationship between gestures
nance into the melodic context. Contour was disrupted for five of and the pitch or loudness content of melodic and loudness change
the melodies. Alterations occurred evenly across beginning, mid- scale stimuli. While different gestures might be associated with
dle, and end locations of the melody, but never occurred on the the communication of different levels of musical sound intensity,
first or last notes.
1A musical key is described by the scale to which most of the notes in a piece of
Loudness change scales music conform. For example, if most of the notes in a piece are in the scale of
For the loudness change imagery task, 16 different loudness change C major, with important pitches such as “C” (tonic) or “G” (dominant) occurring
patterns comprising sequences of crescendi and decrescendi were particularly often, its key is probably C major.

the subjective nature of this was beyond the scope of the present began by filling out the OMSI questionnaire. Trials for each of the
study, and so beyond choosing varied gestures, no attempt was eight experiment conditions (two imagery tasks × four distractor
made to control for level of expressed intensity in their selection. tasks) were blocked, and instructions specific to the condition were
The movements were recorded from the back right of the conduc- provided at the start of that block. Participants then performed a
tor such that the face was not visible, and the white baton could practice trial for the block, and were given an opportunity to ask
be seen against the black background. The baton was visible at all the experimenter any questions that they had before proceeding to
times, and the left hand could not be seen. Original clips that were the experiment trials. Presentation of the eight blocks was random.
shorter than 2 s (the shortest original clip was 1.8 s) were stretched Each of the 24 melody stimuli was presented once without repeat
in Adobe Premiere Pro CS4 to bring them to the requisite length. across melody imagery blocks. Each of the 14 loudness change
Movement sequences were constructed in iMovie. They comprised scale stimuli was presented once or twice across loudness change
three silent clips with a 1-s blank (black) screen in-between each imagery blocks (the Max/MSP program randomly selected with-
clip. For half of the sequences, one of the three clips was replaced out replacement all 14 stimuli, then began the process again until
by another clip to create a corresponding “different” sequence. 10 of the list had been presented a second time).
While letter and tone distractor sequences comprised four distinct
events, pilot testing of gesture sequences suggested no difference in Imagery for melodies
recognition accuracy between three and four gestures. However, in In a melody trial, a melody was sounded, followed by presenta-
the pilot, participants appeared to be discouraged by the difficulty tion of the distractor stimulus (letter sequence, tone sequence,
of remembering a longer sequence. This is comparable to obser- movement sequence, or control period of 8 s). Immediately after
vations from research in working memory for dance movements. the distractor stimulus presentation, a visual score of the melody
Experiments on working memory span for body actions typically appeared on screen, and participants indicated as quickly as pos-
report a mean span of three actions for adult participants (e.g., sible whether the score was the same or different to the melody
Wood, 2007; Wachowicz et al., 2011). that they had heard, by comparing their mental image of the
melody with the score. “Same” and “Different” buttons appeared
APPARATUS next to each other on the screen, and participants used a mouse
The experiment was run from a MacBook (OS X 10.5.8). Stimuli to indicate their response. Following the melody test, the trial
were presented and data were collected using a custom-made patch ended for the control condition, or a distractor recognition test
in Max/MSP. Participants wore Sennheiser HD 650 headphones, appeared, in which participants were presented with the distractor
and data in the loudness change imagery task were collected stimulus (letter, tone, or movement sequence) and used the same
by means of an I-CubeX push v1.1 slider facing away from the buttons to indicate whether the distractor test sequence was the
participant at a slight upwards incline (Figure 1). same or different to the distractor stimulus which had originally
been presented. Figure 2 shows the procedure.
PROCEDURE
The study was approved by the Human Research Ethics Commit- Imagery for loudness change
tee of the University of Western Sydney. Written informed consent In a loudness change scale trial, a scale modulated in acoustic
was first sought to participate in the study, and general instructions intensity (loudness change) was sounded, followed by presenta-
about the format of the experiment were provided. Participants tion of the distractor stimulus (letter sequence, tone sequence,
movement sequence, or control period of 8 s). Immediately after
the distractor stimulus presentation, an unfolding visual score of
the notes of the scale was presented on the screen, and partici-
pants used a volume slider to indicate their mental image of the
loudness change profile of the scale that they had heard. Notes
appeared on the score at the same pace as they had been sounded
at the start of the trial (i.e., one note per 500 ms), and participants
were instructed to match the timing of their slider adjustments
to the timing of the unfolding visual score. To ensure that slider
movements began from the appropriate imagined loudness level
at the start of the scale, a 2-s long orientation period was pro-
vided, visually marked by a yellow circle on the screen. During
this time participants were to move the slider to the level that
they thought best represented the opening loudness of the scale,
before going on to indicate the loudness changes corresponding
to the visually unfolding scale2 . Following the loudness change
FIGURE 1 | Slider used to record imagery for loudness change.

2 Since moving a slider is a motor task, it was important to ensure that any impaired
Photograph of an I-CubeX push v1.1 slider, used to measure the changes in
loudness imagined by participants during the “Imagery for loudness performance associated with movement sequence distractor conditions could not be
change” task. attributable to physical motor production demands. A separate experiment required
12 participants (six from the current experiment and six new participants) to use the
FIGURE 2 | Details of experimental procedure. Schematic of an experimental trial. In the imagery for melodies task, the procedure is represented in the
upper panel of frame 3. In the imagery for loudness change task, the procedure is represented in the lower panel of frame 3.
scale test, the trial ended for the control condition, or a distractor position was sampled continuously (every 100 ms) only when the
recognition test appeared, in which participants were presented slider was in motion. Scale key velocity profiles, originally 16 events
with the distractor stimulus (letter, tone, or movement sequence) in length, were therefore stretched so that they were continuous
and used “Same” and “Different” buttons to indicate whether the and had the same number of events as each individual participant
distractor test sequence was the same or different to the distractor profile. Temporal relationships between loudness changes were
stimulus which had originally been presented. Figure 2 shows the maintained through this step of the analysis. The DTW distance
procedure. between each participant response profile and corresponding key
Each block comprised six trials and a practice. The experiment velocity profile was then calculated. A total of 0.3% of the data
lasted approximately 45 min. from four trials (belonging to three participants) were excluded
as outliers as DTW values were greater than 2.5 × SD from the
ANALYSIS participant’s mean.
Loudness change scale recall scores
Participant responses for the loudness change scale recall task com- Melody recognition scores
prised the series of slider values produced by each participant on Accuracy in the melody recognition was calculated as the propor-
each trial. Figure 3 illustrates sample participant response profiles tion of correct responses (i.e., correct identification of a different
and the corresponding reference scale key velocity profiles for four or same stimulus) from all given responses per condition. Four
trials. Each participant’s performance on the task was assessed by participants were at chance performance only in the nil distractor
measuring the similarity between their response profiles and the (control) condition, and consequently they were excluded from
corresponding scale key velocity profiles. Dynamic time warping analyses on the melody task.
(DTW; Giorgino, 2009) was used to compare participant response
profiles and scale key velocity profiles. DTW is suitable for use Multi-level linear modeling
with time-series data as it does not require independence of data Multi-level linear modeling was used (lme4 in the statistical
points within the series. It identifies points along test (i.e., partic- program “R”) to determine how well the distractor condition
ipant response) and reference (i.e., scale key velocity) data series was able to model the scores. One advantage of this approach
that most likely correspond with each other. An average distance over ANOVA is the possibility of modeling random effects so
between profiles per event is then calculated that is independent that different intercepts and gradients for individuals and block
of profile length. Participant profiles varied in length, since slider order can be included, thus controlling for intersubject vari-
ability or order effects. Models were developed stepwise, using
slider to indicate the loudness changes that they were hearing in the moment. Since interference condition as a predictor, and testing the impact of
this perceptual task did not require participants to rehearse or recall loudness change, individuals, block order, OMSI, and years of musical training
we did not expect any interference from concurrent distractor conditions. Indeed, as random effects. Model selection used the Bayesian Infor-
the participants’ accuracy in marking loudness changes under letter and move- mation Criterion (BIC) to determine the most parsimonious
ment sequence conditions was not significantly different (z = −0.235, p = 0.81),
suggesting that any relative impairment we might find from rehearsing movement
fit. Confidence intervals (CI) were calculated as Highest Poste-
sequences in the current imagery experiment could not be attributable to the motor rior Density estimates obtained by Markov Chain Monte Carlo
production of the task. sampling.

FIGURE 3 | Sample participant response profiles and scale key participant response and scale key velocity profiles per 100 ms time
velocity profiles. Each plot shows one participant’s response profile interval. Plots (A) and (C) show trials in which loudness change
for a single trial and the corresponding scale key velocity profile that profiles were imagined correctly and with precise timing; plots
the participant was attempting to recall and map out while imagining (B) and (D) show trials in which loudness change profiles were not
the scale. DTW provides a measure of the average distance between imagined correctly.
RESULTS p = 0.21) nor movement sequence distraction (β = 0.09, t = 1.91,

IMAGERY FOR LOUDNESS CHANGE 95% CI: −0.19 to 0.01, p = 0.06) was significantly worse than in
In the best fit multi-level linear model, recall of loudness change the nil distractor control. Results are displayed in Figure 5.
after the movement sequence distractor was significantly worse As in the model of DTW distances as an index of imagery for
than recall under the nil distractor condition (β = 1.04, t = 2.07, loudness change, the optimized model of accuracy in recognizing
95% CI: 0.04 to 2.07, p = 0.04). Figure 4 displays the DTW melodies included a random intercept for each individual partic-
distances in each of the distractor conditions. Neither recalling ipant (SD = 0.10). Here no significant contribution of years of
loudness change scales under letter sequence distraction (β = 0.90, musical training or block order was found.
t = 1.86, 95% CI: −0.07 to 1.84, p = 0.06), nor tone sequence dis- Melody recognition in the nil distractor/control condition was
traction (β = 0.42, t = 0.87, 95% CI: −0.50 to 1.44, p = 0.38) was significantly better than chance [t (26) = 33.5, p < 0.0001], making
significantly worse than in the nil distractor control. it unlikely that performance in this task was at floor.
The optimized model also included a random intercept for each
individual participant (SD = 2.39), for years of musical training MEMORY FOR DISTRACTOR STIMULI
(SD = 0.89), and for block order (SD = 0.67). It was also of interest to compare memory for the distractor stim-
uli following performance in each of the quite different loudness
IMAGERY FOR MELODIES change scale recall and melody recognition tasks. Table 1 sum-
In the model of accuracy in the melody recognition task, perfor- marizes the proportions of correctly remembered letter, tone, and
mance was not significantly different from the nil distractor condi- movement sequences.
tion under tone sequence distraction (β = 0.08, t = 1.70, 95% CI: During the loudness change scale task, correct recall of the
−0.17 to 0.03, p = 0.09). Neither melody recognition under letter distractor sequences differed significantly by distractor type,
sequence distraction (β = 0.06, t = 1.26, 95% CI: −0.16 to 0.04, as assessed by a repeated measures ANOVA of accuracy in
F (2,94) = 8.11, p < 0.001. Once again, letter sequences were rec-
ognized better than tone, t (31) = 3.16, p < 0.01, and movement
sequences t (30) = 4.56, p < 0.001, but accuracy recognizing tone
and movement sequences did not differ, t (30) = 1.16, p = 0.05.
IMAGERY AND MUSICAL EXPERIENCE

No relationship was found between OMSI and score when mod-
eling performance in either the melody recall or loudness change
scale reproduction tasks. The OMSI is designed to categorize par-
ticipants as more (>500) or less (<500) musically sophisticated.
Comparing the imagery scores of participants categorized in this
way confirmed the result from linear modeling that there were no
significant differences on either imagery task along this dimen-
sion. However, years of musical training contributed to the model
of accuracy in loudness change scale reproduction. Furthermore,
FIGURE 4 | Mean loudness change scale recall score (DTW). Mean DTW a positive correlation of years of musical training with accu-
distances between slider response and stimulus intensity profiles under
racy on the melody imagery task was found r(30) = 0.4, p < 0.05
different distractor conditions. Small values resemble close loudness
change reproduction, i.e., better task performance. Error bars represent (two-tailed).
standard error.
DISCUSSION
This experiment aimed to determine the disruptive effects of
rehearsing letter, tone, and movement sequences on mental
imagery for changes in loudness. As predicted, rehearsing a move-
ment sequence in mind significantly impaired the recall of loud-
ness change scales. Rehearsing tone sequences did not, though
rehearsing letter sequences, a task which could have involved
subvocal motor rehearsal, came close to producing a signifi-
cant impairment. Analyses of how well participants were able to
remember the different distractor stimuli revealed that movement
sequence recognition was consistently weaker than the recogni-
tion of the other distractor sequences. Equating the difficulty of
tasks that are to be used in working memory experiments is a
vexed issue that receives relatively little discussion. While recog-
nizing conductor gestures might be regarded as more difficult,
it is just as likely that the stimuli and task are less familiar than
performing a task containing letters or musical tones. Familiarity
refers to having knowledge of the material in long-term memory.
FIGURE 5 | Mean accuracy in melody recognition under different Thus familiarizing participants with novel material such as ges-
distractor conditions. Error bars represent standard error. Dashed line
represents chance level accuracy.
tures within an experiment is one way in the future that could
strengthen task comparability. Alternatively, unfamiliar words
and tones could be used to be more comparable with the novel
conducting gestures.
Table 1 | Mean and SD (in parentheses) of accuracy in distractor
It seems likely that retaining the movement sequences pre-
stimulus recognition.
sented a substantial cognitive load during the performance of
Letter Tone Movement any concurrent memory task. However, memory for melodies
was only marginally impaired by the movement distractor task,
Loudness change scale recall 0.87 (0.1) 0.76 (0.2) 0.55 (0.2) and so its impact primarily concerned the specific task of repro-
Melody recognition 0.86 (0.1) 0.74 (0.2) 0.68 (0.2) ducing imagined changes in loudness. While a motor response
was required to reproduce these imagined loudness changes,
evidence from a separate experiment suggests that mentally
the distractor task, F (2,94) = 28.11, p < 0.001. Planned con- rehearsing the movement sequences does not impair use of the
trasts using a Bonferroni adjusted α of 0.02 show that let- slider per se. Consequently, our experiment provides evidence
ter sequences were recognized better than tone sequences that imagery for musical loudness change can involve motor
t (31) = 2.64, p < 0.01, which were recognized better than move- processing.
ment sequences, t (30) = 6.94, p < 0.001. Contrary to expectations, the accurate imagining of melodies
The correct recall of distractor sequences in the melody was not significantly impaired by the concurrent rehearsal of
recognition task also differed significantly by distractor type, tone sequences. Perhaps participants ignored the intervening tone

sequence, choosing to prioritize mental rehearsal of the melodies. were improved by accounting for the variability introduced by
Such a strategy would have been associated with poor performance individual participants.
on the subsequent tone recognition task, yet accuracy was bet- Participants in this study were able to read music, suggesting at
ter than chance [t (31) = 5.32, p < 0.001]. In addition, the verbal least a minimal amount of musical training. Not only has a link
sequence did not impair melody recognition, a finding which is been established between auditory imagery abilities and musi-
at odds with the results of Keller et al. (1995). Indeed, no differ- cal training (Aleman et al., 2000), this has been extended to the
ences in melody recognition were observed between the different particular context of action-effect anticipation (Keller and Koch,
distractor conditions. 2008). This suggests that a tight sensorimotor coupling results
The retention of letter sequences was significantly higher than from extensively rehearsed associations between an action and its
the retention of other distractor sequences during both the loud- consequent sound. In the current study, the participants were not
ness change scale and melody tasks. Yet this superior letter recog- required to physically produce the test stimuli, and so had not
nition did not come at the price of memory for the loudness explicitly learned an association between movement and melody
change scale or melodies. The letters were presented visually, but or loudness change scale items. Nevertheless, musical experience
they were selected in the knowledge that they might be encoded has been found to enhance action-effect coupling quite broadly
phonologically and rehearsed as an acoustic image or by subvocal (Keller and Koch, 2008), and in the current experiment this might
rehearsal. An absence of interference from the letters in the melody have reinforced the ability of participants to imagine motor and
recognition task might suggest a visual rehearsal strategy, while a auditory components of loudness change. Indeed, the optimal
lack of significant interference in the loudness change scale recall model of performance in the imagery for loudness change task
task might point to a similar approach, with the interesting corol- included years of musical training. A correlation between years of
lary that if letter sequences were rehearsed visually, the successful musical training and accuracy in the melody imagery task was also
imagining of loudness change scales must be achieved as a motor found.
or auditory image, and not as a visual image of crescendo and Audio-motor coupling has been argued to be strong for musi-
decrescendo markings. cians, and Baumann et al. (2007) have suggested that the activation
The finding that imagery for musical changes in loudness is of both auditory and motor areas of the brain while listening
disrupted by the concurrent rehearsal of a movement goes some or playing even when participants attend to a distractor task is
way to answering the question of whether patterns of musi- evidence for direct and automatic connections between auditory
cal loudness are best described as a verbal, auditory, or motor and motor areas in music. In an interview study of experienced
representation. To be added to the list of potential modalities musicians, Holmes (2005) found that motor imagery was a signif-
is vision, given that the presentation of the conductor gestures icant part of learning and memorizing music for performance.
was visual, and participants might have been translating the It is interesting that years of training was a significant factor,
changes in loudness that they heard into a visual code of what which amounts to an index of performance experience, while
are called “hair pins” (score annotations to indicate crescendi OMSI score was not, which is a composite measure of musical
and decrescendi). The absence of an impairment from rehears- sophistication, taking into account listening and compositional
ing tones does not seem to suggest an exclusively auditory image. experience. While the current study relied on musically literate
Similarly, the lack of a statistically significant impairment from participants, it would be interesting to investigate the modalities
rehearsing a letter sequence does not point to a uniquely verbal involved in imaging musical loudness change for a wider popula-
labeling of loudness change information such as “up,” “down.” tion. Some evidence points to a common metaphorical association
The most likely scenario is that a balance of representation modal- of movement with musical loudness change (Eitan and Granot,
ity was involved. Such a view is consistent with current accounts 2006).
of working memory that emphasize interference from process In conclusion, we have presented behavioral evidence for motor
rather than content; these accounts recognize the influence of processing in the imagining of musical changes in loudness.
task demands, task relevant, and task irrelevant information, Although concurrent verbal and auditory distractor tasks did
instructions, and context on performance (e.g., Marsh et al., not significantly impair participants’ ability to imagine loudness
2009). change stimuli, we should not conclude that a uniquely motor
Working memory involves simultaneous short-term storage representation drives imagery for musical loudness change. These
and processing of information (Oberauer, 2009) and is limited verbal and auditory distractor tasks failed to impair performance
to three to five meaningful items in adults (Cowan, 2010). Indi- on a melody imagery task, in spite of previous research to sug-
vidual differences in working memory capacity are thought to gest that melodic material is rehearsed as an auditory image.
relate to differences in maintenance and retrieval. More specif- Future work is needed to determine how rehearsing the tonal
ically, working memory limitations arise from differences in and letter sequences employed in the current study should have
both ability to actively maintain information and ability to impaired auditory and verbal imagery for musical stimuli. Indi-
retrieve task relevant information in the presence of highly inter- vidual differences in imagery ability were evident, and it remains
fering or irrelevant information (Unsworth and Engle, 2007). to be seen whether these individual differences are reflected in
Research also points to considerable variation in imagery abil- the processing modalities preferred when imagining musical stim-
ities (Mast and Kosslyn, 2002; Keller and Koch, 2006), and the uli. Our ongoing research is studying the use of mental imagery
current study is consistent with this, as models of performance for loudness change by expert musicians in performance. The
on both types of imagery task (loudness change and melody) current experiment has provided an effective interference task
for loudness change imagery in the guise of conductor ges- ACKNOWLEDGMENTS

tures, allowing us to examine the strategies used by musicians Our thanks to Stephen Fazio for assistance with editing video clips,
when they cannot consciously plan (imagine) their expressive Yvonne Leung for data collection, and to members of the Amateur
intentions. Chamber Music Society for their generous help collecting data.
REFERENCES Dowling, W. J. (1978). Scale and con- ambiguous: insights from individual Trusheim, W. H. (1991). Audiation and
Aleman, A., Nieuwenstein, M. R., tour: two components of a theory of differences in spatial transformation mental imagery: implications for
Böcker, K. B. E., and De Haan, E. H. memory for melodies. Psychol. Rev. abilities. Cognition 86, 57–70. artistic performance. Q. J. Music
F. (2000). Music training and men- 85, 341–354. Mikumo, M. (1994). Motor encoding Teach. Learn. 2, 138–147.
tal imagery ability. Neuropsychologia Eitan, Z., and Granot, R. Y. (2006). strategy for pitches and melodies. Unsworth, N., and Engle, R. W. (2007).
38, 1664–1668. How music moves: musical parame- Music Percept. 12, 175–197. The nature of individual differences
AMEB. (2002). Aural Tests: Graded Exer- ters and listeners’ images of motion. Oberauer, K. (2009). Interference in working memory capacity: active
cises in Aural Skills. Melbourne: Music Percept. 23, 221–247. between storage and process- maintenance in primary memory
Allans. Finney, S., and Palmer, C. (2003). Audi- ing in working memory: feature and controlled search from sec-
Bailes, F. (2009). “Translating the musi- tory feedback and memory for music overwriting, not similarity-based ondary memory. Psychol. Rev. 114,
cal image: case studies of expert performance: sound evidence for an competition. Mem. Cognit. 37, 104–132.
musicians,” in Sounds in Transla- encoding effect. Mem. Cognit. 31, 346–357. Wachowicz, F., Stevens, C. J., and Byron,
tion: Intersections of Music, Technol- 51–64. Ollen, J. (2006). A Criterion-related T. P. (2011). Effects of balance cues
ogy and Society, eds A. Chan and Giorgino, T. (2009). Computing and Validity Test of Selected Indicators of and experience on serial recall of
A. Noble (Canberra: ANU E-Press), visualizing dynamic time warping Musical Sophistication Using Expert human movement. Dance Res. 29,
41–59. alignments in R: the dtw package. J. Ratings. Ph.D., Ohio State Univer- 450–468.
Bailes, F., and Bishop, L. (2012). “Musi- Stat. Softw. 31, 1–24. sity, Columbus. Wiens, P. W. (2002). “Expressive Con-
cal imagery in the creative process,” Grush, R. (2004). The emulation the- Pearson, J., Clifford, C. W. G., and ducting.” [Audiovisual material].
in The Act of Musical Composition: ory of representation: motor con- Tong, F. (2008). The functional Williamson, V. J., Baddeley, A. D., and
Studies in the Creative Process, ed. D. trol, imagery, and perception. Behav. impact of mental imagery on con- Hitch, G. J. (2010). Musicians’ and
Collins (Ashgate). Brain Sci. 27, 377–442. scious perception. Curr. Biol. 18, nonmusicians’ short-term memory
Baumann, S., Koeneke, S., Schmidt, Holmes, P. (2005). Imagination in prac- 982–986. for verbal and musical sequences:
C. F., Meyer, M., Lutz, K., and tice: a study of the integrated roles Pitt, M. A., and Crowder, R. G. (1992). comparing phonological similarity
Jancke, L. (2007). A network for of interpretation, imagery and tech- The role of spectral and dynamic and pitch proximity. Mem. Cognit.
audio-motor coordination in skilled nique in the learning and memori- cues in imagery for musical timbre. J. 38, 163–175.
pianists and non-musicians. Brain sation processes of two experienced Exp. Psychol. Hum. Percept. Perform. Wood, J. N. (2007). Visual working
Res. 1161, 65–78. solo performers. Br. J. Music Educ. 18, 728–238. memory for observed actions. J. Exp.
Berthoz, A. (1996). The role of inhi- 22, 217–235. Rossi-Arnaud, C., Cortese, A., and Psychol. Gen. 136, 639–652.
bition in the hierarchical gating Hubbard, T. L. (2010). Auditory Cestari, V. (2004). Memory span Wu, J., Yu, Z., Mai, X., Wei, J., and Luo, Y.
of executed and imagined move- imagery: empirical findings. Psychol. for movement configurations: the (2010). Pitch and loudness informa-
ments. Brain Res. Cogn. Brain Res. Bull. 136, 302–329. effects of concurrent verbal, motor tion encoded in auditory imagery as
3, 101–113. Intons-Peterson, M. J. (1992). “Compo- and visual interference. Curr. Psy- revealed by event-related potentials.
Berz, W. L. (1995). Working memory nents of auditory imagery,” in Audi- chol. Cognit. 22, 335–349. Psychophysiology 48, 1–5.
in music: a theoretical model. Music tory Imagery, ed. D. Reisberg (Hills- Sacks, O. (2007). Musicophilia: Tales of
Percept. 12, 353–364. dale, NJ: Lawrence Erlbaum Asso- Music and the Brain. London: Pica- Conflict of Interest Statement: The
Brodsky, W., Henik, A., Rubinstein, ciates), 45–71. dor. authors declare that the research was
B., and Zorman, M. (2003). Audi- Kalakoski, V. (2001). “Musical imagery Smith, J. D., Reisberg, D., and Wilson, M. conducted in the absence of any com-
tory imagery from musical nota- and working memory,” in Musical (1992). “Subvocalization and audi- mercial or financial relationships that
tion in expert musicians. Percept. Imagery, eds R. I. Gødoy and H. Jør- tory imagery: interactions between could be construed as a potential con-
Psychophys. 65, 602–612. gensen (Lisse: Swets & Zeitlinger), the inner ear and inner voice,” in flict of interest.
Covington, K. (2005). The mind’s ear: I 43–56. Auditory Imagery, ed. D. Reisberg
hear music and no one is perform- Keller, P. E., and Koch, I. (2006). Exoge- (Hillsdale, NJ: Lawrence Erlbaum Received: 30 April 2012; accepted: 06
ing. Coll. Music Symp. 45, 25–41. nous and endogenous response Association), 95–117. November 2012; published online: 03
Cowan, N. (2010). The magical priming with auditory stimuli. Adv. Smith, J. D.,Wilson, M., and Reisberg, D. December 2012.
mystery four: how is working Cognit. Psychol. 2, 269–276. (1995). The role of subvocalization Citation: Bailes F, Bishop L, Stevens
memory capacity limited, and Keller, P. E., and Koch, I. (2008). Action in auditory imagery. Neuropsycholo- CJ and Dean RT (2012) Mental
why? Curr. Dir. Psychol. Sci. 19, planning in sequential skills: rela- gia 33, 1433–1454. imagery for musical changes in loud-
51–57. tions to music performance. Q. J. Smyth, M. M., and Pendleton, L. R. ness. Front. Psychology 3:525. doi:
Dean, R. T., and Bailes, F. (2008). Is Exp. Psychol. 61, 275–291. (1989). Working memory for move- 10.3389/fpsyg.2012.00525
there a ‘rise-fall temporal archetype’ Keller, T. A., Cowan, N., and Saults, ments. Q. J. Exp. Psychol. 41A, This article was submitted to Frontiers in
of intensity in electroacoustic music? J. S. (1995). Can auditory mem- 235–250. Perception Science, a specialty of Frontiers
Can. Acoust. 36, 112–113. ory for tone pitch be rehearsed? J. Smyth, M. M., and Waller, A. (1998). in Psychology.
Dean, R. T., and Bailes, F. (2010). A rise- Exp. Psychol. Learn. Mem. Cogn. 21, Movement imagery in rock climb- Copyright © 2012 Bailes, Bishop, Stevens
fall temporal asymmetry of intensity 635–645. ing: patterns of interference from and Dean. This is an open-access arti-
in composed and improvised elec- Marsh, J. E., Hughes, R. W., and Jones, D. visual, spatial and kinaesthetic sec- cle distributed under the terms of the
troacoustic music. Organised Sound M. (2009). Interference by process, ondary tasks. Appl. Cogn. Psychol. 12, Creative Commons Attribution License,
15, 147–158. not content, determines semantic 145–157. which permits use, distribution and
Deutsch, D. (1970). Tones and num- auditory distraction. Cognition 110, Todd, N. P. M. (1992). The dynam- reproduction in other forums, provided
bers: specificity of interference in 23–38. ics of dynamics: a model of musical the original authors and source are cred-
immediate memory. Science 168, Mast, F. W., and Kosslyn, S. M. expression. J. Acoust. Soc. Am. 91, ited and subject to any copyright notices
1604–1605. (2002). Visual mental images can be 3540–3550. concerning any third-party graphics etc.

REVIEW ARTICLE
HUMAN NEUROSCIENCE
published: 05 September 2012
doi: 10.3389/fnhum.2012.00247
Imagining is not doing but involves specific motor

commands: a review of experimental data related to
motor inhibition
Aymeric Guillot 1,2 *, Franck Di Rienzo 1 , Tadhg MacIntyre 3 , Aidan Moran 4 and Christian Collet 1
1
Centre de Recherche et d’Innovation sur le Sport (EA 647), équipe Performance Motrice, Mentale et du Matériel, Université de Lyon, Université Claude Bernard
Lyon 1, Villeurbanne, France
2
Institut Universitaire de France, Paris, France
3
Physical Education and Sport Sciences Department, University of Limerick, Limerick, Ireland
4
School of Psychology, University College Dublin, Dublin, Ireland
Edited by: There is now compelling evidence that motor imagery (MI) and actual movement share
common neural substrate. However, the question of how MI inhibits the transmission of
South Wale, Australia
motor commands into the efferent pathways in order to prevent any movement is largely
Reviewed by:
Stephanie D. Preston, University of unresolved. Similarly, little is known about the nature of the electromyographic activity that
Michigan, USA is apparent during MI. In addressing these gaps in the literature, the present paper argues
Yoshikazu Ugawa, Fukushima Medical that MI includes motor execution commands for muscle contractions which are blocked
Universtiy, Japan
at some level of the motor system by inhibitory mechanisms. We first assemble data
*Correspondence:
from neuroimaging studies that demonstrate that the neural networks mediating MI and
Aymeric Guillot , Centre de Recherche
et d’Innovation sur le Sport, motor performance are not totally overlapping, thereby highlighting potential differences
Université Claude Bernard Lyon 1, between MI and actual motor execution. We then review MI data indicating the presence
27-29 Boulevard du 11 Novembre of subliminal muscular activity reflecting the intrinsic characteristics of the motor com-
1918, 69622 Villeurbanne Cedex,
mand as well as increased corticomotor excitability. The third section not only considers
France.
e-mail: aymeric.guillot@univ-lyon1.fr the inhibitory mechanisms involved during MI but also examines how the brain resolves
the problem of issuing the motor command for action while supervising motor inhibition
when people engage in voluntary movement during MI.The last part of the paper draws on
imagery research in clinical contexts to suggest that some patients move while imagining
an action, although they are not aware of such movements. In particular, experimental data
from amputees as well as from patients with Parkinson’s disease are discussed. We also
review recent studies based on comparing brain activity in tetraplegic patients with that
from healthy matched controls that provide insights into inhibitory processes during MI.
We conclude by arguing that based on available evidence, a multifactorial explanation of
motor inhibition during MI is warranted.
Keywords: motor imagery, motor command inhibition, motor performance, mental processes, electromyography,
sensorimotor control
INTRODUCTION motor output (Decety and Grezes, 1999; Collet and Guillot,
One of the most remarkable capacities of the mind is its ability 2010).
to simulate sensations, movements, and other types of expe- The vast majority of experimental investigations dealing with
rience. In most occasions, mentally imagining is like perceiv- the MI experience primarily focused on visual and kinesthetic
ing in the absence of the corresponding sensory information. imagery (KI). While visual imagery (VI) refers as to the visual-
In other words, imagining involves “seeing” with the “mind’s ization of an action, KI involves the sensations of how it feels to
eye,” “hearing” with the “mind’s ear,” and so on for each sen- perform an action, including the force and effort perceived during
sory modality (Kosslyn, 2010). Accordingly, mental imagery movement and balance (Callow and Waters, 2005), hence suggest-
is a multimodal construct which consists of either recalling ing to consider the body as a generator of forces (Jeannerod, 1994).
previously perceived images or feelings, or envisaging forth- Interestingly, previous data showed that KI was close to motor
coming events. Within this construct, “motor imagery” (MI) execution, with an extensive overlap of the corresponding neural
refers to the mental representation of an action without engag- networks (Solodkin et al., 2004). Other researchers introduced the
ing in its actual execution. MI involves an integrated covert concept of imagery perspective and further distinguished between
simulation of physical movement, and may be defined as a first- and third-person VI perspectives. During the first-person
dynamic mental state during which the representation of a given perspective, performers visualize the action as how would hap-
motor act is rehearsed in working memory without any overt pen in the real-life situation, while in the third-person perspective,
Frontiers in Human Neuroscience www.frontiersin.org September 2012 | Volume 6 | Article 247 | 130
Guillot et al. Motor command inhibition and motor imagery
they imagine, as spectators, the action that somebody is perform- strictly identical. This is because when performing MI, participants
ing, regardless of the agency of the movement (i.e., whether they, are aware that movement will not be performed, and therefore that
“see” themselves or others). Although there is some confusion in motor commands must be inhibited.
the usage of these terms in many studies (Moran et al., 2012), a Neuroimaging studies provided preliminary evidence that
considerable amount of experimental research suggests that MI motor-related areas of the brain (e.g., the ventral and dorsal parts
is a valuable and cost-effective technique to improve motor per- of the premotor cortex, as well as the supplementary motor area –
formance and to enhance motor recovery (see reviews by Driskell SMA) and subcortical areas including the cerebellum and the
et al., 1994; de Vries and Mulder, 2007; Guillot and Collet, 2008; basal ganglia, are active during MI of both simple and complex
Munzert et al., 2009). movements (e.g., Lotze and Halsband, 2006; Guillot et al., 2008;
Despite the preceding evidence on the efficacy of MI, several Munzert et al., 2009). Furthermore, research suggests that MI acti-
unresolved issues have emerged with regard to the neural under- vates a subset of areas required for movement execution (Macuga
pinnings of this construct. More precisely, little agreement exists and Frey, 2012), thus leading to a partial overlap in the corre-
among imagery researchers as to the extent to which the neural sponding neural networks. The contribution of the contralateral
substrates of MI overlap with those of actual practice. Also, little primary motor cortex (cM1) to imagined actions is more con-
evidence exists on the question of whether or not motor com- troversial, however (for reviews, see Lotze and Halsband, 2006;
mands are inhibited during imagined movements (Kasess et al., Munzert et al., 2009; Lotze and Zentgraf, 2010). Whereas some
2008). Nevertheless, certain trends exist in the relevant literature. researchers did not report cM1 activations during MI (e.g., Ger-
For example, the pattern of electromyographic (EMG) activity ardin et al., 2000; Hanakawa et al., 2008), others found fleeting
during MI generally supports the hypothesis of residual mus- involvement (Dechent et al., 2004) or significant activation (Lotze
cle activity which might originate from an incomplete inhibition et al., 1999b; Porro et al., 2000; Solodkin et al., 2004; Guillot et al.,
of the motor command (Jeannerod, 1994). Similarly, transcra- 2008; Sharma et al., 2008). Such discrepancies may be due to
nial magnetic stimulation (TMS) experiments also support this methodological differences and difficulties in monitoring com-
assumption by delineating the features of corticospinal facilita- pliance with MI instructions (Sharma et al., 2006). Interestingly,
tion during MI (Stinear, 2010). Finally, clinical studies show that Ehrsson et al. (2003) showed that the content of MI was reflected
some patients with specific brain damage fail to inhibit the motor in the pattern of motor cortical activation, as MI of hand, foot, and
action associated with its mental representation, and thus fully tongue movements specifically activated the corresponding hand,
“execute the imagined action,” hence highlighting uninhibited foot, and tongue sections of cM1. Additional evidence indicates
movements during mental rehearsal. Data in amputees, patients that activation of cM1 might be differentially influenced by MI
with spinal cord injury (SCI) and Parkinson’s disease (PD) fur- instructions, MI ability, and motor expertise (Lotze and Zentgraf,
ther contribute to theoretical understanding of how the motor 2010). Taken together, the bulk of neuroimaging studies suggest
command is inhibited during MI. that cM1 is activated during MI – but more weakly than during
Against this background, the present paper aims to exam- actual movement. Interestingly, Kasess et al. (2008) reported that
ine these issues and provide deeper evidence that MI includes SMA may substantially contribute to inhibit activity of cM1 during
motor commands for muscle contractions, which are blocked at MI.
some level of the motor system by inhibitory mechanisms. On Activation of parietal areas including the inferior and supe-
some occasions (e.g., in sport settings), individuals may retain rior parietal lobules, as well as the precuneus, was also frequently
the potential to move while imagining an action, e.g., miming reported during MI (Gerardin et al., 2000; Hanakawa et al., 2003;
unambiguously some parts of movement execution while rehears- Guillot et al., 2009; Munzert et al., 2009). Experimental studies in
ing the imagined movement. These situations raise an important patients with parietal lesions further support that these structures
but neglected question: How does the brain resolve the para- are critically involved in the generation and guidance of men-
dox whereby it is required to issue the motor command needed tal images, including the ability to achieve temporal congruence
for action when MI is performed, while concurrently issuing an between MI and motor performance (Sirigu et al., 1996; Malouin
inhibitory command when the person is moving during MI? et al., 2004).
The patterns of neural activity underlying imagery types (e.g.,
NEURAL CORRELATES OF MOTOR IMAGERY VI vs. KI) and imagery perspectives (first-person vs. third-person
Understanding the neural correlates of motor performance and MI imagery) are partially mediated through separate neural systems.
has been an important purpose of brain studies since the advent For instance, Solodkin et al. (2004) investigated neural networks
of neuroimaging techniques. Considerable experimental evidence associated with physical execution,VI, and KI of hand movements.
has accumulated to suggest that movement execution and MI share Although some shared neural substrates were evident between
substantial overlap of active brain regions (for review, see Guillot these processes, differences were found in the inputs received from
et al., 2012). Such apparent functional equivalence supports the the superior parietal lobule. Specifically, inputs from SMA to cM1
hypothesis that MI draws on the similar neural networks that are were lower to those observed during motor execution (Gao et al.,
used in actual perception and motor control (Jeannerod, 1994; 2011). In a single group of participants with high MI abilities,
Grezes and Decety, 2001; Holmes and Collins, 2001). Moreover, Guillot et al. (2009) showed that VI activated predominantly the
MI can also activate neural circuits used during tasks investigat- visual pathways including the occipital regions and the precuneus,
ing memory and emotion (Kosslyn et al., 2001). As we will see, whereas KI involved mainly motor-associated structures and the
however, the neural networks underlying these behavior are not inferior parietal lobule.
Neuroimaging studies have also shown that the neural networks Lotze et al., 2003; Hanakawa et al., 2008). In sum, these results
underlying MI differ as a function of both individual expertise seem to suggest that MI involves the intention of not executing
level and imagery ability. For instance, an inverse relationship the movement, with strong inhibitory mechanisms blocking the
between the pattern of brain activity and expertise level has been motor command.
reported (Ross et al., 2003; Milton et al., 2007) with decreased Conversely, similar EMG activity has been observed during
activations in the SMA, the cerebellum, and the basal ganglia. overt motor execution and MI conditions, with a reduced mag-
More recent work by Guillot et al. (2008) confirms that the neural nitude in the simulated action (e.g., Jowdy and Harris, 1990;
networks mediating MI partially differ as a function of imagery Hashimoto and Rothwell, 1999). Gandevia et al. (1997) further
ability. Specifically, whereas strong imagers tended to show acti- demonstrated that not only MI did activate the alpha motoneu-
vation in the parietal and ventrolateral premotor regions, weaker rons, but that the skeletomotor discharge was also accompanied by
imagers tended to recruit the cerebellum, the orbito-frontal, and recruitment of spindle afferents when the covert contraction was
posterior cingulate cortices. In both cases, however, dynamic brain sufficiently strong. Furthermore, there is evidence of increasing
changes were found to become more refined and circumscribed EMG activity accompanying imagined mental effort (Boschker,
with imagery/physical practice (PP) – a trend which was also 2001; Slade et al., 2002; Guillot et al., 2007). Finally, EMG activity
evident during the learning process of motor tasks. has been observed not only in agonistic muscles, but also in antag-
In summary, neuroimaging research clearly demonstrates that onistic muscles, as a function of both the weight to be lifted (Bakker
MI and motor performance of the same task share certain neural et al., 1996) and the muscle contraction type (Guillot et al., 2007).
substrates but the overlap is incomplete. Furthermore, data chal- In this latter study, the authors found that the subliminal muscle
lenge the assumption that neural activity is described as being responses during MI of concentric, isometric, and eccentric con-
in all of the same areas as execution, albeit to a lesser extent. As tractions typically mirrored the configuration of the EMG activity
expected, some areas that are active during motor performance recorded during actual practice. These data support the hypothe-
are not involved during MI. Although less common, the consis that muscle activity recorded during MI is not a general tonic
verse is also true, with some regions being more strongly and/or activation but reflects the content of the specific motor command
selectively activated during MI compared to actual execution of of the movement that is mentally rehearsed. According to Jean-
the same movement (e.g., the pre-SMA – e.g., Hanakawa et al., nerod (1994), an incomplete inhibition of the motor command
2008). From the preceding evidence, we conclude that the main could provide a valid explanation for these muscle discharges. So,
difference between MI and motor performance is probably that it seems that imagined movements produce a qualitatively sim-
MI involves the inhibition of some motor commands triggering ilar, but quantitatively smaller, drive to muscles compared with
movements – although the neural level at which the motor com- actual motor execution, thereby suggesting that a small part of the
mand is stopped has not yet been clearly identified. To address motor command is actually sent to the effectors during MI. Inter-
this latter issue, we believe that EMG recordings may provide a estingly, Solodkin et al. (2004) argued that both the supplementary
reliable means to investigate whether or not the brain activation motor area and the lateral premotor cortex might also play a role
recorded during MI actually originate from mental representa- in increasing muscle tone during MI – especially during KI. This
tion – as opposed to the potential motor activity that could have speculation is plausible given that these brain regions have direct
accompanied the task. In a similar vein, TMS can be used to explore projections to spinal cord through the internal capsule, adjacent to
the degree to which MI modulates both corticomotor excitability the well-known corticospinal path originating from cM1 (Luppino
and intracortical inhibition. et al., 1994; Morecraft et al., 2002).
Inconsistencies in the reports of concomitant EMG activity in
MUSCLE AND TMS ACTIVITY DURING IMAGINED ACTIONS the muscles participating in the movement during MI might be
EMG CORRELATES OF MOTOR IMAGERY explained by differences in the experimental designs, as well as
A great amount of experimental data has been collected on the by the nature of the EMG recordings (Guillot et al., 2010). For
physiological operations involved during MI (Table 1) – notably instance, EMG activity might not be discernible due to the use
the peripheral muscular activity which may occur during the men- of surface EMG electrodes and intramuscular electrodes should
tal representation of an action (Guillot et al., 2010). Since the ideally be preferred, although intrusive and thus rarely used in MI
pioneering work of Jacobson (1930, 1932), who provided the first experiments (except in the case of Gandevia et al., 1997). Analo-
scientific evidence that MI of bending the arm produced small con- gously, the effect of the muscle contraction type, the intensity of
tractions of the flexor arm muscles, debate has existed on whether the mental effort, and the intrinsic nature of MI may also con-
or not MI is accompanied by subliminal muscle contractions. tribute to understand why EMG activity was not systematically
Muscle quiescence during MI has been reported in many exper- reported. Finally, with few exceptions, studies reporting a lack
imental studies (e.g., Yue and Cole, 1992; Decety et al., 1993; Lotze of EMG activity primarily investigated laboratory movements,
et al., 1999b; Mulder et al., 2005). Interestingly, the lack of EMG whereas those experiments providing evidence of a muscle activ-
activity during MI was sometimes considered a precondition prior ity during imagery included more goal-related movements (e.g.,
to engaging in MI practice (Michelon et al., 2006). On some skills in sport). Experimental data reporting muscle activity only
occasions, EMG data were even recorded during scanning ses- in a part of the tested sample (Li et al., 2004a; Dickstein et al.,
sions per se, to demonstrate that variations of cerebral blood flow 2005) lend support to the fact that muscle activity was not system-
were directly related to the mental work and not to any concomi- atically discernible due to such confounding factors. Interestingly,
tant movement (e.g., Gerardin et al., 2000; Jackson et al., 2003; the pattern of muscle activation has never been found to match
Table 1 | Studies investigating the EMG activity during motor imagery.
Study Number of Motor task Main findings

participants
LACK OF MUSCLE ACTIVITY DURING MOTOR IMAGERY

Decety et al. (1993) n=6 Leg contraction to press and No change in phosphocreatine concentration or in pH during motor
release a loaded footplate imagery
Demougeot and n = 17 Vertical arm movements No arm muscle activation during motor imagery
Papaxanthis (2011)
Gentili et al. (2006) n = 40 Pointing arm movement No EMG activity during motor imagery
Gerardin et al. (2000) n=8 Auditory-cued hand Surface EMG did not detect any muscle activity during motor imagery
movements
Gueugneau et al. n=9 Pointing arm movement No EMG activity during motor imagery
(2008)
Hanakawa et al. n = 13 Finger tapping sequence Surface EMG was monitored during fMRI recordings to confirm the
(2008) absence of muscle activity during motor imagery
Jackson et al. (2003) n=9 Foot sequence task No significant difference in the EMG signal between imagery and
baseline conditions, showing that the patterns of cerebral activation
during fMRI recordings are not due to movements
Kleber et al. (2007) n = 16 Singing of an Italian aria No difference between baseline and imagined singing
Lafleur et al. (2002) n=9 Foot sequence task EMG recordings showed no change in muscle activity during scans
compared to baseline levels
Lim et al. (2006) n = 13 Arm movement No EMG activity during motor imagery
Lotze et al. (1999b) n = 10 Making a fist Low EMG activity, which did not differ from the baseline, was a
precondition before fMRI recordings
Lotze et al. (2003) n = 16 Performance of Mozart’s No observable differences between motor imagery and rest
violin concerto in G major
Mulder et al. (2004) n = 37 Abduction of the big toe No EMG activity during motor imagery
Mulder et al. (2005) n = 31 Squat movements with EMG activity recorded during motor imagery did not differ from baseline
additional weights
Naito et al. (2002) n = 10 Palmar flexion and No EMG activity in the motor imagery condition
dorsiflexion of the wrist
Personnier et al. n = 28 Arm movements in the Muscle activation patterns are very similar between motor imagery and
(2008) sagittal plane rest conditions
Ranganathan et al. n = 30 Isometric little finger Muscle activity during motor imagery was near zero
(2004) abduction and elbow flexion
Roosink and n = 20 Finger tapping sequence No EMG activity during motor imagery
Zijdewind (2010)
Shick (1970) n = 10 Volleyball serve skill No EMG activity during motor imagery
Yahagi et al. (1996) n=7 Wrist flexion No EMG activity during motor imagery
Yue and Cole (1992) n = 30 Isometric little finger No EMG activity during motor imagery
abduction
(Continued)
Table 1 | Continued

participants
Zijdewind et al. n = 29 Ankle plantar-flexion No EMG activity during motor imagery in the majority of the
(2003) participants. When little EMG activity was recorded, participants were
asked to concentrate until being able imagining the movement without
muscle activation
MUSCLE ACTIVITY DURING MOTOR IMAGERY
Bird (1984) n=5 Motor imagery of a past The EMG configuration during motor imagery mirrored that observed
athletic event (including during actual practice
riding, rowing, swimming,
water skiing, and basketball)
Bonnet et al. (1997) n = 26 Foot pressure on a pedal EMG activity weakly increased during motor imagery
Boschker (2001) n = 39 Arm lifting movements Significant EMG activity is recorded in the muscles contributing to the
Bakker et al. (1996) (biceps curls) contraction. Greater muscle activity in the active than in the passive
arm, and greater biceps activity when imagining lifting a heavy
compared to a light weight
Dickstein et al. n = 15 Rising on tiptoes EMG activity was recorded in six participants in at least one of the
(2005) target muscles
Gandevia et al. n = 12 Range of simple and complex Imagery increased background EMG in the involved muscles. In some
(1997) movements (e.g., occasions, spindle discharge also increased
flexions/extensions,
handwriting, walking,
threading a needle. . .)
Guillot et al. (2007) n = 30 Biceps dumbbell curls The magnitude of EMG activity is correlated to the mental effort required
to imagine the movement. EMG patterns during imagery of concentric,
isometric, and eccentric contractions mirror those observed during
actual movements. EMG activity is recorded in agonist, antagonist,
synergist, and fixator muscles
Hale (1982) n = 48 Biceps dumbbell curls Internal imagery perspective produced greater biceps activity than the
external imagery perspective
Harris and Robinson n = 36 Arm lifting Significant EMG activity is recorded in the muscles contributing to the
(1986) contraction. Greater EMG activity during the first-person than during the
third-person perspective
Hashimoto and n=9 Wrist flexion and extension Larger EMG responses in flexor and extensor muscles during imagined
Rothwell (1999) flexions and extensions, respectively
Jacobson (1930, The number of Biceps dumbbell curls, EMG activity was recorded in the specific muscle involved with the
1932) participants varied bending the forearm, imagined activity
among tasks sweeping, climbing a rope
Jowdy and Harris n = 38 Juggling task Increased muscle activity during motor imagery. No effect of the
(1990) imagery ability on the magnitude of muscle activity
Lebon et al. (2008) n = 30 Biceps dumbbell curls The median frequency of EMG power spectrum in agonist and
antagonist muscles was significantly higher during motor imagery than
during baseline
Li et al. (2004a) n=9 Flexion and extension EMG activity was recorded in the finger flexors in four participants
movements of the fingers
Livesay and Samaras n = 30 Tightly squeezing a hand-size Increased EMG activity in the dominant forearm
(1998) rubber ball
(Continued)
Table 1 | Continued

participants
Lutz and Linder n = 160 Dart throwing Greater biceps EMG activity was recorded when imagery instructions
(2001) included assertions about behavior, such as motor actions and visceral
responses
Shaw (1938) The number of Range of complex Increased EMG activity during motor imagery was distributed across
participants varied movements (e.g., different muscle groups including those not directly related to the
among tasks flexions/extensions, corresponding movement
handwriting, walking,
threading a needle. . .)
Slade et al. (2002) n = 60 Biceps dumbbell and EMG activity was significantly greater for both curls in the active arm
manipulandum curls during motor imagery when compared to baseline
Suinn (1980) n=1 Skiing a downhill race Recorded muscle patterns were strikingly similar to those observed
during actual practice
Wehner et al. (1984) n = 27 Contour tracking arm task Similar frequency distribution in the power spectrum during actual
practice and motor imagery
the usual triphasic sequence generated during actual motor perfor- within motor cortical areas during MI, including reduced intra-
mance (Murphy et al., 2008). Furthermore, although traditional cortical inhibition, are analogous to those observed during motor
psychoneuromuscular theory (Carpenter, 1894) postulated that preparation and execution (Abbruzzese et al., 1999; Kumru et al.,
muscle activity recorded during MI might generate slight neuro- 2008). Intracortical inhibitory interneurons are known to play an
muscular feedback, strong enough to improve subsequent motor essential role in the shaping of motor commands (Stinear and
performance through priming of the motor pathways, there is no Byblow, 2003a). This phenomenon is thought to mirror an analo-
direct evidence that muscle activation during MI is associated with gous motor activity at the cortical level during both tasks. Contrary
improved motor performance. In other words, research has not yet to PP, intracortical inhibition of cM1 is not entirely removed
demonstrated that the increase in muscle activity fully contributes during MI (Kumru et al., 2008). In general, whilst MI involves
to performance enhancement, but EMG recordings support that improved cortical facilitation and reduced intracortical inhibition,
the motor command is actually prepared, and then blocked by it does so with reduced amplitude compared to PP (Clark et al.,
inhibitory processes, during MI. 2004; Leonard and Tremblay, 2007). It has been suggested that the
CNS manages to keep corticospinal facilitation below the motor
CORTICOSPINAL FACILITATION PATTERNS DURING MI threshold for activating the alpha motor neurons pool during MI
Transcranial magnetic stimulation activates neurons trans- (Stinear, 2010).
synaptically (Rothwell, 1991) and allows the study of corticospinal Does cortical activity during MI effectively reflect shaping of
facilitation (i.e., the level of excitability of the corticomotor path- motor output – which may require motor inhibition? Firstly, cor-
way). Typically, the motor evoked potential (MEP) elicited by a ticospinal facilitation during MI is effector-specific according to
suprathreshold TMS pulse delivered to cM1 is recorded at the MI content (Kasai et al., 1997; Stinear and Byblow, 2003a, 2004).
peripheral level, using surface EMG. The development of repetitive The motor threshold in muscles involved into MI content is lower,
pulse TMS protocols has allowed the study of intracortical facil- while the amplitude of the subsequent MEP is higher (Facchini
itation and inhibition, and also helps to delineate excitatory and et al., 2002). By contrast, MEPs elicited in non-involved mus-
inhibitory interactions between different brain regions mediating cles remain unaffected (Facchini et al., 2002; Stinear and Byblow,
motor control (Reis et al., 2008). In this section, we review some 2003b; Quartarone et al., 2005). Furthermore, Leonard and Trem-
TMS studies suggesting that MI involves elaboration of motor blay (2007) demonstrated that the muscle-specific pattern of
command signals at the CNS level. We also discuss the extent corticospinal facilitation during MI was altered in aging popu-
to which MI activates the descending somatic motor pathways lations, thus reflecting individual ability to shape motor output
originating from pyramidal neurons and projecting toward the coding for isolated finger movements. Corticospinal facilitation
alpha motoneurons pool. Finally, we consider the recent hypothe- during MI is also graded upon the extent to which the muscle is
ses emerging from TMS findings regarding motor inhibition actually recruited during the corresponding motor performance
during MI. (Yahagi and Kasai, 1998). Liang et al. (2007) further reported
CNS activity during MI and PP enables researchers to under- that corticospinal facilitation during MI of wrist flexions mir-
stand whether MI effectively involves motor commands process- rored the synergic pattern of muscle activity produced by PP, a
ing. TMS studies provided converging evidence that MI increases key feature of the motor command (see also Kasai et al., 1997,
the corticomotor excitability (Stinear, 2010). Excitability changes for results suggesting preservation of agonist-antagonist patterns
of muscle activations during MI). Likewise, using TMS evoked corticospinal facilitation during MI primarily reflected changes
muscle twitch, Li et al. (2004b) reported that MI of finger flexion of cortical origin. Consequently, TMS data suggest the analogous
preserved the unintended functional coupling in strength response involvement of cM1 into motor command processing during both
between the digits, described as the “enslaving effect” (Zatsiorsky MI and PP. Both tasks elicit excitability changes at the cortical level
et al., 2000). Further, Facchini et al. (2002) observed that only MI (Kasai et al., 1997; Abbruzzese et al., 1999; Patuzzo et al., 2003; Stin-
of contralateral thumb abduction, but not ipsilateral, facilitated ear and Byblow, 2004), whereas there is a paucity of robust TMS
MEPs in the contralateral effector. This result supports the idea evidences of excitability changes at the spinal level. Consequently,
that MI reproduces the hemispheric specificity with regards to cM1 TMS results indicate that inhibition during MI might intervene
enrollment during imagination of lateralized movements. Finally, during the early stages of motor processing. Several neuroimaging
Fadiga et al. (1999) observed that MI of elbow flexion/extension findings support this contention, revealing that specific cortical
increased MEPs in the biceps brachii merely during the timing and subcortical sites could contribute to prevent overt motor
portions of MI corresponding to the arm flexion, thus suggesting processing during MI (Lotze et al., 1999b; Kasess et al., 2008).
that corticospinal facilitation during MI matched the temporal Challenges to these TMS-based accounts of motor inhibition
features of physical performance. during MI come from some EMG studies. For example, Bonnet
To summarize, the accumulating evidence appears to suggest et al. (1997) reported the sharp increase of both H- and T-reflexes
that the corticospinal facilitation is highly specific to the motor during MI of strong foot pressure above a pedal. In this study,
task. Increased corticomotor excitability during MI may not be a T-reflexes displayed a highly specific pattern of facilitation (i.e.,
result of a general state of arousal due to execution of cognitive lateralized and graded depending on the stimulated movement)
operations (Rossini et al., 1999; Clark et al., 2004; Fourkas et al., which was not observed in H-reflexes facilitation. Bonnet et al.
2006), but instead due to the demands of the internal processing (1997) argued that MI elicited both spinal and spindle activa-
of motor output (Stinear, 2010). tion in the task-relevant corresponding effectors. This finding was
As mentioned previously, most features of corticospinal facil- replicated in studies reporting increased H-reflexes excitability
itation during MI suggest internal elaboration of neural signals during MI (Hale et al., 2003). As mentioned above, Gandevia et al.
for muscle contractions. It is generally assumed that increased (1997) reported increased activity from spindle afferents using
corticospinal facilitation during MI reflects analogous involve- microneurographic recordings from the relevant muscles. The
ment of cM1 between MI and PP (Stinear, 2010). However, such authors concluded that MI recruited both motor units and afferent
increased corticomotor excitability may also reflect more general spindles. Their results further demonstrated that, in some cases,
changes in the balance between excitatory and inhibitory impulses, the motor commands built up during MI might reach the muscle
which can occur at different stages of the somatic motor system. level and elicit neural feedback from muscle receptors. Gandevia
Understanding how the centrally shaped motor output is inhib- et al. (1997) therefore stated that MI may consist of “unintentional
ited during MI initially requires further analysis of the extent to performance of (. . .) planned motor task,” hence suggesting that
which CNS excitability changes (due to MI) affect the descend- somatic activity during MI might account for the observed effects
ing motor pathways. For instance, one may question whether of MI training on motor performance, through reinforcement of
corticospinal facilitation during MI involves the motor system motor output conduction throughout the neuromuscular system
at the spinal level. Yahagi et al. (1996) addressed this issue and (Gandevia, 1999).
observed that whilst MI of wrist flexions facilitated the MEPs in Experimental studies also support the central elaboration of
the flexor carpi radialis, no change was recorded in the H-reflex motor commands during MI. For example, both EMG and TMS
surface EMG traces evoked by electrical stimulation, thus reveal- findings support for the role of concurrent indirect information
ing that corticospinal facilitation during MI occurred without any concerning motor inhibition during MI. Firstly, EMG data indi-
change in spinal excitability. This finding was replicated in several cate that a residual motor command can be partially addressed
studies investigating changes in H-reflex during MI, in combina- to peripheral effectors during MI. Secondly, TMS findings suggest
tion with TMS (Kasai et al., 1997; Hashimoto and Rothwell, 1999; that the motor system keeps the facilitation of the corticomotor
Patuzzo et al., 2003). F-waves elicited by peripheral nerve transcu- pathways below the motor threshold, in spite of a highly action-
taneous electrical stimulation provide an objective measurement specific pattern of arousal. These two ways of understanding motor
of spinal excitability, without interference from descending neural inhibition during MI could also represent different ways of analyz-
impulses of cerebral origin (e.g., cerebral spontaneous regula- ing a multimodal process: specific interactions between cerebral
tion of spinal reflexes). Rossini et al. (1999) reported a 9.8–14% regions could result in the transmission of a residual motor com-
increase in F-waves amplitude during MI of index and little finger mand toward the descending volleys, whilst interactions between
abduction. These robust results challenge previous observations cerebral sites and/or spinal influences could keep corticospinal
where no change in F-waves was recorded during MI of finger excitability below the motor threshold. For instance, a recent TMS
actions (Facchini et al., 2002; Stinear and Byblow, 2003a; Stin- finding asserts that the ipsilateral inferior parietal lobe might exert
ear et al., 2006). However, in the two movements investigated an inhibitory influence on cM1 during MI (Lebon et al., 2012).
by Rossini et al. (1999), only MI of finger abduction elicited a
slight increase by 5.9% in the TMS evoked MEP with no change MOTOR INHIBITION
recorded in the MEP latencies when compared to a non-motor Earlier in this paper, we showed that motor performance and MI
mental activity (mental arithmetic). Rossini et al. (1999) stated are mediated by distinct neural networks, despite an extensive
that while MI may have increased spinal motoneuronal excitability, overlap between KI and PP. In particular, while mental operations
of motor planning and programming are actually performed of the motor command? One of the most plausible outcomes is
during MI, motor commands must be inhibited before being sent that sensory afferent information provided to the CNS should
to peripheral effectors within the descending pathways. This inhi- serve as feedback in the hypothesis of a forthcoming actual move-
bition process aims at preventing the performer from engaging in ment. Secondly, the cerebellum might be involved in the inhibition
any movement during mental rehearsal. However, we previously of movement execution during MI (Lotze et al., 1999b).
underlined significant differences in cerebral activations elicited The question of inhibiting movement execution, after the
by MI and actual movement (Hanakawa et al., 2008), some of motor commands have been prepared, has often been asked (e.g.,
which being probably responsible of the inhibition of motor com- see de Jong et al., 1990). Motor inhibition is usually tested with
mands. Interestingly, Schwoebel et al. (2002) described unique the “Go/No-Go” paradigm. Here, participants are requested to
behavior from a patient with bilateral parietal lesions. When imag- give a motor response when a specific stimulus is presented and
ining hand movements, the patient simultaneously executed the to withhold the response occasionally when another stimulus is
imagined motor act but without being aware of the movements. triggered. Reaction time to the “Go” signal is recorded, thus facil-
Surprisingly, these movements were also significantly more accu- itating the study of how the motor system inhibits the response
rate than volitional movements. The findings from this clinical when the “No-Go” signal is randomly given. Typically, Go/No-Go
case study are consistent with previous accounts suggesting that paradigms elicit a race between response activation and response
MI may normally involve the inhibition of movements. inhibition processes. de Jong et al. (1990) postulated the existence
Most studies dealing with MI generally give information about of two inhibitory mechanisms: inhibition of central activation
the inhibition of motor commands by providing EMG recordings. processes and inhibition of transmission of motor commands
This is a reliable means to ensure that brain activation recorded from central to peripheral structures. Unfortunately, the way in
during MI actually comes from mental representation and not which these inhibitory processes work may not directly be applied
from potential motor activity that could have accompanied the to MI. For example, the processes of motor command inhibition
mental task. Thus, EMG recordings during MI should be com- during MI may not work exactly as during those elicited by the
parable to those that occur during rest. However, a common Go/No-Go paradigms because, in the latter, participants do not
challenge is that across most investigations, MI and rest condi- know in advance whether they will have to act or to inhibit action.
tions are rarely accurately compared. Guillot et al. (2007) observed By contrast, when participants are requested to mentally represent
significant increased pattern of EMG activity in all muscles of an action, they implicitly know that they will restrict their cere-
the arm, forearm, and even shoulder during MI of forearm flex- bral activity to covert movement only. Thus, motor command
ion, when compared to the rest condition, while goniometric data inhibition should be integrated into the process of movement
did not reveal any movement. The magnitude of this activation preparation through motor representation. As postulated by de
was correlated with the mental effort required to imagine lifting a Jong et al. (1990), a possible mechanism for response inhibi-
weight. Indeed, MI of heavy concentric contraction (80% of the tion (which could be applied to MI), involves the inhibition of
best mark) resulted in greater pattern of EMG activity than dur- central response activation processes. Thus, response initiation
ing MI of light concentric condition (50% of the best mark). The might be inhibited by preventing central response activation from
intensity of the imagined contraction was thus paralleled by the reaching the targeted muscles. In this way, the interruption of an
magnitude of the subliminal EMG activity, thereby highlighting a already initiated response can be achieved by discontinuing the
close link between the central nervous system and the periphery output from central to peripheral motor structures. This specula-
during MI. Bakker et al. (1996) and Boschker (2001) had previ- tion was experimentally attested by large fronto-central positivity
ously found that mentally lifting a 9 kg dumbbell resulted in a when the response was successfully inhibited (de Jong et al., 1990).
larger EMG activity than lifting 4 kg 1/2. Jeannerod (1994) and The inhibitory mechanisms are effective before sending the infor-
Bonnet et al. (1997) attributed changes in EMG activity during mation, elaborated within the associate cortices, to the primary
MI to an incomplete inhibition of the motor command. This motor cortex. Especially, this inhibition may originate from the
hypothesis was emphasized by differential muscle activity asso- prefrontal cortical areas associated with limbic structures and cin-
ciated with the contraction type. Interestingly, different types of gulate cortices (the behavioral inhibition system early postulated
mentally rehearsed contractions elicited specific changes in EMG by Gray, 1990). This behavioral system has other connections
activity that closely corresponded to those observed during actual with the parietal associative cortices involved in No-Go perfor-
contraction. mance (Watanabe et al., 2002). However, response inhibition could
The preceding evidence shows that MI might recruit the same also come from active mechanisms at different subcortical levels
movement pattern as the actual motor command, although at sub- including the spinal cord (Bonnet et al., 1997), the brainstem, and
liminal intensity, hence involving the same neural substrate. Thus, the cerebellum (Lotze et al., 1999b). A particular example is when
EMG activity during MI seems to mirror that observed during the programming of a movement is not entirely well-adapted to
actual motor execution. Importantly, this was not a tonic non- its expected goal and requires changing one or several parame-
specific activity as the patterns of EMG distinguished among the ters, such as movement direction or amplitude. In this case, we
different types of contraction to the same extent as actual execution do not need to fully inhibit actual performance but only to better
would have done. adapt the programming of movement to the environmental con-
Arising from the argument so far, two questions need to be straints under which it occurs. Thus, a flexible central inhibitory
addressed. First, how useful is this specific residual motor com- mechanism may become crucial when selective motor inhibition
mand? Second, what are the neural substrates of partial inhibition is required. Many results from neuroimaging research suggest that
the right inferior frontal gyrus is integrated within a fronto-basal- If we consider that motor planning and programming are central
ganglia network (Aron et al., 2007), which could intercept the “Go” processes, we can assume that these are common to actual execu-
process and stop the motor responses (Lenartowicz et al., 2011). tion and MI. Hence, the difference between these two behaviors
This function is also consistent with a role in reprogramming of would be the existence of an active process of motor command
action plans, which may comprise inhibition, and its activity can inhibition. Now, the question of how motor command inhibition
be triggered through automated, bottom-up processing. is neurally implemented remained unresolved. But what exactly
If central inhibition processes do not succeed in preventing does research tell us about motor command inhibition and how
central motor outflow, the overt response can be inhibited by pre- can this knowledge be applied to MI? Most experiments on motor
venting the transmission of motor commands to peripheral motor command inhibition were conducted using stop signal paradigms,
structures. This possibility is consistent with the hypothesis that early formalized by Logan and Cowan (1984). The key compo-
motor commands could be inhibited at any time (de Jong et al., nent of response inhibition depends upon the relative finishing
1990). In other words, the inhibition of overt movements may time between Go and No-Go operations. In other words, the Go
still be achieved by means of peripheral mechanisms. By contrast response is inhibited by the activation of a stop-process. The major
with central inhibitory mechanisms, peripheral inhibitory mech- difference between inhibition of motor command in the context
anisms may be useful only when actions have to be inhibited or of action execution vs. MI is mainly related to uncertainty. Uncer-
interrupted unselectively. Brunia (2003) proposed that there are tainty is emphasized in an updated model by Boucher et al. (2007).
several inhibitory mechanisms at work in the periphery of the A Go response may also be inhibited by the preparation of an alter-
motor system, all depending upon activity of local propriospinal native go response. In this case, response inhibition would depend
interneurons, situated at the same or neighboring segments of the on the relative finishing time of the primary-task response and
spinal cord as that of the motoneurons of agonist or synergist the alternative response (Verbruggen and Logan, 2009). In both
muscles. Normally, corticospinal fibers contact alpha motoneu- models of response inhibition, the participant does not know in
rons mono-synaptically. However, while this organization works advance whether he/she will have to withhold the motor com-
for hands and fingers, it probably does not hold for other body mand in the Go/No-Go paradigm. When requested to mentally
segments. To result in a movement, the influence of propriospinal imagine a movement, the participant is clearly aware that no com-
neurons upon motoneurons has to be excitatory. The intrinsic mand should be transferred to peripheral effectors when he/she
organization of the spinal cord enables movement production is requested to mentally imagine the action. Inhibition does not
including several inhibitory systems such as the short feedback rely on the same process in both conditions. In the stop paradigm,
system from the Renshaw cells or the reciprocal inhibition reflex response inhibition depends on triggering No-Go signal. Once the
system. These processes are beyond the scope of the present paper, Go or No-Go stimulus is perceived, the participant should decide
however. We thus hypothesize that motor inhibition during MI is to act or to inhibit action, taking into account the information
mainly related to the central but not to the peripheral system. provided. Although action vs. inhibition could simultaneously
Logan (1983) conducted several studies on the degree to which be prepared as an alternative response, this is an all or nothing
people inhibit the thoughts that underlie their actions when they process. There is no such uncertainty during MI. However, the
inhibit action. Participants were requested to make category and inhibition of movement may be total or partial and may also
rhyme judgments about words and were given stop signals that take even several intermediate degrees. In other words, the par-
required them to inhibit the actions they executed to express their ticipant could nevertheless accompany the mental representation
judgments. They pressed one key if the word was a member of the by residual execution, e.g., miming partially some significant steps
category or rhymed with a target, and pressed another key if the of execution or movement rhythm. The other main difference
word was not a member of the category or did not rhyme. If a is related to the time course of these processes. On the basis of
stop signal occurred, they were supposed to inhibit the response. event-related brain potentials, EMG recordings, and continuous
These researchers then presented the materials again to test par- behavioral response measures, experimental data from de Jong
ticipants’ memory for words whose responses were inhibited, and et al. (1990) evidenced that responses could be interrupted at any
used recognition memory judgments: they presented both words time. Thus, actual movement is inhibited as early as the stop signal
for which that they had made or not judgments about, and asked is triggered whereas MI could accept several conditions from no
them to indicate whether words had been presented before. Sub- movement at all until residual movements related to actual move-
sequently, Logan (1985) used repetition priming to test memory ment that accompany and facilitate mental representation. This
by presenting similar kind of words, and then asking whether may explain why MI could also keep some elements of motor exe-
response time was faster for old words than for new ones. Both cution during mental representation. However, and with reference
studies revealed that thoughts went on to completion when actions to the casual definition of MI, we should wonder whether we could
were inhibited, suggesting that mental activity was independent still call this process “imagery” when associated with residual parts
from the motor response. As far as we consider MI, mental activity of movements.
is directly related to action, thus suggesting two related processes Finally, central inhibition processes are well-summarized by
differing from the relationships highlighted by Logan (1983, 1985). Garavan et al. (2002) who postulated two main neural networks
As previously mentioned, de Jong et al. (1990) described two mediating inhibition. Right dorsolateral prefrontal and right infe-
inhibitory mechanisms that could work to withdraw actual motor rior parietal areas are associated with response inhibition while a
command: inhibition of central activation and inhibition of trans- region of the cingulate cortex is involved in “difficult” inhibitions.
mission of motor commands from central to peripheral structures. Left dorsolateral prefrontal cortex was activated when subjects
adjusted their ongoing behavior in response to an error or to speaking, is it valid to describe MI performed with associated
unexpected changes in the environment. With regard to selective movement as “MI”? Although this question goes beyond the scope
inhibitory mechanisms, it is therefore not surprising that resid- of the present paper, it is important to remember that specific
ual muscle activity remains observable during MI, through EMG subliminal muscle activity is detectable during MI of any given
recordings, as described in preceding paragraphs. This is compat- movement (Guillot et al., 2007). We would also point out that
ible with both hypotheses previously described: On the one hand, the theoretical mechanisms we described above have the poten-
motor commands could be inhibited at any time and in different tial to explain how inhibition works during MI. In particular, the
ways. On the other hand, MI could be dependent upon the central inhibition process could occur at every stage of the represented
process of inhibition only. Overall, we should point out again that action: complete inhibition during MI would mean that actual
inhibition of the actual command, based on Go/No-Go designs, movement is entirely removed from MI (this corresponds to the
does not exactly correspond to that during MI, mainly because this usual definition which is often given, “MI is the mental represen-
inhibition is not under the control of decision making under time tation of an action without any overt movement ”). The hypothesis
pressure. Therefore, a specific cerebral organization might con- of partial motor inhibition could also be invoked, and as previ-
trol motor commands inhibition during MI, nevertheless sharing ously mentioned, we may combine movements to their mental
most features of the central processes inhibition we previously representation, even if they are only partially outlined.
described. How does MI affect motor commands? Performing MI might
In summary, the issue of inhibiting motor performance can activate somatic and autonomic motor commands differently.
be explained by two theoretical models. We first assume that MI From the intention to act, direct voluntary commands are nor-
results in a subliminal activity of the motor system. As postulated mally transmitted through the pyramidal tract to elicit movement.
by Jeannerod and Frak (1999), the motor system is involved not The process of an incomplete inhibition that accompanies MI may
only in the production of movements, but also in the mental rep- be viewed at organizing peripheral effectors during the prepa-
resentation of action. The authors extended its function to the ration phase of the forthcoming actual execution. Duclos et al.
process of learning by observation, even until understanding the (2008) provided evidence of anticipatory changes in patterns of
behavior of others. Therefore, if we consider MI as a subliminal human motoneurons discharge during motor preparation. This
motor command, it will not cause muscle activity and there is may also be observed during MI. Conversely, Bonnet et al.’s (1997)
no need for active inhibition process. The functional similarity view is that MI should be compared to action, rather than to motor
between actual movement and MI comes from the identity of the preparation, hence considering MI as the intention to avoid move-
motor structures that are believed to control them. Thus, the only ment execution, although MI might be more closely related to
difference between actual execution and its mental representation pre-executive processes of a movement than its actual execution
would be the degree of mobilization of motor commands: the itself (Hanakawa et al., 2008). Michelon et al. (2006) claimed that
preparatory phase would be common to actual action, its men- the MI process does not necessarily require a motor simulation
tal representation, and the consequences of action both in terms which would integrate the mapping of the effector-specific com-
of sensations generated and knowledge of result (did the action mands required to achieve the movement. This would be a quite
reach its goal?). In this regard, Macuga and Frey (2012) recently different MI as that followed by actual execution. The close rela-
postulated that the neural representations of observed, imagined, tionship (temporal, structural) between MI and actual execution
and imitated actions were dissociable and hierarchically organized. appears to favor an upstream organization of inhibition, implying
The differential activity among these three conditions favored an the behavioral inhibition system. This would also explain why the
alternative hierarchical model in which these behaviors rely on most automated parts of movement commands are not inhibited,
partially independent mechanisms. This result might challenge as they are controlled at the subcortical level. Nevertheless, Bonnet
the hypothesis of complete similarity between actual movement et al. (1997) and Jeannerod (2006) stated that the inhibitory mech-
and MI, and therefore favor the second hypothesis. anisms may also be localized downstream of the motor cortex,
More pragmatically, it is easy to experience motor representa- perhaps at the spinal cord, or brainstem level. In addition, Lotze
tion in association with movements or sequences of movements, et al. (1999b) postulated that the posterior cerebellum might also
more or less related to the imagined action and supposed to accom- play a crucial role in the inhibition of the motor command. There
pany and facilitate MI. One of the most remarkable examples is are probably several systems and processes of motor inhibition,
when observing some elite athletes during the preparation phase, coordinated at different levels of the central nervous system from
just before competing. For example, some skiers prepare for races the premotor cortex to the spinal cord. The question of a selective
by closing their eyes and mentally rehearsing the course that they inhibition remains to be asked. It could explain the subliminal
are about to traverse (Louis et al., 2012). Although they probably muscle activity and even the somatic commands addressed to the
do not experience the entire course, some portions are neverthe- low levels of the CNS (e.g., controlling postural regulations).
less mentally rehearsed using symbolic limb movements – which Several experimental data provide evidence of incomplete inhi-
enable them to mime the represented action. These movements of bition of the motor commands addressed to the different effectors.
arms and hands symbolize the turns and the timing at which these This finding should be used in the field of clinical rehabilitation
should be done. Finally, as revealed by Lorey et al. (2009), we are (whatever the nature of the neurological damage either periph-
all familiar with pictures of athletes moving while imagining their eral or central), MI use is known to benefit to functional recovery
subsequent performance during pre-performance routines. The- (e.g., Braun et al., 2006; Zimmermann-Schlatter et al., 2008). Fur-
oretically, such phenomena raise an interesting question. Strictly ther research should also investigate the processes of somatic and
autonomic motor commands inhibition during MI. So far, two Imagery studies on stroke patients are largely consistent with
different mechanisms are thought being involved in the inhibition results obtained from studies in healthy subjects – showing that
of a voluntary action. The first is related to central programming MI is a dynamic state of motor processing, reproducing the fea-
processes whereas the second is responsible for central motor com- tures of CNS activity in a similar way to that during actual motor
mand transmission from central structures to peripheral effectors performance. Examining MI ability after brain lesions can thus
(de Jong et al., 1990). As early hypothesized by Jeannerod (1994), contribute to understanding of the neural processes mediating
the neural commands for muscle contractions may be blocked at actual motor performance. Sirigu et al. (1996) reported that mis-
some level of the motor system by active inhibitory mechanisms. matching between MI and PP times characterized the parietal
This purpose is, however, associated with incomplete inhibition of brain lesion in stroke patients. The authors inferred that the pari-
the motor command that would provide a consistent explanation etal cortex might play a key role in elaboration of movement
for the recording of muscle activity during MI. representation during motor preparation. More recently, Stinear
et al. (2007) found that right hemisphere stroke patients overesti-
CLINICAL EXPERIMENTS mated MI duration as compared to PP, while left hemisphere stroke
Earlier in this paper, we reviewed evidence that motor com- patients achieved more accurate temporal congruence between
mands are involved during MI before being inhibited. In this actual and imagined time. These data support hemispheric speci-
section, we discuss the extent to which clinical data obtained ficity with regard to internal generation of the temporal parame-
from patients with stroke, PD and also from amputees and those ters of actual execution. As the temporal characteristics of MI are
with SCI provide further evidence on this issue. Firstly, analyz- also affected by lesions, these results corroborate previous find-
ing central activity during MI in patients suffering from central ings with regard to the neural substrates mediating movement
nervous system disorders reinforces the postulate of analogous preparation during MI.
central processing between MI and motor performance (Table 2). To summarize, studies of MI ability in stroke patients show
MI interventions can thus be used to improve motor process- that MI and PP share common neural substrates and involve
ing after various cases of neurological disorders (Sharma et al., similar motor commands. But to what extent do imagery data
2006; Dickstein and Deutsch, 2007). Secondly, motor impair- obtained from stroke patients shed light on the issue of motor
ments due to neurological diseases are reflected by changes in inhibition during MI? As MI ability is usually preserved following
MI ability. This conclusion holds in most clinical populations, trauma to the nervous system, one should assume that inhibition
and therefore validates the assumption that MI reproduces actual remain possible even after cerebral damage. As several studies have
motor performance states at the CNS level – yet without going reported that MI ability is preserved after brain damage affecting
to completion, hence the hypothesis of motor inhibition. Eventu- both cortical and subcortical motor networks, we may postulate
ally, changes in MI ability after neurological disorder provide new that cerebral structures mediating motor control do not play a
perspectives to the study of motor command inhibition during critical inhibitory role. This assumption would be congruent with
MI. some TMS findings assuming that there is no specific suppressive
mechanism occurring at the brain level to inhibit motor output
STROKE during MI, which would rather be caused by an incomplete level
Motor imagery ability seems preserved in most cases follow- of CNS facilitation during MI (i.e., the level of excitability would
ing stroke (Johnson et al., 2002; Sabate et al., 2007; Malouin not reach the motor threshold during MI, contrary to during PP).
et al., 2008). However, MI accuracy impairments were described Nonetheless, Schwoebel et al. (2002) reported the case of a stroke
in stroke and brain injured patients. These changes support hemiparetic patient with bilateral parietal brain lesions around the
the assumption that, during MI, the CNS reproduces a state primary somatosensory cortex, who fully executed the “imagined”
of actual motor processing. Indeed, MI ability changes mirror actions. As the patient performed more efficiently the demanded
those observed during actual motor performance. For instance, motor act during MI than during PP, authors argued that overt
the time required to perform mental rehearsal of actions involv- movements during MI reflected overt processing of the forward
ing impaired limbs increased compared to that of actions per- models for overt actions. These would be preserved and recalled
formed with spared effectors (e.g., Decety and Boisson, 1990; during MI. Further, the authors nicely demonstrated that sen-
Sirigu et al., 1995; Wu et al., 2010; Dettmers et al., 2012). Stroke sory integration, mediated by the primary somatosensory cortex,
patients also reported decreased imagery vividness during imag- distinguished between overt and covert performance. Impaired
ination of movements performed with the affected side in the sensory integration due to brain lesion thus explained both accu-
case of lateralized brain lesions (Malouin et al., 2008). Malouin rate actual executions during MI and altered voluntary motor
et al. (2004) suggested that temporal uncoupling between MI performance during PP. Critically, this case report indicated that
and PP could also occur during mental simulation of actions the motor command was effectively built up during MI, and may
involving the stroke-unaffected side. Nonetheless, most findings normally be actively inhibited throughout motor processing by
in stroke patients support the assumption that MI mirrors motor specific interactions between sensorimotor brain regions. While
impairments resulting from cerebral damage (Table 2). Incon- the precise mechanisms underlying this effect remain unclear, the
sistencies regarding MI ability changes after stroke (i.e., specific hypothesis that sensory feedback integration may be a key com-
or non-specific to the motor impairment) might account for ponent for efficient motor suppression during MI is supported
the nature and localization of the stroke lesion (Liepert et al., by neuroimaging findings (Solodkin et al., 2004; Alkadhi et al.,
2012). 2005).
Table 2 | Studies looking at the motor command during motor imagery.
Study Sample of patients Methodology Result regarding MI ability changes and potential implications for understanding
and controls motor command processing and/or subsequent motor inhibition
Alkadhi et al. SCI (n = 8) fMRI MI recruited the neural networks subserving MI and actual movements in healthy controls.
(2005) Controls (n = 8) Primary motor cortex activity during MI in patients was activated to the same extent than
during actual practice in healthy controls, suggesting weakened inhibition
Cramer et al. SCI (n = 12) fMRI No task modulation in cerebral activity between MI and PP. Reduced activation volumes in
(2005) Controls (n = 12) the primary sensorimotor cortex and increased activity within the primary sensorimotor
cortex during MI reflect brain function changes after SCI
Gustin et al. SCI (n = 11) fMRI Contrary to controls, MI elicited activity within the primary motor area and several brain
(2010) Controls (n = 19) regions included in the pain neuromatrix. Activity correlated to pain perception during MI
Hotz- SCI (n = 9) fMRI Cerebral activity during attempted and imagined movement supports motor program
Boendermaker Controls (n = 12) preservation. Recruitment of additional brain regions during MI (compared to healthy
et al. (2008) controls) reflects altered sensorimotor integration
Lacourse et al. SCI (n = 19) EEG Isomorphic electrophysiological correlates during MI and attempted execution in SCI
(1999) Controls (n = 10) patients, but not in healthy controls. Weakened inhibitory mechanisms as a consequence
of SCI, due to deafferentation
Olsson (2012) SCI (n = 1) fMRI Changes in MI ability according to the remaining capabilities of the motor system
Controls (n = 8)
Battaglia et al. Stroke (n = 8) TMS Reduced corticospinal facilitation supporting that unilateral stroke patients have lateralized
(2006) Controls (n = 10) MI deficits
Daprati et al. Stroke (n = 32) Mental Impaired MI ability. Patients may have developed MI strategies independently from the
(2010) Controls (n = 12) rotation actual state of the motor system
Decety and SCI (n = 4) Mental Contrary to SCI patients, stroke patients presented longer MI times when engaging the
Boisson (1990) Brain injury (n = 6) chronometry paralyzed upper/lower limb, comparing to MI of actions with unaffected limbs. For
movements that could be physically executed, patients achieved the temporal congruence
between MI and executed actions
Dettmers et al. Stroke (n = 31) Mental MI ability is impaired on the affected side of the lesion, specifically after stroke eliciting
(2012) Tetraparetic chronometry deafferentation. In both clinical populations, the features of MI ability reflect the actual
(n = 10) KVIQ state of the motor system
VMIQ
Gonzalez et al. Stroke (n = 11) Mental Higher MI and PP times in patients who recovered from stroke than in healthy controls
(2005) Controls (n = 11) chronometry
Kagerer et al. Brain injury (n = 4) Mental Patients exhibited longer MI and PP times than for actions involving the more affected side
(1998) Controls (n = 4) chronometry with preserved temporal congruence between MI and PP
Kimberley Stroke (n = 10) fMRI Cerebral activity during MI reflects the ipsilateral control of the stroke-affected hand, a
et al. (2006) Controls (n = 10) common plastic brain change after lateralized stroke lesions
Liepert et al. Stroke (n = 20) TMS MI ability impaired for movements involving the stroke-affected hand, but only in patients
(2012) Mental suffering from a somatosensory brain lesion compared to patients with “pure motor
chronometry strokes”
Malouin et al. Stroke (n = 32) KVIQ Patients obtained better scores when MI concerned the unaffected side of the lesion, but
(2008) Controls (n = 32) only for MI of lower limb actions
Sabate et al. Stroke (n = 9) Mental Decreased movement velocity during PP also observed during MI. The
(2004) Controls (n = 10) chronometry hemispheric-dependent effects of lateralized stroke on the actual motor performance of
each hand (affected/non-affected) was reproduced during MI
Sabate et al. Stroke (n = 33) Mental Strong correlation between MI and PP times after stroke. Mismatches between MI and PP
(2007) PD (n = 8) chronometry times support that changes in MI ability reflect the actual state of the motor system
Controls (n = 18)
(Continued)
Table 2 | Continued
Schwoebel Stroke (n = 1) Motor tasks A patient with bilateral parietal brain lesion fully executed the mentally rehearsed actions.
et al. (2002) Mental Inhibition during MI was impaired, presumably due to disturbances within a fronto-parietal
rotation circuit mediating motor inhibition
Sirigu et al. Stroke (n = 1) Mental Temporal parameters of MI predicted that of PP in a variety of situations, altogether
(1995) chronometry reflecting hypokinesia after unilateral stroke
Sharma et al. Stroke (n = 8) fMRI Abnormal functional connectivity patterns within the motor network during MI correlated
(2009a) Controls (n = 13) with motor outcome after stroke recovery
Sharma et al. Stroke (n = 20) fMRI During MI of the affected hand, activation of the anterior subdivision of cM1 was similar to
(2009b) Controls (n = 17) that during PP, and activity of the ipsilesional posterior subdivision of M1 correlated with
motor performance. The result support that MI reveals the actual state of the motor
system after stroke
Stinear et al. Stroke (n = 12) Mental Absence of corticospinal facilitation during MI in the stroke-affected hand
(2007) Controls (n = 8) chronometry
TMS
Szameitat Stroke (n = 5) fMRI Cortical activations during MI resemble that during attempted overt execution within
et al. (2012) Controls (n = 21) sensorimotor and premotor cortices. Potential analogous involvement of the sensorimotor
system in the two tasks
Vromen et al. Stroke (n = 21) Mental Stroke patients (n = 20) without spatial neglect outperformed a patient (n = 1) with spatial
(2011) rotation neglect during a visual mental rotation task involving the upper limb
Wu et al. Stroke (n = 18) Mental Longer times required to imagine upper limb actions involving stroke-affected effectors
(2010) chronometry
Cohen et al. PD (n = 24) Mental Temporal discrepancies between times required to imagine and actually walk through a
(2011) Controls (n = 10) chronometry narrow doorway characterized PD patients with freezing of gait syndrome
Cunnington PD (n = 14) EEG Impaired motor preparation, while potentials associated with motor execution seemed
et al. (1997) Controls (n = 10) relatively preserved
Cunnington PD (n = 6) PET Reduced pre-SMA activation and compensatory brain activity during MI altogether
et al. (2001) Controls (n = 3) characterized the motor deficit in PD patients
Dominey et al. PD (n = 7) Mental Asymmetrical slowing of MI times according to the affected side in lateralized PD patients,
(1995) Controls (n = 7) chronometry hence supporting that MI and PP shared common neural structures
Helmich et al. PD (n = 19) fMRI MI involving the affected side in lateralized PD patients recruited additional cognitive
(2007) resources compared to MI involving the unaffected side
Helmich et al. PD (n = 38) fMRI Distinct sensorimotor processing at the subcortical level during MI characterized patients
(2012) Controls (n = 19) with and without resting state tremor
Heremans PD (n = 14) MI question- MI ability was preserved in PD patients, but was performed more slowly than in healthy
et al. (2011) Controls (n = 14) naires controls
Mental
chronometry
Kuhn et al. PD (n = 8) EEG Analogous contribution of subthalamic nucleus to feedforward organization during MI and
(2006) PP of wrist actions. Electrophysiological correlates of MI within these structures support
its role in sensory feedback integration for overt motor and postural regulations after PD
Samuel et al. PD (n = 6) PET MI yielded decreased activity in frontal areas (dorsolateral frontal cortex), hence reflecting
(2001) Controls (n = 6) impaired motor preparation in PD patients as compared to healthy controls
Thobois et al. PD (n = 8) PET MI elicited reduced activations for movement with the affected side in lateralized PD
(2000) Controls (n = 8) patients. MI of the unaffected side was impaired, but to a lesser extent
(Continued)
Table 2 | Continued
Thobois et al. PD (n = 7) PET Subthalamic nucleus stimulation analogously improved cerebral activity during MI and PP
(2002)
Diers et al. Amputees fMRI MI activated different neural substrates depending on whether amputee patients
(2010) (n = 14) experienced phantom limb pain or not. MI activated the contralateral primary sensorimotor
Controls (n = 9) cortex only in non-pain patients
MacIver et al. Amputees fMRI MI training elicited reversed sensorimotor plasticity in amputee that corresponded to
(2008) (n = 13) decreased phantom limb pain symptoms
Controls (n = 6)
Marconi et al. Amputees (n = 8) TMS MI mirrored sensorimotor reorganizations in the patients. Upper/lower limb inhibitory
(2007) Controls (n = 9) relationships within cM1 might be removed after amputation
Nico et al. Amputees Mental MI was affected by amputation in patients as compared to healthy controls. Selective MI
(2004) (n = 16) rotation task impairments were observed according to whether amputation affected the
Controls (n = 7) dominant/non-dominant limb
Raffin et al. Amputees fMRI Partially overlapping, albeit non-identical, neural networks mediating MI, and attempted
(2012b) (n = 14) physical practice with the phantom limb
PARKINSON’S DISEASE from MI to PP with regards to actual motor processing (Mannix

MI ability changes in patients suffering from PD are consistent et al., 1999; Lim et al., 2006; Braun et al., 2011). Tamir et al. (2007)
with those observed in stroke patients. Central processing during reported the adjunctive benefits of MI training in motor rehabilita-
MI is selective according to the limbs affected by PD and mir- tion following PD. MI practice contributed to reduce bradykinesia
rors the actual motor impairment (Dominey et al., 1995; Helmich (Subramanian et al., 2011). PD typically refers to basal ganglia dys-
et al., 2007), even though MI of non-affected body regions may function. These structures are known to play a role in overt motor
also be disturbed to a lesser extent (Thobois et al., 2000). In spite performance inhibition in healthy subjects, through specific neural
of basal ganglia dysfunction, MI ability is preserved in early and interactions with cM1 during the early stages of motor process-
mid-stage PD patients (Heremans et al., 2011). However, several ing (Stinear et al., 2009). However, it remains unclear whether
neuroimaging studies have discovered abnormal brain activation these structures also participate in motor inhibition during MI,
patterns during MI in PD patients as compared to healthy con- but recent findings suggest that PD patients lose the ability to
trols (Cunnington et al., 1997; Samuel et al., 2001). Specifically, elicit corticospinal facilitation during MI (Tremblay et al., 2008).
reduced premotor and sensorimotor activations, as well decreased The authors assumed that the patients failed to involve the motor
cerebellum activation during MI were reported (Thobois et al., system during MI. Nonetheless as the MI ability seems preserved
2000; Cunnington et al., 2001; Samuel et al., 2001). Compensatory and to elicit sensorimotor activity at the brain level in most neu-
activations occurring during MI of actions involving affected effec- roimaging studies, it can be hypothesized that changes in basal
tors were also reported in PD patients, which could reflect the ganglia activity during MI could increase inhibitory interactions
actual motor deficit (Cunnington et al., 2001; Thobois et al., 2002; during MI. This postulate remains a working hypothesis awaiting
Helmich et al., 2007). Of great interest, however, is the finding experimental proofs.
that subthalamic nucleus electrical stimulation enabled reduction
of compensatory activations during MI in PD patients (Thobois AMPUTEES
et al., 2002). These data provide meaningful evidence that elab- Consistent reorganizations of the cortical sensorimotor map occur
oration of motor command does, in fact, occur during MI after after limb amputation, due to neuroplasticity (i.e., the capability
PD, as the central MI activity seems to mirror the effect of PD on of synapses to adapt their structure and function in response to
actual motor performance (Cunnington et al., 1997; Samuel et al., environmental and behavioral demand). The expansion within
2001; Tremblay et al., 2008). Spontaneous eye movements occur primary sensorimotor cortices of the cortical surface correspond-
during MI and resemble those occurring during PP (Heremans ing to unaffected body parts toward the adjacent deafferented
et al., 2008). Heremans et al. (2012) observed that external visual and deefferented areas (i.e., corresponding to the missing limb)
cueing even reinforced MI accuracy in PD patients, thus confirm- is now well-established (Knecht et al., 1996; Pascual-Leone et al.,
ing that the central processing of somatic motor signals was an 1996; Karl et al., 2001; Ramachandran et al., 2010). Nonethe-
intrinsic component of the MI experience. Finally, several stud- less, the motor system preserves the ability to process central
ies evidenced that PD patients may benefit from MI training in commands controlling the missing limb following amputation,
the rehabilitation of motor disorders, which suggests skill transfer as suggested in several neuroimaging studies reporting similar
sensorimotor activations during actions performed either with phantom, as no muscle activity could occur (a similar observation
the phantom limb or with the contralateral unaffected one (Ers- can be made in SCI patients – see below). If MI does involve
land et al., 1996; Roux et al., 2001, 2003). EMG recordings at the active motor suppression at the CNS level, these mechanisms will
level of the stump revealed specific motor commands suggest- no longer be relevant after amputation and might potentially be
ing preservation of motor programs controlling the amputated reshaped by the emergence of a new body schema, prompting
limb (Reilly et al., 2006). Are motor commands processed during considerable neural reorganization. Challenging considerations to
MI of actions with the phantom limb? Recent findings suggest this approach comes from the fact that amputee patients usually
that the neural networks mediating MI and PP of actions with preserve a perceptual representation of their missing limb. Fur-
the phantom limb in amputee patients consistently overlap, in ther, no study yet reported weakened inhibition during MI after
spite of significant differences (Raffin et al., 2012b). Most findings amputation and no EMG activity was recorded at the level of
regarding imagery in these patients are derived from studies inves- the stump (Raffin et al., 2012a). This is an interesting perspec-
tigating the therapeutic management of phantom limb pain (i.e., tive for future studies: If there is no changes in the ability to
a frequent disabling consequence following amputation; Shukla inhibit the motor commands during MI, conclusions regarding
et al., 1982; Ehde et al., 2000). A causal relationship was found the neural underpinnings of motor suppression during MI might
between sensorimotor reorganizations and phantom limb pain be drawn.
(Flor et al., 1995; MacIver et al., 2008). Expansion of homunculus
regions corresponding to spared body parts toward deafferented SCI PATIENTS
areas (e.g., hand region “invaded” by regions corresponding to face Motor activity during MI can be inferred from the effects of MI
body parts), in both primary somatosensory and motor cortices, practice in neuropathic pain SCI patients. Gustin et al. (2008)
characterize phantom limb pain patients as compared to pain-free observed that MI increased neuropathic pain intensity in six SCI
amputees (Lotze et al., 1999a, 2001; Karl et al., 2001). Interest- patients out of seven during mental rehearsal of actions involving
ingly, involving the motor system into overt motor processing (i.e., infra-lesional effectors. Gustin et al. (2010) delineated the neural
attempted executed actions) with the phantom limb (e.g., using substrates mediating this pain response to MI. They observed sev-
visual feedback) tends to produce pain relief (Chan et al., 2007; eral activations within the pain neuromatrix network, correlated
Mercier and Sirigu, 2009). An interesting practical implication of to increased pain perception when SCI patients performed MI.
this finding is that if MI reproduces overt motor processing states Both the ipsilateral premotor cortex and the SMA participated
at the CNS level, then MI training might also be efficient in the to actual motor processing. When compared to healthy controls,
management of neuropathic pain after amputation. MacIver et al. SCI patients elicited greater activation within cM1 attesting that
(2008) investigated the effects of a 6-week MI training program motor output during MI reached the circuitry underlying pain
on phantom limb pain. fMRI scanning sessions were performed response, due to cerebral reorganizations after the neurological
before and after the experimental intervention. MI training signif- lesion. Therefore, it seems reasonable to assume that motor sig-
icantly reduced pain symptoms. fMRI investigations highlighted nals do occur during MI, and might act as a triggering stimulus for
a reversed plasticity as compared to plastic changes due to ampu- pain in specific clinical cases of neurological disorders (see above
tation observed in the patients during the pretest. These results for considerations in amputee patients).
promote the therapeutic relevance of MI in the rehabilitation As in the case of stroke, PD, and amputees patients, research
of pain disorders, presumably due to actual motor processing at investigating MI ability after SCI provides further insight into
the CNS level. Indeed, change in the cortical representation of a central processing of motor signals during MI. Using men-
body segment is usually achieved through repetitive motor prac- tal chronometry, Decety and Boisson (1990) observed that SCI
tice (Wolf et al., 2002). In the study by MacIver et al. (2008), MI patients achieved close temporal congruence between MI and
elicited reversed plasticity, hence supporting the assumption that PP, whereas brain injured patients failed to do so. These data
amputees are able to process motor commands during MI, even support the theory that MI accuracy is preserved following SCI
for actions with their injured limb. In this regard, however, the because, contrary to what happens after stroke, SCI does not result
fact that phantom limb pain patients and those with and non- in cerebral damage. However, SCI elicits consistent reorganiza-
phantom limb pain present different brain responses to MI is of tion of the sensorimotor cortical maps, with changes in neuronal
special interest (Diers et al., 2010). Only non-pain patients acti- excitability (Topka et al., 1991; Curt et al., 2002; Dunlop, 2008;
vated the contralateral primary sensorimotor cortex during MI. Kokotilo et al., 2009) due to deafferentation and deefferentation
Some authors have reported that neuropathic pain after ampu- (Bruehlmeier et al., 1998). As MI is mediated by cerebral substrates
tation could be due to the mismatching between motor output overlapping with motor-related regions reorganized consecutively
and sensory feedback (Mayer et al., 2008), in agreement with the to the lesion, plastic changes after SCI are likely to affect MI
model of pathological pain by Harris (1999). MI might there- ability. Cramer et al. (2005) reported altered MI processing in
fore reproduce a pathological state where the motor system fails SCI patients as compared to healthy controls during MI of right
to elicit the primary sensorimotor activity demand of the cogni- foot movements. In this study, task modulation in central activity
tive task. These findings concur with a large body of literature between MI and PP was absent in SCI patients whereas different
supporting the role of differential neuroplasticity in the gener- brain activation patterns mediated the two tasks in healthy con-
ation of pain symptoms, which seems reproduced during MI. trols. Similarly, Alkadhi et al. (2005) reported that during MI, SCI
Further, it can be assumed that amputees no longer require to patients recruited those neural networks that typically mediate
inhibit the motor command during MI of actions involving the both MI and PP in healthy controls. As already mentioned, MI
usually results in lower activation intensities than PP in healthy the sensorimotor cortex (allowing the reshaping of actual motor
participants, which is typically imputable to inhibitory processes performance) may be reflected during MI, hence confirming some
(Porro et al., 1996). Therefore, as early hypothesized by Lacourse shared neural substrates between these two processes.
et al. (1999), deafferentation and deefferentation following SCI Taken together, clinical data from SCI, stroke, amputees, and PD
may result in weakened motor command inhibition during MI. patients converge to suggest that central patterns elicited during
SCI patient no longer require to inhibit the mentally rehearsed MI effectively reflect the internal elaboration of motor commands,
movement due to the spinal cord lesion, which prevents neural although specific clinical impairment provides different and com-
transmissions toward peripheral effectors. This clinical topic is plementary insights to our current knowledge regarding motor
different than that in amputees (see above), in that after SCI inhibition during MI.
patients no longer feel their deafferented and deefferented body
regions. Alkadhi et al. (2005) suggested that deafferentation may CONCLUSION
be a key component for adaptive brain changes regarding inhibi- In this paper, we reviewed data that support the emerging hypoth-
tion during MI, hence matching the conclusions by Schwoebel et al. esis that both central and peripheral neurophysiological correlates
(2002). In a recent magnetoencephalographic control-case study of MI tightly resemble those elicited by actual practice of the same
(Di Rienzo et al., submitted), a SCI patient presented remarkably task, even in the absence of overt movement production. Such data
similar activations within cM1 during both MI and PP, hence sug- also suggest that motor commands are involved during MI. Fur-
gesting weakened inhibition of this area during MI. This pattern thermore, we highlighted evidence that the isomorphism between
of activation was associated with disturbed functional network of the representation of imagined and executed actions is preserved, if
interrelations between cM1, S1, and SMA. By contrast, a healthy not strengthened, in specific cases of neurological disorders. Neu-
age-matched control participant presented a significant reduction roimaging studies clearly support the involvement of both primary
in cM1 activation during MI, with significant inter-relationships in and secondary motor-related areas during MI, hence suggesting
neural activities between cM1 and both SMA and S1. Interestingly, that neural impulses for motor commands may be elaborated at
both S1 and SMA are thought to play a key role in motor sup- the cerebral level and addressed, at least partially, from cM1 to
pression during MI (Schwoebel et al., 2002; Solodkin et al., 2004; the anterior part of the spinal cord via the descending pathways.
Alkadhi et al., 2005; Kasess et al., 2008). Hotz-Boendermaker et al. TMS studies of MI also confirmed this assumption and elucidated
(2008) discovered that patients remained nonetheless able to sub- our understanding of the neurophysiological processes mediat-
jectively distinguish MI from attempted PP for actions involving ing the involvement of cM1 during MI. Accordingly, corticospinal
infra-lesional effectors, but the authors mentioned that MI elicited facilitation during MI might result from changes in excitability
activation of additional brain regions in SCI patients in compari- at the cortical level. Some authors have also reported that resid-
son to controls – presumably to assist motor commands process- ual EMG activity during MI reflects the features of the motor
ing. Neuroimaging studies in SCI patients support the theory that commands.
central reorganization after SCI results in increased congruence Based on these issues, we should now consider the unresolved
between MI and PP, due to weakened inhibitory processes as a con- question of how motor commands are inhibited throughout the
sequence of deafferentation. Accordingly, motor inhibition during motor system to prevent overt execution during MI. We pos-
MI may have a cortical component involving specific interactions tulate that motor inhibition during MI may not result from a
between perirolandic sites and cM1. These results are complemen- parallel neurophysiological process, concomitant to MI, designed
tary to those from the TMS approach to understanding inhibition to prevent muscle contractions. First, TMS studies have shown
(i.e., intracortical inhibition within M1 and/or incomplete state of that MI produces opposite effects to those elicited by voluntary
CNS facilitation). However, central activity in SCI patients suggests relaxation of peripheral effectors regarding corticospinal facilita-
that MI recruits motor programs for overt movements with para- tion (Taniguchi et al., 2008). Secondly, neuroimaging studies have
lyzed effectors (Sabbah et al., 2002), thereby confirming that these failed to highlight specific neural structures mediating motor inhi-
patients preserve the ability to process motor command signals bition during MI, while TMS data support the idea of increased
during MI. neuronal excitability and reduced intracortical inhibition within
A discussed previously, neuroimaging studies support the likeli- cM1 during MI. A notable finding from neuroimaging research is
hood of weakened motor inhibition during MI after SCI. However, that secondary motor-related areas like the cerebellum and SMA
whether the reductions in sensorimotor activity between MI and might play a key role in motor output suppression during MI. Also,
PP result in greater transmission of neural signals to the descend- impaired sensory feedback integration following deafferentation
ing pathways during MI remains unknown. Roy et al. (2011) or brain lesions around the primary somatosensory cortex result
obtained evidence of the downregulation of intracortical inhi- in weakened inhibition during MI, thus promoting the role of
bition during MI after SCI using paired-pulse TMS. These data sensory sites in motor output suppression during MI. Therefore,
support previous findings reporting increased excitability after inhibition during MI may be a functional process resulting from
complete SCI in spared neural pathways (Topka et al., 1991). the specific contribution of neural sites usually dedicated to overt
Reduced intracortical inhibition after SCI is thought to enable motor processing. This theoretical stance might account for the
unmasking of latent synaptic connections at the cortical level, pro- fact that MI activates the motor system in a lesser extent to actual
viding a possible causal mechanism for cerebral plasticity after SCI practice.
(Saturno et al., 2008). As MI and PP are assumed to be functionally If we assume that motor inhibition may be intrinsic to the
equivalent, it is plausible that changes in intracortical inhibition of motor command during MI, and that it is potentially mediated by a
Table 3 | Studies addressing the question of the motor inhibition during motor imagery.
Authors Type of study Method Participants Potential inhibitory regions
Alkadhi et al. (2005) Motor imagery of foot movement fMRI Healthy (n = 8) Motor command suppression
Patients (n = 8) but no clear inhibitory regions
Bonnet et al. (1997) Motor imagery of a foot pressure on a pedal Reflex stimulation Healthy (n = 20) Inhibitory spinal influences
Di Rienzo et al. Case study with a C6–C7 quadriplegic patient MEG Patients (n = 1) Primary sensory area and
(submitted) supplementary motor area
Jeannerod (2001, Review papers – – prefrontal cortical areas and/or
2006) brainstem and spinal influences
Kasess et al. (2008) Motor imagery of finger movements fMRI Healthy (n = 8) Supplementary motor area
Lotze et al. (1999b) Motor imagery of hand movements fMRI Healthy (n = 10) Posterior cerebellum
Schwoebel et al. Case study with a patient suffering from Psychophysic Patients (n = 1) Fronto-parietal network
(2002) bilateral parietal lesions experiment
Solodkin et al. Motor imagery of a finger-to-thumb fMRI Healthy (n = 18) Superior parietal lobule and
(2004) opposition task supplementary motor area
Deiber et al. (1998) Motor imagery of finger movements TEP Healthy (n = 10) Inferior frontal cortex
Lebon et al. (2012) Motor imagery and mental rotation of a TMS Healthy (n = 11) Inferior parietal lobule
pinching movement
FIGURE 1 | The three possible routes of motor command inhibition during motor imagery.
highly specific interplay between motor-related neural structures, origin during MI. Similarly, it is vital to investigate whether or not
a promising focus of future research may be identified. Specifi- neural impulses elicited by MI are blocked by spinal mechanisms
cally, priority for future investigators will be to explore the extent triggered by descending input of cerebral origin. Interestingly,
to which cM1 is subjected to active inhibition during MI. Using inhibition of actual actions in decision making experiments (i.e.,
advanced statistical modeling, several authors have shown that Go/No-Go paradigms) or during motor control of complex skills
reduced cM1 activation during MI is related to the suppressive that require real-time adaptation to changing environmental con-
influence of other motor-related brain regions, thus suggesting straints, can happen in the very late stages, and even during overt
that motor inhibition during MI may also intervene at the early motor processing. We should also investigate to a greater extent
stages of motor planning (Kasess et al., 2008). how these inhibitory processes exactly work as these do not oper-
Based on current understandings and literature evidence, we ate under a principle of all or nothing. As mentioned by Band and
promote a multifactorial explanation of motor inhibition dur- van Boxtel, 1999, p. 190), “A crucial question is whether it is most
ing MI that might involve both cerebral and spinal mecha- important to define the locus of [motor] inhibition by the source of
nisms (Table 3). From these findings, we postulate that there are inhibitory activity (agent), by the process at which inhibition was
three possible routes of motor command inhibition during MI exerted (site), or by the location where reduction of response activity
(Figure 1). One may first hypothesize that motor inhibition is a can be recorded (manifestation).”An important aspect is that imag-
part of the imagery experience, hence only subthreshold motor ined movement may be accompanied by reduced motor activity
commands are sent to the effectors to prevent movement exe- at a level which the subject is unaware of, such as when volun-
cution. A second alternative is the possibility that the inhibitory tarily performing movements simultaneously with its own action
cerebral regions progressively weaken the motor command during representation for improving MI vividness. Therefore, as the func-
the time course of the MI process, so that only a residual activity tional equivalence between imagined and executed actions has
is sent and can be recorded in the corresponding muscles (for a mainly been considered at the scope of neurophysiological corre-
review on a similar chain of processes during inhibitory motor lates in activation, it might also be observable through the neural
control in No-Go paradigms, see Band and van Boxtel, 1999). processes mediating motor output suppression during these two
Finally, it is possible that downstream regions including brain- behaviors.
stem and spinal influences contribute to motor inhibition at a
later stage than in the case of the other two possibilities. There ACKNOWLEDGMENTS
is still, however, one key element requiring further experimental We thank Patrice Thiriet, Alix Poulot, and Emmanuel Bettler
investigation. Specifically, research should establish the degree to (Innovation Conception et Accompagnement en Pédagogie – ICAP –
which cM1 is inhibited by suppressive neural impulses of cerebral Université Claude Bernard Lyon 1) for the Figure template.
REFERENCES Battaglia, F., Quartarone, A., Ghilardi, Braun, S., Beurskens, A., Kleynen, M., limb pain. N. Engl. J. Med. 357,
Abbruzzese, G., Assini, A., Buccolieri, M. F., Dattola, R., Bagnato, S., Rizzo, Schols, J., and Wade, D. (2011). 2206–2207.
A., Marchese, R., and Trompetto, V., Morgante, L., and Girlanda, P. Rehabilitation with mental practice Clark, S., Tremblay, F., and Ste-Marie,
C. (1999). Changes of intracortical (2006). Unilateral cerebellar stroke has similar effects on mobility as D. (2004). Differential modulation
inhibition during motor imagery in disrupts movement preparation and rehabilitation with relaxation in of corticospinal excitability during
human subjects. Neurosci. Lett. 263, motor imagery. Clin. Neurophysiol. people with Parkinson’s disease: observation, mental imagery and
113–116. 117, 1009–1016. a multicentre randomised trial. J. imitation of hand actions. Neuropsy-
Alkadhi, H., Brugger, P., Boendermaker, Bird, E. I. (1984). EMG quantification of Physiother. 57, 27–34. chologia 42, 105–112.
S. H., Crelier, G., Curt, A., Hepp- mental rehearsal. Percept. Mot. Skills Bruehlmeier, M., Dietz, V., Leenders, Cohen, R. G., Chao, A., Nutt, J. G.,
Reymond, M. C., and Kollias, S. 59, 899–906. K. L., Roelcke, U., Missimer, J., and Horak, F. B. (2011). Freezing
S. (2005). What disconnection tells Bonnet, M., Decety, J., Jeannerod, M., and Curt, A. (1998). How does the of gait is associated with a mis-
about motor imagery: evidence from and Requin, J. (1997). Mental sim- human brain deal with a spinal match between motor imagery and
paraplegic patients. Cereb. Cortex 15, ulation of an action modulates the cord injury? Eur. J. Neurosci. 10, motor execution in narrow door-
131–140. excitability of spinal reflex pathways 3918–3922. ways, not with failure to judge door-
Aron, A. R., Durston, S., Eagle, D. in man. Brain Res. Cogn. Brain Res. Brunia, C. H. M. (2003). “How is way passability. Neuropsychologia 49,
M., Logan, G. D., Stinear, C. M., 5, 221–228. stopping realized?” in Errors, Con- 3981–3988.
and Stuphorn,V. (2007). Converging Boschker, M. S. J. (2001). Action- flicts, and The Brain. Current Opin- Collet, C., and Guillot, A. (2010).
evidence for a fronto-basal-ganglia Based Imagery: On the Nature ions on Performance Monitoring, eds “Autonomic nervous system activi-
network for inhibitory control of of Mentally Imagined Motor M. Ullsperger and M. Falkenstein ties during imagined movements,”in
action and cognition. J. Neurosci. 27, Actions. Amsterdam: University of (Liepzig: MPI of Cognitive Neuro- The Neurophysiological Foundations
11860–11864. Amsterdam. science), 96–103. of Mental and Motor Imagery, eds
Bakker, F. C., Boschker, M. S. J., Boucher, L., Palmeri, T. J., Logan, G. D., Callow, N., and Waters, A. (2005). A. Guillot and C. Collet (New York:
and Chung, T. (1996). Changes in and Schall, J. D. (2007). Inhibitory The effect of kinesthetic imagery Oxford University Press), 95–108.
muscular activity while imagining control in mind and brain: an on the sport confidence of flat-race Cramer, S. C., Lastra, L., Lacourse, M.
weight lifting using stimulus or interactive race model of counter- horse jockeys. Psychol. Sport. Exerc. G., and Cohen, M. J. (2005). Brain
response propositions. J. Sport Exerc. manding saccades. Psychol. Rev. 114, 6, 443–459. motor system function after chronic,
Psychol. 18, 313–324. 376–397. Carpenter, W. B. (1894). Principles complete spinal cord injury. Brain
Band, G. P. H., and van Boxtel, G. J. Braun, S. M., Beurskens, A. J., Borm, P. of Mental Physiology. Appleton: D. 128, 2941–2950.
M. (1999). Inhibitory motor con- J., Schack, T., and Wade, D. T. (2006). Appleton. Cunnington, R., Egan, G. F., O’Sullivan,
trol in stop paradigms: review and The effects of mental practice in Chan, B. L., Witt, R., Charrow, A. P., J. D., Hughes, A. J., Bradshaw, J. L.,
reinterpretation of neural mecha- stroke rehabilitation: a systematic Magee, A., Howard, R., Pasquina, P. and Colebatch, J. G. (2001). Motor
nisms. Acta Psychol. (Amst.) 101, review. Arch. Phys. Med. Rehabil. 87, F., Heilman, K. M., and Tsao, J. W. imagery in Parkinson’s disease: a
179–211. 842–852. (2007). Mirror therapy for phantom PET study. Mov. Disord. 16, 849–857.
Cunnington, R., Iansek, R., John- (2005). EMG activity in selected tar- Flor, H., Elbert, T., Knecht, S., Wien- theoretical investigation of motor
son, K. A., and Bradshaw, J. L. get muscles during imagery rising on bruch, C., Pantev, C., Birbaumer, imagery use. Int. Rev. Sport Exerc.
(1997). Movement-related poten- tiptoes in healthy adults and post- N., Larbig, W., and Taub, E. (1995). Psychol. 1, 31–44.
tials in Parkinson’s disease. Motor stroke hemiparetic patients. J. Mot. Phantom-limb pain as a perceptual Guillot, A., Collet, C., Nguyen, V.
imagery and movement prepara- Behav. 37, 475–483. correlate of cortical reorganization A., Malouin, F., Richards, C., and
tion. Brain 120(Pt 8), 1339–1353. Diers, M., Christmann, C., Koeppe, following arm amputation. Nature Doyon, J. (2008). Functional neu-
Curt, A., Alkadhi, H., Crelier, G. C., Ruf, M., and Flor, H. (2010). 375, 482–484. roanatomical networks associated
R., Boendermaker, S. H., Hepp- Mirrored, imagined and executed Fourkas, A. D., Avenanti, A., Urgesi, C., with expertise in motor imagery.
Reymond, M. C., and Kollias, S. movements differentially activate and Aglioti, S. M. (2006). Corti- Neuroimage 41, 1471–1483.
S. (2002). Changes of non-affected sensorimotor cortex in amputees cospinal facilitation during first and Guillot, A., Collet, C., Nguyen, V.
upper limb cortical representation with and without phantom limb third person imagery. Exp. Brain Res. A., Malouin, F., Richards, C., and
in paraplegic patients as assessed by pain. Pain 149, 296–304. 168, 143–151. Doyon, J. (2009). Brain activity
fMRI. Brain 125, 2567–2578. Dominey, P., Decety, J., Broussolle, Gandevia, S. C. (1999). Mind, mus- during visual versus kinesthetic
Daprati, E., Nico, D., Duval, S., and Lac- E., Chazot, G., and Jeannerod, cles, and motoneurones. J. Sci. Med. imagery: an fMRI study. Hum. Brain
quaniti, F. (2010). Different motor M. (1995). Motor imagery of Sport. 2, 167–180. Mapp. 30, 2157–2172.
imagery modes following brain a lateralized sequential task is Gandevia, S. C., Wilson, L. R., Inglis, Guillot, A., Di Rienzo, F., and Collet, C.
damage. Cortex 46, 1016–1030. asymmetrically slowed in hemi- J. T., and Burke, D. (1997). Men- (2012). “The neurofunctional archi-
de Jong, R., Coles, M. G., Logan, G. D., Parkinson’s patients. Neuropsycholo- tal rehearsal of motor tasks recruits tecture of motor imagery,” in Func-
and Gratton, G. (1990). In search gia 33, 727–741. alpha-motoneurones but fails to tional Magnetic Resonance Imag-
of the point of no return: the con- Driskell, J. E., Cooper, C., and Moran, recruit human fusimotor neurones ing/Book 1, eds T. D. Papageorgiou,
trol of response processes. J. Exp. A. (1994). Does mental practice selectively. J. Physiol. (Lond.) 505(Pt G. Christopoulos, and S. Smirnakis
Psychol. Hum. Percept. Perform. 16, enhance performance? J. Appl. Psy- 1), 259–266. (InTech) (in press).
164–182. chol. 79, 481–492. Gao, Q., Duan, X., and Chen, H. (2011). Guillot, A., Lebon, F., and Collet, C.
de Vries, S., and Mulder, T. (2007). Duclos, Y., Schmied, A., Burle, B., Evaluation of effective connectiv- (2010). “Electromyographic activity
Motor imagery and stroke rehabil- Burnet, H., and Rossi-Durand, C. ity of motor areas during motor during motor imagery,” in The Neu-
itation: a critical discussion. J. Reha- (2008). Anticipatory changes in imagery and execution using con- rophysiological Foundations of Men-
bil. Med. 39, 5–13. human motoneuron discharge pat- ditional Granger causality. Neuroim- tal and Motor Imagery, eds A. Guil-
Decety, J., and Boisson, D. (1990). Effect terns during motor preparation. J. age 54, 1280–1288. lot and C. Collet (New York: Oxford
of brain and spinal cord injuries on Physiol. (Lond.) 586, 1017–1028. Garavan, H., Ross, T. J., Murphy, K., University Press), 83–93.
motor imagery. Eur. Arch. Psychiatry Dunlop, S. A. (2008). Activity- Roche, R. A., and Stein, E. A. (2002). Guillot,A., Lebon, F., Rouffet, D., Cham-
Clin. Neurosci. 240, 39–43. dependent plasticity: implications Dissociable executive functions in pely, S., Doyon, J., and Collet, C.
Decety, J., and Grezes, J. (1999). Neural for recovery after spinal cord injury. the dynamic control of behavior: (2007). Muscular responses during
mechanisms subserving the percep- Trends Neurosci. 31, 410–418. inhibition, error detection, and cor- motor imagery as a function of mus-
tion of human actions. Trends Cogn. Ehde, D. M., Czerniecki, J. M., Smith, rection. Neuroimage 17, 1820–1829. cle contraction types. Int. J. Psy-
Sci. (Regul. Ed.) 3, 172–178. D. G., Campbell, K. M., Edwards, W. Gentili, R., Papaxanthis, C., and Pozzo, chophysiol. 66, 18–27.
Decety, J., Jeannerod, M., Durozard, T., Jensen, M. P., and Robinson, L. T. (2006). Improvement and gener- Gustin, S. M., Wrigley, P. J., Gandevia,
D., and Baverel, G. (1993). Cen- R. (2000). Chronic phantom sensa- alization of arm motor performance S. C., Middleton, J. W., Henderson,
tral activation of autonomic effec- tions, phantom pain, residual limb through imagery practice. Neuro- L. A., and Siddall, P. J. (2008). Move-
tors during mental simulation of pain, and other regional pain after science 137, 761–772. ment imagery increases pain in peo-
motor actions in man. J. Physiol. lower limb amputation. Arch. Phys. Gerardin, E., Sirigu, A., Lehericy, S., ple with neuropathic pain following
(Lond.) 461, 549–563. Med. Rehabil. 81, 1039–1044. Poline, J. B., Gaymard, B., Marsault, complete thoracic spinal cord injury.
Dechent, P., Merboldt, K. D., and Frahm, Ehrsson, H. H., Geyer, S., and Naito, E. C., Agid, Y., and Le Bihan, D. Pain 137, 237–244.
J. (2004). Is the human primary (2003). Imagery of voluntary move- (2000). Partially overlapping neural Gustin, S. M., Wrigley, P. J., Hender-
motor cortex involved in motor ment of fingers, toes, and tongue networks for real and imagined son, L. A., and Siddall, P. J. (2010).
imagery? Brain Res. Cogn. Brain Res. activates corresponding body-part- hand movements. Cereb. Cortex 10, Brain circuitry underlying pain in
19, 138–144. specific motor representations. J. 1093–1104. response to imagined movement in
Deiber, M. P., Ibanez, V., Honda, M., Neurophysiol. 90, 3304–3316. Gonzalez, B., Rodriguez, M., Ramirez, people with spinal cord injury. Pain
Sadato, N., Raman, R., and Hall- Ersland, L., Rosen, G., Lundervold, A., C., and Sabate, M. (2005). Distur- 148, 438–445.
ett, M. (1998). Cerebral processes Smievoll, A. I., Tillung, T., Sundberg, bance of motor imagery after cere- Hale, B. D. (1982). The effects of inter-
related to visuomotor imagery and H., and Hugdahl, K. (1996). Phan- bellar stroke. Behav. Neurosci. 119, nal and external imagery on muscu-
generation of simple finger move- tom limb imaginary fingertapping 622–626. lar and ocular concomitants. J. Sport
ments studied with positron emis- causes primary motor cortex activa- Gray, J. A. (1990). Brain systems that Psychol. 4, 379–387.
sion tomography. Neuroimage 7, tion: an fMRI study. Neuroreport 8, mediate both emotion and cogni- Hale, B. S., Raglin, J. S., and Koceja, D.
73–85. 207–210. tion. Cogn. Emot. 4, 269–288. M. (2003). Effect of mental imagery
Demougeot, L., and Papaxanthis, C. Facchini, S., Muellbacher, W., Battaglia, Grezes, J., and Decety, J. (2001). Func- of a motor task on the Hoffmann
(2011). Muscle fatigue affects men- F., Boroojerdi, B., and Hallett, tional anatomy of execution, men- reflex. Behav. Brain Res. 142, 81–87.
tal simulation of action. J. Neurosci. M. (2002). Focal enhancement of tal simulation, observation, and Hanakawa, T., Dimyan, M. A., and Hal-
31, 10712–10720. motor cortex excitability during verb generation of actions: a meta- lett, M. (2008). Motor planning,
Dettmers, C., Benz, M., Liepert, J., and motor imagery: a transcranial mag- analysis. Hum. Brain Mapp. 12, 1–19. imagery, and execution in the dis-
Rockstroh, B. (2012). Motor imagery netic stimulation study. Acta Neurol. Gueugneau, N., Crognier, L., and Papax- tributed motor network: a time-
in stroke patients, or plegic patients Scand. 105, 146–151. anthis, C. (2008). The influence of course study with functional MRI.
with spinal cord or peripheral dis- Fadiga, L., Buccino, G., Craighero, L., eye movements on the temporal Cereb. Cortex 18, 2775–2788.
eases. Acta Neurol. Scand. (in press). Fogassi, L., Gallese, V., and Pavesi, features of executed and imagined Hanakawa, T., Immisch, I., Toma, K.,
Dickstein, R., and Deutsch, J. E. (2007). G. (1999). Corticospinal excitabil- arm movements. Brain Res. 1187, Dimyan, M. A., Van Gelderen, P.,
Motor imagery in physical therapist ity is specifically modulated by 95–102. and Hallett, M. (2003). Functional
practice. Phys. Ther. 87, 942–953. motor imagery: a magnetic stim- Guillot, A., and Collet, C. (2008). Con- properties of brain areas associated
Dickstein, R., Gazit-Grunwald, M., Plax, ulation study. Neuropsychologia 37, struction of the motor imagery inte- with motor execution and imagery.
M., Dunsky, A., and Marcovitz, E. 147–158. grative model in sport: a review and J. Neurophysiol. 89, 989–1002.
Harris, A. J. (1999). Cortical origin Jacobson, E. (1932). Electrophysiology after amputation. Brain 119(Pt 4), Li, S., Kamper, D. G., Stevens, J. A.,
of pathological pain. Lancet 354, of mental activities. Am. J. Psychol. 1213–1219. and Rymer, W. Z. (2004a). The effect
1464–1466. 44, 677–694. Kokotilo, K. J., Eng, J. J., and Curt, A. of motor imagery on spinal seg-
Harris, D. V., and Robinson, W. J. Jeannerod, M. (1994). The represent- (2009). Reorganization and preser- mental excitability. J. Neurosci. 24,
(1986). The effects of skill level on ing brain: neural correlates of motor vation of motor control of the 9674–9680.
EMG activity during internal and intention and imagery. Behav. Brain brain in spinal cord injury: a sys- Li, S., Latash, M. L., and Zatsiorsky,V. M.
external imagery. J. Sport Psychol. 8, Sci. 17, 187–202. tematic review. J. Neurotrauma 26, (2004b). Effects of motor imagery
105–111. Jeannerod, M. (2001). Neural simula- 2113–2126. on finger force responses to transcra-
Hashimoto, R., and Rothwell, J. C. tion of action: a unifying mechanism Kosslyn, S. M. (2010). “Multimotal nial magnetic stimulation. Brain Res.
(1999). Dynamic changes in corti- for motor cognition. Neuroimage 14, images in the brain,” in The Neuro- Cogn. Brain Res. 20, 273–280.
cospinal excitability during motor S103–S109. physiological Foundations of Mental Liang, N., Ni, Z., Takahashi, M.,
imagery. Exp. Brain Res. 125, 75–81. Jeannerod, M. (2006). Motor Cognition. and Motor Imagery, eds A. Guillot Murakami, T., Yahagi, S., Funase, K.,
Helmich, R. C., Bloem, B. R., and Toni, New York: Oxford University Press. and C. Collet (New York, NY: Oxford Kato, T., and Kasai, T. (2007). Effects
I. (2012). Motor imagery evokes Jeannerod, M., and Frak, V. (1999). University Press), 3–16. of motor imagery are dependent
increased somatosensory activity in Mental imaging of motor activity in Kosslyn, S. M., Ganis, G., and Thomp- on motor strategies. Neuroreport 18,
parkinson’s disease patients with humans. Curr. Opin. Neurobiol. 9, son, W. L. (2001). Neural founda- 1241–1245.
tremor. Hum. Brain Mapp. 33, 735–739. tions of imagery. Nat. Rev. Neurosci. Liepert, J., Greiner, J., Nedelko, V.,
1763–1779. Johnson, S. H., Sprehn, G., and Saykin, 2, 635–642. and Dettmers, C. (2012). Reduced
Helmich, R. C., De Lange, F. P., Bloem, A. J. (2002). Intact motor imagery Kuhn, A. A., Doyle, L., Pogosyan, A., upper limb sensation impairs men-
B. R., and Toni, I. (2007). Cere- in chronic upper limb hemiplegics: Yarrow, K., Kupsch, A., Schneider, tal chronometry for motor imagery
bral compensation during motor evidence for activity-independent G. H., Hariz, M. I., Trottenberg, T., after stroke: clinical and electro-
imagery in Parkinson’s disease. Neu- action representations. J. Cogn. Neu- and Brown, P. (2006). Modulation physiological findings. Neurorehabil.
ropsychologia 45, 2201–2215. rosci. 14, 841–852. of beta oscillations in the subthal- Neural Repair 26, 470–478.
Heremans, E., Feys, P., Nieuwboer, A., Jowdy, D. P., and Harris, D. V. (1990). amic area during motor imagery Lim, V. K., Polych, M. A., Hollander,
Vercruysse, S., Vandenberghe, W., Muscular responses during mental in Parkinson’s disease. Brain 129, A., Byblow, W. D., Kirk, I. J., and
Sharma, N., and Helsen, W. (2011). imagery as a function of motor 695–706. Hamm, J. P. (2006). Kinesthetic but
Motor imagery ability in patients skill level. J. Sport Exerc. Psychol. 12, Kumru, H., Soto, O., Casanova, J., and not visual imagery assists in normal-
with early- and mid-stage Parkinson 191–201. Valls-Sole, J. (2008). Motor cortex izing the CNV in Parkinson’s disease.
disease. Neurorehabil. Neural Repair Kagerer, F. A., Bracha,V., Wunderlich, D. excitability changes during imagery Clin. Neurophysiol. 117, 2308–2314.
25, 168–177. A., Stelmach, G. E., and Bloedel, J. R. of simple reaction time. Exp. Brain Livesay, J. R., and Samaras, M. R. (1998).
Heremans, E., Helsen, W. F., and Feys, (1998). Ataxia reflected in the sim- Res. 189, 373–378. Covert neuromuscular activity of
P. (2008). The eyes as a mir- ulated movements of patients with Lacourse, M. G., Cohen, M. J., Lawrence, the dominant forearm during visual-
ror of our thoughts: quantification cerebellar lesions. Exp. Brain Res. K. E., and Romero, D. H. (1999). ization of a motor task. Percept. Mot.
of motor imagery of goal-directed 121, 125–134. Cortical potentials during imag- Skills 86, 371–374.
movements through eye movement Karl, A., Birbaumer, N., Lutzenberger, ined movements in individuals with Logan, G. D. (1983). On the ability to
registration. Behav. Brain Res. 187, W., Cohen, L. G., and Flor, H. chronic spinal cord injuries. Behav. inhibit simple thoughts and actions:
351–360. (2001). Reorganization of motor Brain Res. 104, 73–88. I. Stop signal studies of decision
Heremans, E., Nieuwboer, A., Feys, P., and somatosensory cortex in upper Lafleur, M. F., Jackson, P. L., Malouin, and memory. J. Exp. Psychol. Learn.
Vercruysse, S., Vandenberghe, W., extremity amputees with phan- F., Richards, C. L., Evans, A. C., Mem. Cogn. 9, 585–606.
Sharma, N., and Helsen, W. F. tom limb pain. J. Neurosci. 21, and Doyon, J. (2002). Motor learn- Logan, G. D. (1985). On the ability to
(2012). External cueing improves 3609–3618. ing produces parallel dynamic func- inhibit simple thoughts and actions:
motor imagery quality in patients Kasai, T., Kawai, S., Kawanishi, M., and tional changes during the execu- II. Stop-signal studies of repetition
with Parkinson disease. Neuroreha- Yahagi, S. (1997). Evidence for facil- tion and imagination of sequen- priming. J. Exp. Psychol. Learn. Mem.
bil. Neural Repair 26, 27–35. itation of motor evoked potentials tial foot movements. Neuroimage 16, Cogn. 11, 675–691.
Holmes, P. S., and Collins, D. J. (2001). (MEPs) induced by motor imagery. 142–157. Logan, G. D., and Cowan, W. B. (1984).
The PETTLEP approach to motor Brain Res. 744, 147–150. Lebon, F., Byblow, W. D., Collet, C., On the ability to inhibit thought and
imagery: a functional equivalence Kasess, C. H., Windischberger, C., Cun- Guillot,A., and Stinear, C. M. (2012). action: a theory of an act of control.
model for sport psychologists. J. nington, R., Lanzenberger, R., Peza- The modulation of motor cortex Psychol. Rev. 91, 295–327.
Appl. Sport Psychol. 13, 60–83. was, L., and Moser, E. (2008). The excitability during motor imagery Lorey, B., Bischoff, M., Pilgramm, S.,
Hotz-Boendermaker, S., Funk, M., suppressive influence of SMA on depends on imagery quality. Eur. J. Stark, R., Munzert, J., and Zentgraf,
Summers, P., Brugger, P., Hepp- M1 in motor imagery revealed by Neurosci. 35, 323–331. K. (2009). The embodied nature of
Reymond, M. C., Curt, A., and fMRI and dynamic causal modeling. Lebon, F., Rouffet, D., Collet, C., and motor imagery: the influence of pos-
Kollias, S. S. (2008). Preservation Neuroimage 40, 828–837. Guillot, A. (2008). Modulation of ture and perspective. Exp. Brain Res.
of motor programs in paraplegics Kimberley, T. J., Khandekar, G., Skraba, EMG power spectrum frequency 194, 233–243.
as demonstrated by attempted and L. L., Spencer, J. A., Van Gorp, during motor imagery. Neurosci. Lotze, M., Flor, H., Grodd, W., Larbig,
imagined foot movements. Neu- E. A., and Walker, S. R. (2006). Lett. 435, 181–185. W., and Birbaumer, N. (2001). Phan-
roimage 39, 383–394. Neural substrates for motor imagery Lenartowicz, A., Verbruggen, F., Logan, tom movements and pain. An fMRI
Jackson, P. L., Lafleur, M. F., Malouin, in severe hemiparesis. Neurorehabil. G. D., and Poldrack, R. A. (2011). study in upper limb amputees. Brain
F., Richards, C. L., and Doyon, J. Neural Repair 20, 268–277. Inhibition-related activation in the 124, 2268–2277.
(2003). Functional cerebral reorga- Kleber, B., Birbaumer, N., Veit, R., right inferior frontal gyrus in the Lotze, M., Grodd, W., Birbaumer, N.,
nization following motor sequence Trevorrow, T., and Lotze, M. (2007). absence of inhibitory cues. J. Cogn. Erb, M., Huse, E., and Flor, H.
learning through mental practice Overt and imagiend singing of Neurosci. 23, 3388–3399. (1999a). Does use of a myoelectric
with motor imagery. Neuroimage 20, an Italian aria. Neuroimage 36, Leonard, G., and Tremblay, F. (2007). prosthesis prevent cortical reorgani-
1171–1180. 1238–1246. Corticomotor facilitation associated zation and phantom limb pain? Nat.
Jacobson, E. (1930). Electrical measure- Knecht, S., Henningsen, H., Elbert, with observation, imagery and imi- Neurosci. 2, 501–502.
ments of neuromuscular states dur- T., Flor, H., Hohling, C., Pantev, tation of hand actions: a compara- Lotze, M., Montoya, P., Erb, M., Huls-
ing mental activities. Am. J. Physiol. C., and Taub, E. (1996). Reorgani- tive study in young and old adults. mann, E., Flor, H., Klose, U., Bir-
91, 567–608. zational and perceptional changes Exp. Brain Res. 177, 167–175. baumer, N., and Grodd, W. (1999b).
Activation of cortical and cerebel- overlap between hand and forearm Nico, D., Daprati, E., Rigal, F., Parsons, Ranganathan, V. K., Siemionow, V.,
lar motor areas during executed and motor cortical representations dur- L., and Sirigu, A. (2004). Left and Liu, J. Z., Sahgal, V., and Yue, G.
imagined hand movements: an fMRI ing motor cognitive tasks. Clin. Neu- right hand recognition in upper limb H. (2004). From mental power to
study. J. Cogn. Neurosci. 11, 491–501. rophysiol. 118, 1767–1775. amputees. Brain 127, 120–132. muscle power-gaining strength by
Lotze, M., and Halsband, U. (2006). Mayer, A., Kudar, K., Bretz, K., and Olsson, C. J. (2012). Complex motor using the mind. Neuropsychologia
Motor imagery. J. Physiol. Paris 99, Tihanyi, J. (2008). Body schema and representations may not be 42, 944–956.
386–395. body awareness of amputees. Pros- preserved after complete spinal Reilly, K. T., Mercier, C., Schieber,
Lotze, M., Scheler, G., Tan, H. R., Braun, thet. Orthot. Int. 32, 363–382. cord injury. Exp. Neurol. 236, 46–49. M. H., and Sirigu, A. (2006). Per-
C., and Birbaumer, N. (2003). The Mercier, C., and Sirigu, A. (2009). Pascual-Leone, A., Peris, M., Tormos, J. sistent hand motor commands in
musician’s brain: functional imag- Training with virtual visual feed- M., Pascual, A. P., and Catala, M. the amputees’ brain. Brain 129,
ing of amateurs and professionals back to alleviate phantom limb D. (1996). Reorganization of human 2211–2223.
during performance and imagery. pain. Neurorehabil. Neural Repair cortical motor output maps follow- Reis, J., Swayne, O. B., Vandermeeren, Y.,
Neuroimage 20, 1817–1829. 23, 587–594. ing traumatic forearm amputation. Camus, M., Dimyan, M. A., Harris-
Lotze, M., and Zentgraf, K. (2010). Michelon, P., Vettel, J. M., and Zacks, Neuroreport 7, 2068–2070. Love, M., Perez, M. A., Ragert, P.,
“Contribution of the primary motor J. M. (2006). Lateral somatotopic Patuzzo, S., Fiaschi, A., and Mangan- Rothwell, J. C., and Cohen, L. G.
cortex to motor imagery,” in The organization during imagined and otti, P. (2003). Modulation of (2008). Contribution of transcra-
Neurophysiological Foundations of prepared movements. J. Neurophys- motor cortex excitability in the left nial magnetic stimulation to the
Mental and Motor Imagery, eds A. iol. 95, 811–822. hemisphere during action obser- understanding of cortical mecha-
Guillot and C. Collet (New York: Milton, J., Solodkin, A., Hlustik, P., vation: a single- and paired-pulse nisms involved in motor control. J.
Oxford University Press), 31–45. and Small, S. L. (2007). The mind transcranial magnetic stimulation Physiol. (Lond.) 586, 325–351.
Louis, M., Collet, C., Champely, S., and of expert motor performance is study of self- and non-self-action Roosink, M., and Zijdewind, I. (2010).
Guillot, A. (2012). Differences in cool and focused. Neuroimage 35, observation. Neuropsychologia 41, Corticospinal excitability during
motor imagery time when predict- 804–813. 1272–1278. observation and imagery of sim-
ing task duration in skiers and eques- Moran, A., Guillot, A., MacIntyre, T., Personnier, P., Paizis, C., Ballay, Y., ple and complex hand tasks: impli-
trian riders. Res. Q. Exerc. Sport 83, and Collet, C. (2012). Re-imagining and Papaxanthis, C. (2008). Men- cations for motor rehabilitation.
86–93. motor imagery: building bridges tally represented motor actions in Behav. Brain Res. 213, 35–41.
Luppino, G., Matelli, M., Camarda, between cognitive neuroscience and normal aging II. The influence of Ross, J. S., Tkach, J., Ruggieri, P. M.,
R., and Rizzolatti, G. (1994). Cor- sport psychology. Br. J. Psychol. 103, the gravito-inertial context on the Lieber, M., and Lapresto, E. (2003).
ticospinal projections from mesial 224–247. duration of overt and covert arm The mind’s eye: functional MR
frontal and cingulate areas in the Morecraft, R. J., Herrick, J. L., Stilwell- movements. Behav. Brain Res. 186, imaging evaluation of golf motor
monkey. Neuroreport 5, 2545–2548. Morecraft, K. S., Louie, J. L., 273–283. imagery. AJNR Am. J. Neuroradiol.
Lutz, R., and Linder, D. E. (2001). Schroeder, C. M., Ottenbacher, J. Porro, C. A., Cettolo, V., Francescato, 24, 1036–1044.
Does electromyographic activity G., and Schoolfield, M. W. (2002). M. P., and Baraldi, P. (2000). Ipsilat- Rossini, P. M., Rossi, S., Pasqualetti,
during motor imagery predict per- Localization of arm representation eral involvement of primary motor P., and Tecchio, F. (1999). Corti-
formance? A test of bioinformati- in the corona radiata and internal cortex during motor imagery. Eur. J. cospinal excitability modulation to
onal theory and functional equiv- capsule in the non-human primate. Neurosci. 12, 3059–3063. hand muscles during movement
alences. J. Sport Exerc. Psychol. 23, Brain 125, 176–198. Porro, C. A., Francescato, M. P., Cet- imagery. Cereb. Cortex 9, 161–167.
s63. Mulder, T., De Vries, S., and Zijlstra, tolo, V., Diamond, M. E., Baraldi, Rothwell, J. C. (1991). Physiological
MacIver, K., Lloyd, D. M., Kelly, S., S. (2005). Observation, imagina- P., Zuiani, C., Bazzocchi, M., and studies of electric and magnetic
Roberts, N., and Nurmikko, T. tion and execution of an effort- Di Prampero, P. E. (1996). Primary stimulation of the human brain.
(2008). Phantom limb pain, cortical ful movement: more evidence for motor and sensory cortex activa- Electroencephalogr. Clin. Neurophys-
reorganization and the therapeutic a central explanation of motor tion during motor performance and iol. Suppl. 43, 29–35.
effect of mental imagery. Brain 131, imagery. Exp. Brain Res. 163, motor imagery: a functional mag- Roux, F. E., Ibarrola, D., Lazorthes, Y.,
2181–2191. 344–351. netic resonance imaging study. J. and Berry, I. (2001). Virtual move-
Macuga, K. L., and Frey, S. H. (2012). Mulder, T., Zijlstra, S., Zijlstra, W., and Neurosci. 16, 7688–7698. ments activate primary sensorimo-
Neural representations involved in Hochstenbach, J. (2004). The role of Quartarone, A., Bagnato, S., Rizzo, tor areas in amputees: report of three
observed, imagined, and imitated motor imagery in learning a totally V., Morgante, F., Sant’Angelo, A., cases. Neurosurgery 49, 736–741; dis-
actions are dissociable and hierar- novel movement. Exp. Brain Res. Crupi, D., Romano, M., Messina, cussion 741–732.
chically organized. Neuroimage 59, 154, 211–217. C., Berardelli, A., and Girlanda, Roux, F. E., Lotterie, J. A., Cassol, E.,
2798–2807. Munzert, J., Lorey, B., and Zentgraf, K. P. (2005). Corticospinal excitability Lazorthes, Y., Sol, J. C., and Berry,
Malouin, F., Richards, C. L., Desrosiers, (2009). Cognitive motor processes: during motor imagery of a simple I. (2003). Cortical areas involved
J., and Doyon, J. (2004). Bilat- the role of motor imagery in the tonic finger movement in patients in virtual movement of phantom
eral slowing of mentally simulated study of motor representations. with writer’s cramp. Mov. Disord. 20, limbs: comparison with normal sub-
actions after stroke. Neuroreport 15, Brain Res. Rev. 60, 306–326. 1488–1495. jects. Neurosurgery 53, 1342–1352;
1349–1353. Murphy, S., Nordin, S. M., and Cum- Raffin, E., Giraux, P., and Reilly, K. discussion 1352–1343.
Malouin, F., Richards, C. L., Durand, ming, J. (2008). “Imagery in sport, T. (2012a). The moving phantom: Roy, F. D., Zewdie, E. T., and Gorassini,
A., and Doyon, J. (2008). Clinical exercise and dance,” in Advances motor execution or motor imagery? M. A. (2011). Short-interval intra-
assessment of motor imagery after in Sport Psychology, ed. T. Horn Cortex 48, 746–757. cortical inhibition with incomplete
stroke. Neurorehabil. Neural Repair (Champaign, IL: Human Kinetics), Raffin, E., Mattout, J., Reilly, K. T., spinal cord injury. Clin. Neurophys-
22, 330–340. 306–315. and Giraux, P. (2012b). Disentan- iol. 122, 1387–1395.
Mannix, L. K., Chandurkar, R. S., Naito, E., Kochiyama, T., Kidata, R., gling motor execution from motor Sabate, M., Gonzalez, B., and Rodriguez,
Rybicki, L. A., Tusek, D. L., and Nakamura, S., Matsumura, M., Yon- imagery with the phantom limb. M. (2004). Brain lateralization of
Solomon, G. D. (1999). Effect of ekura, Y., and Sadato, N. (2002). Brain 135, 582–595. motor imagery: motor planning
guided imagery on quality of life for Internally simulated movement sen- Ramachandran, V. S., Brang, D., and asymmetry as a cause of movement
patients with chronic tension-type sations during motor imagery acti- Mcgeoch, P. D. (2010). Dynamic lateralization. Neuropsychologia 42,
headache. Headache 39, 326–334. vate cortical motor areas and reorganization of referred sensations 1041–1049.
Marconi, B., Pecchioli, C., Koch, G., and the cerebellum. J. Neurosci. 22, by movements of phantom limbs. Sabate, M., Gonzalez, B., and Rodriguez,
Caltagirone, C. (2007). Functional 3683–3691. Neuroreport 21, 727–730. M. (2007). Adapting movement
planning to motor impairments: the parietal cortex damage. Science 273, and stroke patients. Neuroimage 62, movement therapy in stroke. Neu-
motor-scanning system. Neuropsy- 1564–1568. 266–280. rologist 8, 325–338.
chologia 45, 378–386. Slade, J. M., Landers, D. M., and Martin, Tamir, R., Dickstein, R., and Huber- Wu, A. J., Hermann, V., Ying, J., and
Sabbah, P., De, S. S., Leveque, C., Gay, P. E. (2002). Muscular activity dur- man, M. (2007). Integration of Page, S. J. (2010). Chronometry of
S., Pfefer, F., Nioche, C., Sarrazin, ing real and imagined movements: a motor imagery and physical prac- mentally versus physically practiced
J. L., Barouti, H., Tadie, M., and test of inflow explanations. J. Sport tice in group treatment applied tasks in people with stroke. Am. J.
Cordoliani, Y. S. (2002). Sensorimo- Exerc. Psychol. 24, 151–167. to subjects with Parkinson’s dis- Occup. Ther. 64, 929–934.
tor cortical activity in patients with Solodkin, A., Hlustik, P., Chen, E. E., and ease. Neurorehabil. Neural Repair 21, Yahagi, S., and Kasai, T. (1998). Facil-
complete spinal cord injury: a func- Small, S. L. (2004). Fine modulation 68–75. itation of motor evoked potentials
tional magnetic resonance imaging in network activation during motor Taniguchi, S., Kimura, J., Yamada, T., (MEPs) in first dorsal interosseous
study. J. Neurotrauma 19, 53–60. execution and motor imagery. Cereb. Ichikawa, H., Hara, M., Fujisawa, R., (FDI) muscle is dependent on
Samuel, M., Ceballos-Baumann, A. O., Cortex 14, 1246–1255. Shimizu, H., and Tani, T. (2008). different motor images. Electroen-
Boecker, H., and Brooks, D. J. (2001). Stinear, C. M. (2010). “Corticospinal Effect of motion imagery to counter cephalogr. Clin. Neurophysiol. 109,
Motor imagery in normal subjects facilitation during motor imagery,” rest-induced suppression of F-wave 409–417.
and Parkinson’s disease patients: an in The Neurophysiological Founda- as a measure of anterior horn cell Yahagi, S., Shimura, K., and Kasai,
H215O PET study. Neuroreport 12, tions of Mental and Motor Imagery, excitability. Clin. Neurophysiol. 119, T. (1996). An increase in cortical
821–828. eds A. Guillot and C. Collet (New 1346–1352. excitability with no change in spinal
Saturno, E., Bonato, C., Miniussi, C., York, NY: Oxford University Press), Thobois, S., Dominey, P., Fraix, V., excitability during motor imagery.
Lazzaro, V., and Callea, L. (2008). 47–61. Mertens, P., Guenot, M., Zimmer, L., Percept. Mot. Skills 83, 288–290.
Motor cortex changes in spinal cord Stinear, C. M., and Byblow, W. D. Pollak, P., Benabid, A. L., and Brous- Yue, G., and Cole, K. J. (1992). Strength
injury: a TMS study. Neurol. Res. 30, (2003a). Motor imagery of phasic solle, E. (2002). Effects of subthal- increases from the motor program:
1084–1085. thumb abduction temporally and amic nucleus stimulation on actual comparison of training with max-
Schwoebel, J., Boronat, C. B., and spatially modulates corticospinal and imagined movement in Parkin- imal voluntary and imagined mus-
Branch Coslett, H. (2002). The man excitability. Clin. Neurophysiol. 114, son’s disease: a PET study. J. Neurol. cle contractions. J. Neurophysiol. 67,
who executed “imagined” move- 909–914. 249, 1689–1698. 1114–1123.
ments: evidence for dissociable com- Stinear, C. M., and Byblow, W. D. Thobois, S., Dominey, P. F., Decety, J., Zatsiorsky, V. M., Li, Z. M., and Latash,
ponents of the body schema. Brain (2003b). Role of intracortical inhi- Pollak, P. P., Gregoire, M. C., Le M. L. (2000). Enslaving effects in
Cogn. 50, 1–16. bition in selective hand mus- Bars, P. D., and Broussolle, E. (2000). multi-finger force production. Exp.
Sharma, N., Baron, J. C., and Rowe, cle activation. J. Neurophysiol. 89, Motor imagery in normal subjects Brain Res. 131, 187–195.
J. B. (2009a). Motor imagery after 2014–2020. and in asymmetrical Parkinson’s dis- Zijdewind, J., Toering, S. T., Bessen, B.,
stroke: relating outcome to motor Stinear, C. M., and Byblow, W. D. ease: a PET study. Neurology 55, Van-Der-Laan, O., and Diercks, R.
network connectivity. Ann. Neurol. (2004). Modulation of corticospinal 996–1002. L. (2003). Effects of imagery motor
66, 604–616. excitability and intracortical inhibi- Topka, H., Cohen, L. G., Cole, R. A., training on torque production of
Sharma, N., Simmons, L. H., Jones, tion during motor imagery is task- and Hallett, M. (1991). Reorganiza- ankle plantar flexor muscles. Muscle
P. S., Day, D. J., Carpenter, T. A., dependent. Exp. Brain Res. 157, tion of corticospinal pathways fol- Nerve 28, 168–173.
Pomeroy, V. M., Warburton, E. A., 351–358. lowing spinal cord injury. Neurology Zimmermann-Schlatter, A., Schuster,
and Baron, J. C. (2009b). Motor Stinear, C. M., Byblow, W. D., Steyvers, 41, 1276–1283. C., Puhan, M. A., Siekierka, E., and
imagery after subcortical stroke: M., Levin, O., and Swinnen, S. P. Tremblay, F., Leonard, G., and Tremblay, Steurer, J. (2008). Efficacy of motor
a functional magnetic resonance (2006). Kinesthetic, but not visual, L. (2008). Corticomotor facilitation imagery in post-stroke rehabilita-
imaging study. Stroke 40, 1315–1324. motor imagery modulates cortico- associated with observation and tion: a systematic review. J. Neuro-
Sharma, N., Jones, P. S., Carpenter, T. A., motor excitability. Exp. Brain Res. imagery of hand actions is impaired eng. Rehabil. 5, 8.
and Baron, J. C. (2008). Mapping the 168, 157–164. in Parkinson’s disease. Exp. Brain
involvement of BA 4a and 4p dur- Stinear, C. M., Coxon, J. P., and Byblow, Res. 185, 249–257. Conflict of Interest Statement: The
ing motor imagery. Neuroimage 41, W. D. (2009). Primary motor cortex Verbruggen, F., and Logan, G. D. (2009). authors declare that the research was
92–99. and movement prevention: where Models of response inhibition in the conducted in the absence of any
Sharma, N., Pomeroy, V. M., and Baron, Stop meets Go. Neurosci. Biobehav. stop-signal and stop-change para- commercial or financial relationships
J. C. (2006). Motor imagery: a back- Rev. 33, 662–673. digms. Neurosci. Biobehav. Rev. 33, that could be construed as a potential
door to the motor system after Stinear, C. M., Fleming, M. K., Barber, P. 647–661. conflict of interest.
stroke? Stroke 37, 1941–1952. A., and Byblow, W. D. (2007). Later- Vromen, A., Verbunt, J. A., Rasquin,
Shaw, W. A. (1938). The distribution alization of motor imagery follow- S., and Wade, D. T. (2011). Motor Received: 29 March 2012; accepted: 10
of muscular action potentials during ing stroke. Clin. Neurophysiol. 118, imagery in patients with a right August 2012; published online: 05 Sep-
imaging. Psychol. Rec. 2, 195–216. 1794–1801. hemisphere stroke and unilateral tember 2012.
Shick, J. (1970). Effects of mental prac- Subramanian, L., Hindle, J. V., John- neglect. Brain Inj. 25, 387–393. Citation: Guillot A, Di Rienzo F, Mac-
tice on selected volleyball skills for ston, S., Roberts, M. V., Husain, M., Watanabe, J., Sugiura, M., Sato, K., Sato, Intyre T, Moran A and Collet C (2012)
college women. Res. Q. 41, 88–94. Goebel, R., and Linden, D. (2011). Y., Maeda, Y., Matsue, Y., Fukuda, Imagining is not doing but involves spe-
Shukla, G. D., Sahu, S. C., Tripathi, R. P., Real-time functional magnetic res- H., and Kawashima, R. (2002). cific motor commands: a review of exper-
and Gupta, D. K. (1982). Phantom onance imaging neurofeedback for The human prefrontal and pari- imental data related to motor inhibi-
limb: a phenomenological study. Br. treatment of Parkinson’s disease. J. etal association cortices are involved tion. Front. Hum. Neurosci. 6:247. doi:
J. Psychiatry 141, 54–58. Neurosci. 31, 16309–16317. in NO-GO performances: an event- 10.3389/fnhum.2012.00247
Sirigu, A., Cohen, L., Duhamel, J. R., Suinn, R. M. (1980). “Body thinking: related fMRI study. Neuroimage 17, Copyright © 2012 Guillot , Di Rienzo,
Pillon, B., Dubois, B., Agid, Y., and psychology for Olympic champi- 1207–1216. MacIntyre, Moran and Collet . This is an
Pierrot-Deseilligny, C. (1995). Con- ons,”in Psychology in Sports: Methods Wehner, T., Vogt, S., and Stadler, open-access article distributed under the
gruent unilateral impairments for and Applications, ed. R. M. Suinn M. (1984). Task-specific EMG- terms of the Creative Commons Attribu-
real and imagined hand movements. (Mineapolis: Burgess), 306–315. characteristics during mental train- tion License, which permits use, distrib-
Neuroreport 6, 997–1001. Szameitat, A. J., Shen, S., Conforto, ing. Psychol. Res. 46, 389–401. ution and reproduction in other forums,
Sirigu, A., Duhamel, J. R., Cohen, L., A., and Sterr, A. (2012). Cortical Wolf, S. L., Blanton, S., Baer, H., Bres- provided the original authors and source
Pillon, B., Dubois, B., and Agid, activation during executed, imag- hears, J., and Butler, A. J. (2002). are credited and subject to any copy-
Y. (1996). The mental represen- ined, observed, and passive wrist Repetitive task practice: a criti- right notices concerning any third-party
tation of hand movements after movements in healthy volunteers cal review of constraint-induced graphics etc.
REVIEW ARTICLE
Mental imagery of speech: linking motor and perceptual

systems through internal simulation and estimation
Xing Tian* and David Poeppel
Poeppel Lab, Department of Psychology, New York University, New York, NY, USA
Edited by: The neural basis of mental imagery has been investigated by localizing the underlying
Joel Pearson, University of New neural networks, mostly in motor and perceptual systems, separately. However,
how modality-specific representations are top-down induced and how the action and
Reviewed by:
perception systems interact in the context of mental imagery is not well understood.
Arthur M. Jacobs, Freie Universität
Berlin, Berlin Imagined speech production (“articulation imagery”), which induces the kinesthetic
Joel Pearson, University of New feeling of articulator movement and its auditory consequences, provides a new angle
South Wales, Australia because of the concurrent involvement of motor and perceptual systems. On the basis
*Correspondence: of previous findings in mental imagery of speech, we argue for the following regarding
Xing Tian, Poeppel Lab, Department
the induction mechanisms of mental imagery and the interaction between motor and
of Psychology, New York University,
6 Washington Pl, Suite 275, perceptual systems: (1) Two distinct top-down mechanisms, memory retrieval and motor
New York, NY 10003, USA. simulation, exist to induce estimation in perceptual systems. (2) Motor simulation is
e-mail: xing.tian@nyu.edu sufficient to internally induce the representation of perceptual changes that would be
caused by actual movement (perceptual associations); however, this simulation process
only has modulatory effects on the perception of external stimuli, which critically depends
on context and task demands. Considering the proposed simulation-estimation processes
as common mechanisms for interaction between motor and perceptual systems, we
outline how mental imagery (of speech) relates to perception and production, and how
these hypothesized mechanisms might underpin certain neural disorders.
Keywords: internal forward model, efference copy, corollary discharge, sensory-motor integration, mirror neurons,
auditory hallucination, stuttering, phantom limb
INTRODUCTION contextual influence. Studies of mental imagery of speech are

Mental imagery can be characterized as a quasi-perceptual summarized to provide evidence for the proposed account—
experience, induced in the absence of external stimulation. and for the coexistence of both mechanisms. We discuss the
Neuroimaging studies have shown that common neural sub- motor-to-sensory integration process and propose some working
strates mediate mental imagery and the corresponding percep- hypotheses regarding certain neural and neuropsychiatric disor-
tual processes, such as in visual (e.g., Kosslyn et al., 1999; ders from the perspective of the proposed internal simulation and
O’Craven and Kanwisher, 2000), auditory (e.g., Zatorre et al., estimation mechanisms.
1996; Kraemer et al., 2005), somatosensory (e.g., Yoo et al., 2003;
Zhang et al., 2004), and olfactory domains (e.g., Bensafi et al., DIFFERENT ROUTES FOR INDUCING MENTAL IMAGES
2003; Djordjevic et al., 2005). The demonstration of activation MENTAL IMAGERY OF PERCEPTION AS MEMORY RETRIEVAL
in corresponding perceptual regions during mental imagery has (DIRECT SIMULATION)
provided strong evidence to support the claim that the percep- Mental imagery has been proposed to be essentially a mem-
tual experience during mental imagery is mediated by modality- ory retrieval process. That is, perceptual experience is simulated
specific neural representations (see the review by Kosslyn et al., by reconstructing stored perceptual information in modality-
2001). However, the top-down “induction mechanism” for the specific cortices (Kosslyn, 1994, 2005; Kosslyn et al., 1994). In
neural activity mediating mental imagery is not well understood. particular, the process, guided by attention, retrieves object and
We focus here on the role of the motor system in the construc- spatial properties stored in long-term memory to reactivate the
tion of perceptual experience in mental imagery. We propose a topographically organized sensory cortices that represent the
motor-based mechanism that is an alternative (additional) mech- object features. Through top-down (re)construction of the neural
anism to Kosslyn’s memory-attention-based account (Kosslyn, representation that is similar to the result of bottom-up percep-
1994, 2005; Kosslyn et al., 1994): planned action is simulated tual processes, the perceptual experience can be re-elicited with-
in motor systems to internally derive the representation of per- out the presence of any physical stimuli during mental imagery.
ceptual changes that would be caused by the actual action This attention-guided memory retrieval process has been demon-
(perceptual associations). We suggest that the deployment of strated, for example, in the visual imagery of faces (Ishai et al.,
these two distinct mechanisms depends on task demands and 2002).
Tian and Poeppel Internal simulation and estimation
Mental imagery is further hypothesized to be a predictive this hypothesis, we propose here that the kinesthetic feeling in
process (for future perceptual states), in which the dynamics motor imagery is the result of somatosensory estimation, derived
of perceptual experience can be retrieved/calculated and recon- from internal simulation that closely mimics the dynamics of a
structed internally (Moulton and Kosslyn, 2009). That is, given motor action. We refer to this account as motor simulation and
an initial point, the series of future perceptual states can be estimation.
internally simulated by following the regularity (temporal and The motor simulation and estimation account differs from the
causal constraints) stored in declarative memory (general knowl- direct simulation (memory retrieval) account in that it requires a
edge). The mapping between internal simulation and the per- transformation between motor and somatosensory systems. Our
ception of external stimulation is thought not to be necessarily question here, though, extends beyond this: can a motor simula-
isomorphic (Goldman, 1989), as only the essential intermedi- tion deliver perceptual consequences that extend to other sensory
ate states are required to have a one-to-one mapping (Fisher, domains (such as visual and auditory) as well? If so, internal sim-
2006). Because this proposed simulation process is executed ulation and estimation processes would serve as an additional
entirely within perceptual domains on the basis of memory path to induce modality-specific neural representations similar to
retrieval—without any representational transformation between the ones induced on the basis of memory retrieval. In the next
motor and perceptual systems—we refer to this account as direct section, we discuss this possibility in the framework of internal
simulation. forward models and propose a sequential simulation and estima-
tion account. We will use the interaction of motor, somatosensory,
MENTAL IMAGERY OF MOTOR ACTION AS ESTIMATION DERIVING and auditory systems in speech production as an example to illus-
FROM SIMULATION trate such internal cascaded processes, which can generalize to
Motor imagery is thought to be the process that internally sim- other sensory domains.
ulates planned actions, by activating similar neural substrates
that mediate motor intention and preparation (Jeannerod, 1995, MENTAL IMAGERY OF SPEECH AS SEQUENTIAL ESTIMATION
2001; Decety, 1996). Numerous studies have demonstrated both Perception and production systems are functionally connected:
frontal and parietal activity during motor imagery (Decety et al., perceptual systems analyze the sensory input generated by self
1994; Lotze et al., 1999; Gerardin et al., 2000; Ehrsson et al., actions; the motor system is also regulated by perceptual feed-
2003; Hanakawa et al., 2003; Dechent et al., 2004; Meister et al., back to perform updates on actions in the future. For exam-
2004; Nikulin et al., 2008). However, motor system activation ple, when people talk, they move their articulators, feel the
does not necessarily link to the kinesthetic feeling generated movement, and hear the self-produced speech that can be used
during motor imagery. The residual neural activity, resulting to detect and correct any pronunciation errors. The temporal
from the absence of external somatosensory feedback, is thought sequence of physical articulation, proprioception of the articu-
to mediate the kinesthetic experience during motor imagery lators, and auditory perception of one’s own vocalization makes
(Jeannerod, 1994, 1995). The implicit assumption of the “resid- it possible—on the basis of co-occurrence and associative learn-
ual activity account” is that the internal motor simulation during ing during development—to create internal connections among
imagery should be transformed into the same representational the neural processes that mediate motor action, somatosen-
format as the one resulting from somatosensory feedback. That sory feedback, and auditory perception. After establishing the
is, the somatosensory consequences of motor simulation should connections, motor commands can cycle internally through
be estimated. This is consistent with the view that parietal somatosensory regions and “reach” auditory regions. That is,
rather than frontal motor regions mediate motor awareness the estimation in the somatosensory system can serve as a link
(Desmurget and Sirigu, 2009). In support of the claim that pari- between motor and auditory systems. Theoretically, such a cas-
etal regions mediate somatosensory estimation, direct current caded estimation architecture has been hypothesized by Hesslow
stimulation over parietal cortex induces false belief of movement (2002). Anatomically and functionally, the connections between
(Desmurget et al., 2009); parietal lesions also impaired the tem- parietal regions and auditory temporal regions have also been
poral precision of performing motor imagery tasks (Sirigu et al., demonstrated (Schroeder et al., 2001; Foxe et al., 2002; Fu et al.,
1996). Cumulatively, the results suggest that motor simulation in 2003).
frontal cortex converges in parietal regions to form a kinesthetic On the basis of recent neurophysiological (MEG) studies, we
representation. proposed that a process of auditory inference after somatosen-
The internal transformation between motor simulation and sory estimation occurs during overt speech processing [Figure 1;
somatosensory estimation has been proposed in the context of adapted from Tian and Poeppel (2010)]. Specifically, the esti-
internal forward models in the motor control literature [see the mation of auditory consequences relies on the somatosensory
review by Wolpert and Ghahramani (2000)]. The core presup- estimation that derives from the simulation of planned action.
position is that the neural system can predict the perceptual That is, the internal auditory prediction is the result of a coor-
consequences by internal simulating a copy of a planned action dinate transformation from the somatosensory to the auditory
command (the efference copy). Mental imagery has been linked domain. This sequential estimation mechanism (motor plan →
to the concept of internal forward models by the argument that somatosensory estimation → auditory prediction/estimation)
the subjective feeling in mental imagery is the result of the can derive detailed auditory predictions that are then com-
internal estimation of the perceptual consequences following the pared with auditory feedback for self-monitoring and online
internal simulation of an action (Grush, 2004). Consistent with control.
Motor Sensory error

Inverse model correction _
planning/updating +
(desired motor state)
(motor system) Somatosensory
+ error correction _
Somatosensory
Perceptual
prediction
prediction
Motor Perceptual
efference copy First efference copy Second
forward model forward model
(parietal cortex) (modality specific)
Somatosensory
feedback
feedback
Sensory
Motor output Motor Sensory
action consequence
FIGURE 1 | Model of speech processing and its implication for mental somatosensory consequences. A copy of the somatosensory estimation is
imagery of speech. The internal simulation and estimation model further sent to modality-specific areas, and the associated perceptual
proposed as a second route to generate mental images. The motor consequences that would be produced by the overt action are estimated.
systems that mediate action preparation carry out the same functions in The quasi-perceptual experience during mental imagery (the feeling of
mental imagery of speech, but only perform motor simulation, in the movement of the articulators and the feeling of auditory perception in the
sense that the planned motor commands are truncated along the path to case of articulation imagery) is the result of residual activity from these
primary motor cortex and are not executed (the red cross over external internal estimation processes, because of the absence of cancellation from
outputs). A copy of such planned motor commands (motor efference copy) the external feedback (the red crosses over external somatosensory and
is processed internally and is used to estimate the associated perceptual feedback).
In the case of the mental imagery of speech, we propose condition 1. Following such a left parietal response pattern, a
that the quasi-perceptual experience of articulator movement second pattern was identified at a latency of 150–170 ms after
and the subsequent auditory percept are induced by the same the parietal response. This second pattern was very similar to the
sequential estimation mechanism. However, the “cancellation” response elicited by external auditory stimuli. Moreover, in a fur-
deriving from somatosensory and auditory feedback, which is ther experimental condition, hearing imagery, we also observed
generated by the overt outputs during production, is absent in the an auditory-like neural response pattern; however, its latency
imagery case (Figure 1). Therefore, similar to the case of motor was faster than the same auditory pattern observed in articula-
imagery (Jeannerod, 1994, 1995), the feeling of articulator move- tion imagery. The existence of these two spatially highly similar
ment is the result of residual somatosensory representation result- auditory-like neural representations, with different latencies for
ing from motor simulation; the subsequent auditory perceptual articulation versus hearing imagery tasks, suggests that the same
experience, we suggest, is the residual auditory representation (or strikingly similar) neural representations can be generated
from the second estimation stage. either by internal estimation or by memory retrieval, based on
On the basis of sequential estimation account, particular neu- contextual variation and task demands.
ral activity patterns for the two sequential estimates are predicted Note that the auditory estimation is presumably formed along
to occur in a temporal order. Specifically, an auditory pat- the canonical auditory hierarchy, but the induction process will
tern should follow a somatosensory one during mental imagery be in reversed order. That is to say, the abstract representation
of speech. Applying a novel multivariate technique (Tian and is (re-)constructed first in higher level associative areas and con-
Huber, 2008; Tian et al., 2011) to MEG data, we observed such veyed to a perceptual-sensory representation in lower areas. The
a temporal order for somatosensory and auditory estimations observation of neural activity in the posterior superior temporal
during articulation imagery (Tian and Poeppel, 2010), mani-
fested in the sequential activity patterns over modality-specific 1 Because previous findings suggest that the time courses for completing exe-
regions at different latencies (Figure 2). A left parietal response cution and imagery are comparable (Decety and Michel, 1989; Decety et al.,
pattern was observed during articulation imagery at the same 1989; Sirigu et al., 1995, 1996), the observed neural responses over parietal
latency as when motor responses occurred in the articulation regions presumably mediate somatosensory estimation.
Heursitc
&
Articulation ~170ms later Modulatory
Imagery
Perceptual Motor
~ Similar retrieval simulation
Latency Hearing
Hearing
Imagery
Sufficient
Perceptual Motor
&
representation representation
Isomorphic
~
Similar
Latency
Sensory Motor
Articulation
−100 −50 0 50 100 receptors effectors
Magnetic field (fT)
Causal
FIGURE 2 | Results from Tian and Poeppel (2010). The sequential &
estimation during articulation imagery revealed by MEG recordings. All Ecological
plots are MEG topographies (response patterns) when participants actually valid
speak (lower row), imagine hearing (middle row), and imagine speaking (top
row). The activity patterns in the first column are temporally aligned with
the onset of articulation movement. At a similar latency, bilateral frontal, FIGURE 3 | Sufficiency and necessity between motor simulation and
bilateral temporal, and left lateralized parietal activity patterns are observed perceptual estimation. The characteristics of the proposed motor
in articulation, hearing imagery, and articulation imagery conditions. In simulation and perceptual estimation processes, and the nature of motor
articulation imagery, about 150–170 ms later after the parietal activity, involvement during perceptual tasks. The internal motor simulation can take
bilateral temporal activity is also observed. All the bilateral temporal activity a similar path as motor preparation to derive a corresponding motor
patterns in hearing imagery and articulation imagery resemble the representation that in turn derives associated perceptual representations in
topography of the auditory response during actual hearing (highlighted in a a one-to-one fashion. Such one-to-one mapping is the same as the one in
blue box, response pattern when participants listen to the same auditory the external connections between the similar motor action and perceptual
stimuli as in other conditions). consequences. In the other direction, when the perceptual representation
is needed, different paths can be taken. It can rely on memory retrieval to
directly recreate the perceptual representation. It can also take another less
demanding path that relies on the motor simulation to derive the associated
perceptual representation.
sulcus (pSTS) during silent speaking (Price et al., 2011) could be
the result of an earlier reconstruction. Whereas the observations
of similarity between responses to mental imagery and to exter-
nal stimulation, such as in visual (e.g., Kosslyn et al., 1999) and perceptual estimation (Figure 3). The entire transformation
auditory (e.g., Figure 2, Tian and Poeppel, 2010) domains, are process is carried out in a continuous manner, beginning with
the results of process continuation to lower perceptual-sensory motor simulation, then somatosensory estimation, and ending
regions. How much further back the reconstruction process with modality-specific perceptual estimation. In the second part
might go seems to depend on the sensory modality and demands of this section, we argue that the implementation of motor simu-
of the imagery tasks (Kosslyn and Thompson, 2003; Kraemer lation depends on context and task demands and may only exert
et al., 2005; Zatorre and Halpern, 2005). modulatory effects on perception.
INTERNAL SIMULATION-ESTIMATION AND RELATION MOTOR-TO-SENSORY MAPPING: ISOMORPHISM VIA ESTABLISHED

TO SENSORY-MOTOR INTEGRATION CONNECTIONS
Mental imagery of speech exemplifies a top-down mechanism for The central idea underpinning motor simulation and subsequent
sensory-motor integration. The proposal here is motor simula- perceptual estimation is the conjectured one-to-one mapping or
tion and sequential estimation. In the first part of this section, isomorphism between mental and physical processes. This iso-
we describe the nature of this sequential transformation between morphism has been proposed for motor simulation (Jeannerod,
motor, somatosensory, and other perceptual systems. We pos- 1994) and visual mental rotation (Shepard and Cooper, 1986):
tulate that there is a one-to-one transformation between motor the intermediate stages of the internal process must have a one-
simulation and somatosensory estimation, as well as isomorphic to-one correspondence to intermediate stages of an actualized
mapping between somatosensory estimation and subsequent physical process. We extend this isomorphism to the associations
between the motor simulation and perceptual estimation: the perception (general suppression). However, the enhancement
one-to-one mapping between the trajectory of motor simulation caused by feedback perturbation suggests that the internal signal
and perceptual estimation is a close analog to the causal relation during articulation is not a generic gain control mechanism for
between motor outputs and perceptual changes. That is, not only all auditory stimuli, but rather provides a precise perceptual pre-
should the starting and ending points of an action simulation lead diction and only blocks the feedback that is identical to the pre-
to the initiation and results of perceptual estimation, but interme- diction. In other words, there is a one-to-one mapping between
diate points on this action simulation trajectory should result in a motor simulation and auditory estimation, and the precise audi-
sequence of perceptual estimates, even though no external signals tory consequence can be predicted based on particular motor
are physically presented. Notice that the analogy between inter- trajectory.
nal simulation-estimation and external action-perception does The hypothesized intermediate neurocomputational step of
not require the preservation of first-order isomorphism: only somatosensory estimation that lies between motor simulation
the one-to-one relation in the transformation of internal rep- and auditory estimation has also been suggested by recent exper-
resentation from motor to perceptual systems is required, as if iments. The sequential neural activity underlying somatosensory
the action was actually performed and the percept was actually and auditory estimation has been observed during articulation
induced. imagery using MEG (Tian and Poeppel, 2010), as discussed
The isomorphic transformation from motor to perceptual sys- above (Figure 2). Lesions over the left pars opercularis (pOp)
tems relies on the established internal associations between motor in the inferior frontal gyrus (IFG) as well as adjacent to the
and perceptual representations, which are presumably formed left supramarginal gyrus (SMG) in parietal cortex correlate with
following the causal and ecologically valid sequential occurrence the ability to imagine speech; this demonstrates the possible
of action-perception pairs, through the mechanisms of associa- neural implementation underlying the proposed simulation and
tive learning (Mahon and Caramazza, 2008). For example, the (somatosensory) estimation (Geva et al., 2011). Moreover, the
movement of articulators can induce somatosensory feedback causal role of somatosensory feedback in speech perception
and subsequent auditory perception of one’s own speech. On the has also been demonstrated (Ito et al., 2009). There, partic-
basis of the occurrence order (action first, then somatosensory ipants were asked to listen to ambiguous stimuli (e.g., head-
activation, followed by auditory perception), an internal associa- had vowel continuum) while their facial skin was manipulated
tion can be established to link a particular movement trajectory with a robotic device. When the skin at the side of mouth
of articulators with the specific somatosensory sensation, fol- was stretched upward (as in the case of pronouncing “head”),
lowed by a given auditory perception of speech. Note that we participants were biased toward hearing the ambiguous sound
do not exclude the possible existence of a parallel estimation as “head.” That is, the somatosensory status affected the audi-
process that links motor simulation to somatosensory and auditory perception in a systematic way: there was a one-to-one
tory systems separately (Guenther et al., 2006; Price et al., 2011). representational mapping between somatosensory and auditory
Such an additional mechanisms which runs in parallel may medi- systems.
ate the early comparison between auditory estimation from an
articulatory plan and intended auditory targets during speech THE SIMULATION-ESTIMATION PROCESS IN PERCEPTION
production (Hickok, 2012). The redundancy of the compensa- The debates surrounding motor theories of perception and cog-
tion in somatosensory and auditory domains offers a hint for nition [see the review by Scheerer (1984)] have heated up since
the co-existence of sequential and parallel estimation structures the discovery of the putative “mirror neuron system” in mon-
(Lametti et al., 2012). We suggest that the serial updating struc- keys (di Pellegrino et al., 1992; Gallese et al., 1996; see Rizzolatti
ture as one of the possible underlying estimation mechanisms and Craighero, 2004 for a review) and the observation of motor
naturally follows the biological sequences, providing advantages activity observed during numerous perceptual studies in humans
in learning and plasticity during development as well as online (e.g., Rizzolatti et al., 1996; Iacoboni et al., 1999; Buccino et al.,
speech control. 2001; Wilson et al., 2004). Although these debates are beyond
Speech-induced suppression and enhancement caused by the scope of this review, the proposed mechanism of sequen-
feedback perturbation provides strong evidence for the one-to- tial estimation following motor simulation may provide insight
one mapping between motor simulation and estimation of per- to reconcile some of the observations, providing a top-down
ceptual consequences. When participants speak and listen to their perspective.
own speech, the evoked auditory responses are smaller compared We propose, building on arguments in the recent litera-
with the auditory responses to the same speech played back with- ture (Mahon and Caramazza, 2008; Hickok, 2009; Lotto et al.,
out spoken outputs (Numminen et al., 1999; Houde et al., 2002; 2009; Rumiati et al., 2010), that the deployment of motor sim-
Eliades and Wang, 2003, 2005; Ventura et al., 2009). However, ulation in perceptual tasks is (1) strategy-dependent and (2)
when the auditory feedback is perturbed (manipulating, e.g., exerts modulatory effects on the formation of perceptual rep-
pitch or format frequencies), the auditory responses during resentations. That is, the selection of motor involvement in
speaking become larger compared with the ones during playback perceptual tasks depends on context and task demands. It is
(Eliades and Wang, 2008; Tourville et al., 2008; Zheng et al., 2010; a top-down strategic step to provide modality-specific repre-
Behroozmand et al., 2011). The suppression caused by articu- sentations in advance (cf. Moulton and Kosslyn, 2009) and
lation demonstrates that an internal signal labels the onset of reduce perceptual variance by generating more precise estima-
movement and down-regulates sensitivity to subsequent auditory tion (Mahon and Caramazza, 2008; but also see Pulvermüller
and Fadiga, 2010 for an opposite view from a embodied perception of movement (Heilman et al., 1982). A deficit in
perspective). gesture recognition has also been linked to inferior parietal
cortex lesions but not lesions in the frontal lobe (Buxbaum
The implementation of motor-to-sensory transformations et al., 2005). Action comprehension also relies on a network
is strategy-dependent that includes inferior parietal cortex but not IFG (Saygin et al.,
We describe two types of evidence. First, the recruit- 2004). Although patients with IFG lesions demonstrated deficits
ment/involvement of motor simulation is influenced by in action comprehension in the same study, the static stim-
task demands. For example, motor imagery can be performed uli (pictures of pantomimed actions or objects) could require
from a “first person” perspective that relies on kinesthetic feeling, participants to implement the strategy of motor simulation to
in contrast with when a task is executed from a “third person” form the dynamic display of action and to derive the perceptual
perspective in which the action-related visual changes are recre- consequences so that they can fulfill the action-object associ-
ated (Jeannerod, 1994, 1995). Reaction times of hand rotation ation task. Such lesion results indicate that a damaged motor
imagery showed an interaction between imagery perspectives system (and the deficits in motor simulation) dissociates from
and limb posture: when asked to imagine rotating their hands action-perception and comprehension. The abstract meaning of
from first person perspective, participants responded faster when motor action is probably “stored” in parietal regions, and the
their hands were on the lap but slower when their hands are motor simulation mediated by frontal regions is one of many
in the back; the reverse pattern was observed when imagining paths to access the stored representation (in line with our pro-
from third person perspective (Sirigu and Duhamel, 2001). posed simulation over frontal cortex and estimation over pari-
Activation in the motor system was observed when participants etal cortex). Therefore, motor simulation to estimate perceptual
were explicitly told to imagine rotating an object with their own consequences is only modulatory and not necessary for percep-
hands, but was absent when they were told to imagine rotating tual tasks.
the same object with a robotic motor (Kosslyn et al., 2001). Analogous to the advantage of multisensory integration in
Both behavioral and neuroimaging results highlight that the minimizing perceptual variance (Ernst and Banks, 2002; Alais
task demands influence the implementation of neural pathways and Burr, 2004; van Wassenhove et al., 2005; von Kriegstein and
that mediate either direct simulation (memory retrieval) or Giraud, 2006; Morgan et al., 2008; Poeppel et al., 2008; Fetsch
motor simulation-estimation (transformation between motor et al., 2009), the modulatory effects of motor simulation con-
and perceptual systems). vey benefits by providing additional, more detailed information
Second, motor-to-sensory transformations are influenced by to enrich the perceptual representation using internal sequen-
context and the properties of stimuli. For example, neural tial estimation mechanism (cf. Mahon and Caramazza, 2008).
responses in frontal motor regions have been observed dur- Human observers can adopt motor strategies to provide more
ing observation of meaningful actions, contrasted with occipital precise perceptual representations and deal with perceptual ambi-
activity for meaningless actions (Decety et al., 1997). Relatedly, guity, for example in the case of speech perception. That is, the
when participants mentally rotated their hands, premotor, pri- motor simulation and estimation can provide improved priors to
mary motor, and posterior parietal cortices were activated. reduce perceptual variance.
However, frontal motor areas were silent when they mentally In summary, various perceptual tasks can use the motor system
rotated objects (Kosslyn et al., 1998). These results suggest to derive perceptual consequences, by implementing the same
that contextual influence and task demands can determine the top-down motor simulation and perceptual estimation mecha-
implementation of motor simulation in a top-down, voluntary, nism, as in mental imagery of speech. We hypothesize that this
strategic way. motor simulation is modulatory and only serves as one of many
In the context of action observation, understand- possible corridors to induce perceptual representations. Such
ing/comprehension and imitation could be the result of strategies of sensory-to-motor and motor-to-sensory transfor-
heuristic engagement of motor simulation. That is, humans mation would be implemented depending on task demands and
can deploy a top-down mechanism that transfers perceptual contextual influence.
goals into the motor domain and initiates motor simulation to
derive perceptual consequences (Figure 3). The strategic and IMPLICATIONS FOR THE NEURAL CORRELATES OF SOME
heuristic initiation of motor involvement can be considered as a DISORDERS
top-down mental imagery process (possibly exclusive to humans) In this section we argue that the internal processes of motor
(cf. Iacoboni et al., 1999; Papeo et al., 2009), wherein the motor simulation and estimation, revealed originally for the mental
action is internally simulated and perceptual consequences imagery of speech, can shed light on possible neural correlates
estimated thereafter (cf. Tkach et al., 2007). of certain disorders, including auditory hallucinations, stutter-
ing, and phantom limb syndrome. We outline some working
Modulatory function of motor simulation on perception hypotheses regarding these disorders, complementing other exist-
The major evidence supporting a modulatory role of motor ing hypotheses. It is suggested that the proposed idea for mental
simulation in perception (rather than a primary causal role) imagery generation, motor simulation, and sequential percep-
comes from lesion studies. For example, lesions in the frontal tual estimation, points to the practical value of mental imagery
lobe only caused deficits in action production, whereas lesions research for understanding the internal mechanisms of such
in the parietal lobe caused deficits both during production and neural disorders.
AUDITORY HALLUCINATIONS: INTACT ESTIMATION VERSUS BROKEN motor simulation, but their functional roles are still separate. The
MONITORING efference copy is used to estimate the detailed perceptual conse-
Internal simulation and sequential estimation has been proposed quences, whereas the corollary discharge labels the internally and
to be a way to distinguish between the perceptual changes externally induced perceptual consequences.
caused by self-generated actions and exogenous external events Empirically, the finding that auditory hallucination patients
(Blakemore and Frith, 2003; Jeannerod and Pacherie, 2004; can generate inner speech (e.g., Shergill et al., 2003) demon-
Tsakiris and Haggard, 2005). The perceptual consequences of strates the relatively intact motor-to-sensory transformation
intended movement can be predicted, and the processing of function. The neural responses in IFG and superior temporal
external sensory feedback can be dampened by the internal pre- gyrus/sulcus (STG/STS) were observed during auditory halluci-
diction, such as in the case of speech production (e.g., Houde nations, hinting at the derivation of auditory perceptual con-
et al., 2002; Eliades and Wang, 2003, 2005) and somatosensory sequences from motor simulation during the positive symptom
perception in tickling (e.g., Blakemore et al., 1998). This sug- (e.g., McGuire et al., 1993; Shergill et al., 2003). Moreover, the
gests that the action-induced perceptual signals are identified as left lateralization during covert speech versus right lateraliza-
self-generated and cancelled by the virtually identical represen- tion during auditory hallucinations offers tantalizing hints about
tation generated by internal perceptual prediction. However for the independence between self-monitoring and the sequential
patients suffering from auditory hallucinations, deficits of these simulation-estimation (Sommer et al., 2008).
hypothesized dampening mechanisms for self-induced percep- We summarize the hypothetical mechanistic account for audi-
tual changes have been observed in both somatosensory (e.g., tory hallucinations (of this type) as follows: when patients prepare
Blakemore et al., 2000) and auditory (e.g., Ford et al., 2007; to articulate speech covertly or subvocally (either consciously or
Heinks-Maldonado et al., 2007) domains. These results sug- unconsciously), the internal motor simulation leads to percep-
gest that patients with auditory hallucinations cannot separate tual estimation (intact efference copy). But the source monitoring
self-induced from external-induced perceptual signals. process malfunctions (broken corollary discharge). Therefore, the
Critically, deficits of distinguishing self-induced from exter- internal prediction of a perceptual consequence, which has the
nally induced perceptual changes are not enough to account for same neural representation as an external perception, is erro-
auditory hallucinations, because the positive symptoms typically neously interpreted as the result of external sources, resulting in
occur in the absence of any external stimuli. There must exist an an auditory hallucination.
internal mechanism to induce the auditory representations that
are then misattributed to an external source/voice. In fact, we STUTTERING: NOISY ESTIMATION AND CORRECTION PROCESSES
face a similar situation during mental imagery: the neural rep- The comparison between internal estimation and external feed-
resentations mediating perception and mental imagery are very back provides information to fine-tune motor control. However,
similar, but there is no mechanism in the perceptual system to if the internal estimate from motor simulation malfunctions
distinguish them. A source monitoring function is required to and generates imprecise perceptual predictions, an inaccurate or
keep track of the origins of the perceptual neural representation. incorrect feedback control signal would be conveyed. Stuttering
Therefore, we hypothesize that a higher order function monitors could be an example of such erroneous correction. We suggest,
and distinguishes internally versus externally induced neural rep- along the lines of similar theories (Max et al., 2004; Hickok et al.,
resentations. Such a monitoring operation is functionally inde- 2011), that one of the neural mechanisms causing stuttering is
pendent from the perceptual estimation process that internally a deficit in the motor-to-sensory transformation. That is, the
reconstructs the perceptual representation. Under this hypoth- noisy perceptual estimation is mismatched to the external feed-
esis, auditory hallucinations are caused by incorrect operation back. Such a discrepancy would signal an incorrect error message,
of the monitoring function, resulting in incorrectly labeling the and the feedback control system would interpret such an appar-
self-induced auditory representation during the intact internal ent error as the requirement to correct motor action. Hence,
perceptual estimation processes. unnecessary attempts would be performed to modify the cor-
Computationally, the independence of the monitoring func- rect articulation, resulting in repetitive/prolonged sound or silent
tion versus internal simulation and estimation is demonstrated pauses/blocks.
by the nuanced differences between corollary discharge and the The noise in the estimation process can come both from the
efference copy [see the review by Crapse and Sommer (2008)]. somatosensory and auditory domains (since there is sequential
The efference copy is a duplicate of the planned motor com- estimation). Stutterers showed speed and latency deficits when
mand and provides the dynamics of an action trajectory that can required to sequentially update articulator movement (Caruso
be used to estimate the perceptual consequences (von Holst and et al., 1988). Smaller magnitude compensation with longer
Mittelstaedt, 1950, 1973). Corollary discharge is a more general latency adjustment to the perturbation on the jaws was also
motor related mechanism that can be available at all levels of a observed in stutterers (Caruso et al., 1987). In the auditory
motor process. The corollary discharge does not necessarily con- domain, smaller magnitude compensation to the perturbation of
tain the same representational information as an efference copy; F1 formant in auditory feedback is observed (Cai et al., 2012).
rather, it serves as a generic signal to inform sensory-perceptual The inaccurate compensation to external perturbation in both
systems of the potential occurrence of perceptual changes caused somatosensory and auditory domains (with intact somatosensory
by one’s own actions (Sperry, 1950). In the case of speech and auditory processes) demonstrates that inaccurate prediction
articulation, these two functions originate at the same stage of in both domains could be causal for stuttering.
Interestingly, dramatically altering auditory feedback (e.g., reduction has been demonstrated to correlate with more granular
by delaying feedback onset or shifting frequency) can enhance organization in the same areas (MacIver et al., 2008). Motor
speech fluency in people who stutter (Martin and Haroldson, imagery can lead to cortical reorganization that correlates with
1979; Stuart et al., 1997, 2008). The improvement could be pain reduction in phantom limbs (Moseley, 2006). Seeing the
because the magnitude of error signals is scaled down when movement of the opposite functioning arm in a mirror can reduce
the distance between feedback and prediction is beyond some the pain associated with the phantom limb (Ramachandran et al.,
threshold, so that fewer correction attempts are made. 1995). Such behavioral and psychological training can provide
more precise topographic maps in both motor and somatosen-
PHANTOM LIMBS: MISMATCH BETWEEN INTERNAL ESTIMATION sory cortices and hence reduce the inaccurate motor firing caused
AND EXTERNAL FEEDBACK by the “take over” effect (e.g., cortex of lip movement expand to
The mismatch between internal prediction and external feedback the cortex mediated a lost hand), as well as erroneous somatosen-
could also be caused by an acute change of conditions leading sory estimation. The internal estimation hypothesis offers a new
to the absence of feedback. One such example is the phantom perspective on pain induction. However, there is neither a clear
limb phenomenon, where amputees feel control over a lost limb pain center (Mazzola et al., 2012) nor a mechanistic pain induc-
(phantom limb) accompanied with chronic and sometimes acute tion account (Flor, 2002). Further research is needed to under-
pain. We hypothesize that the apparent awareness and control of stand how the proposed mismatch hypothesis could underpin
a lost limb occurs as follows: the missing somatosensory feedback pain induction.
is “replaced” by the results of internal estimation (cf. Frith et al.,
2000; Fotopoulou et al., 2008). Such a hypothesis is similar to CONCLUSION
the mislabeling of the internal estimation as an external percep- In this perspective, we argued that mental imagery is an inter-
tion (due to the malfunction of source monitoring) in auditory nal predictive process. Using mental imagery of speech as an
hallucinations. example, we demonstrated a variety of principles underlying
The causes of pain in phantom limbs are more intriguing. how the mechanism of motor simulation and sequential percep-
The most significant physical changes are loss of propriocep- tual estimation in mental imagery works. We conclude that the
tion, or somatosensory afference, after lost limbs. Because motor simulation-estimation mechanism provides a novel conceptual
control as well as motor simulation of the lost limb are still in and practical perspective that allows for new types of research
some sense valid (e.g., Raffin et al., 2012), we hypothesize that on predictive functions and sensory-motor integration, as well
a mismatch between the intact internal estimation and absent as stimulating some new insights into several neural disorders.
external somatosensory feedback can cause the pain associated Typically, mental imagery has been studied in cognitive psychol-
with phantom limbs. In fact, consistent with our hypothesis, ogy and cognitive neuroscience, while the concepts of internal
limb pain can be induced in normal participants by mismatch- forward models (and sensory-motor integration) are the focus
ing visual and proprioceptive feedback (McCabe et al., 2005) of motor control research from an engineering perspective. Our
and spinal cord injured patients report that neuropathic pain atypical pairing of internal models as an additional source for
increases while they imagine moving their ankles (Gustin et al., mental imagery yields, in our view, some provocative new angles
2008). on mental imagery in both basic research and applied contexts.
This mismatch hypothesis may represent an intermediate step
between cortical reorganization and pain induction. Lost limbs ACKNOWLEDGMENTS
cause reorganization in both motor (Maihöfner et al., 2007) This study was supported by MURI ARO #54228-LS-MUR and
and somatosensory (Maihöfner et al., 2003) cortices, and pain NIH 2R01DC 05660.
REFERENCES Blakemore, S. J., Smith, J., Steel, Buxbaum, L. J., Kyle, K. M., and during speech in stutterers. Brain
Alais, D., and Burr, D. (2004). The ven- R., Johnstone, E., and Frith, C. Menon, R. (2005). On beyond 111, 439.
triloquist effect results from near- (2000). The perception of self- mirror neurons: internal represen- Caruso, A. J., Gracco, V., and Abbs, J.
optimal bimodal integration. Curr. produced sensory stimuli in patients tations subserving imitation and H. (1987). “A speech motor control
Biol. 14, 257–262. with auditory hallucinations and recognition of skilled object-related perspective on stuttering: prelimi-
Behroozmand, R., Liu, H., and Larson, passivity experiences: evidence for actions in humans. Cogn. Brain Res. nary observations,” in Speech Motor
C. R. (2011). Time-dependent neu- a breakdown in self-monitoring. 25, 226–239. Dynamics in Stuttering, eds H. F.
ral processing of auditory feedback Psychol. Med. 30, 1131–1139. Cai, S., Beal, D. S., Ghosh, S. S., M. Peters and W. Hulstijn (Wien,
during voice pitch error detection. Blakemore, S. J., Wolpert, D. M., and Tiede, M. K., Guenther, F. H., Austria: Springer-Verlag), 245–258.
J. Cogn. Neurosci. 23, 1205–1217. Frith, C. D. (1998). Central cancel- and Perkell, J. S. (2012). Weak Crapse, T. B., and Sommer, M. A.
Bensafi, M., Porter, J., Pouliot, S., lation of self-produced tickle sensa- responses to auditory feedback per- (2008). Corollary discharge across
Mainland, J., Johnson, B., Zelano, tion. Nat. Neurosci. 1, 635–640. turbation during articulation in the animal kingdom. Nat. Rev.
C., et al. (2003). Olfactomotor activ- Buccino, G., Binkofski, F., Fink, G. persons who stutter: evidence for Neurosci. 9, 587–600.
ity during imagery mimics that dur- R., Fadiga, L., Fogassi, L., Gallese, abnormal auditory-motor transfor- Decety, J. (1996). The neurophysiolog-
ing perception. Nat. Neurosci. 6, V., et al. (2001). Action observa- mation. PLoS ONE 7:e41830. doi: ical basis of motor imagery. Behav.
1142–1144. tion activates premotor and pari- 10.1371/journal.pone.0041830 Brain Res. 77, 45–52.
Blakemore, S. J., and Frith, C. (2003). etal areas in a somatotopic manner: Caruso, A. J., Abbs, J. H., and Gracco, Decety, J., Grezes, J., Costes, N., Perani,
Self-awareness and action. Curr. an fMRI study. Eur. J. Neurosci. 13, V. L. (1988). Kinematic analysis D., Jeannerod, M., Procyk, E.,
Opin. Neurobiol. 13, 219–224. 400–404. of multiple movement coordination et al. (1997). Brain activity during
observation of actions. Influence vestibular cues during self-motion Gustin, S. M., Wrigley, P. J., Gandevia, mechanism for motor cognition.
of action content and subject’s perception. J. Neurosci. 29, 15601. S. C., Middleton, J. W., Henderson, Neuroimage 14, S103–S109.
strategy. Brain 120, 1763. Fisher, J. C. (2006). Does simulation L. A., and Siddall, P. J. (2008). Jeannerod, M., and Pacherie, E.
Decety, J., Jeannerod, M., and Prablanc, theory really involve simulation? Movement imagery increases pain (2004). Agency, simulation and
C. (1989). The timing of mentally Philos. Psychol. 19, 417–432. in people with neuropathic pain self-identification. Mind Lang. 19,
represented actions. Behav. Brain Flor, H. (2002). Phantom-limb pain: following complete thoracic spinal 113–146.
Res. 34, 35–42. characteristics, causes, and treat- cord injury. Pain 137, 237–244. Kosslyn, S. M. (1994). Image and
Decety, J., and Michel, F. (1989). ment. Lancet Neurol. 1, 182–189. Hanakawa, T., Immisch, I., Toma, K., Brain: The Resolution of the Imagery
Comparative analysis of actual Ford, J. M., Gray, M., Faustman, W. Dimyan, M. A., Van Gelderen, P., Debate. Cambridge, MA: MIT Press.
and mental movement times in O., Roach, B. J., and Mathalon, and Hallett, M. (2003). Functional Kosslyn, S. M. (2005). Mental images
two graphic tasks. Brain Cogn. 11, D. H. (2007). Dissecting corol- properties of brain areas associated and the brain. Cogn. Neuropsychol.
87–97. lary discharge dysfunction in with motor execution and imagery. 22, 333–347.
Decety, J., Perani, D., Jeannerod, M., schizophrenia. Psychophysiology 44, J. Neurophysiol. 89, 989–1002. Kosslyn, S. M., Alpert, N. M.,
Bettinardi, V., Tadary, B., Woods, R., 522–529. Heilman, K. M., Rothi, L. J., and Thompson, W. L., Chabris, C.
et al. (1994). Mapping motor rep- Fotopoulou, A., Tsakiris, M., Haggard, Valenstein, E. (1982). Two forms F., Rauch, S. L., and Anderson, A.
resentations with positron emission P., Vagopoulou, A., Rudd, A., and of ideomotor apraxia. Neurology 32, K. (1994). Identifying objects seen
tomography. Nature 371, 600–602. Kopelman, M. (2008). The role of 342–342. from different viewpoints A PET
Dechent, P., Merboldt, K. D., and motor intention in motor aware- Heinks-Maldonado, T. H., Mathalon, investigation. Brain 117, 1055.
Frahm, J. (2004). Is the human ness: an experimental study on D. H., Houde, J. F., Gray, M., Kosslyn, S. M., Digirolamo, G. J.,
primary motor cortex involved in anosognosia for hemiplegia. Brain Faustman, W. O., and Ford, J. M. Thompson, W. L., and Alpert,
motor imagery? Cogn. Brain Res. 19, 131, 3432–3442. (2007). Relationship of imprecise N. M. (1998). Mental rotation of
138–144. Foxe, J. J., Wylie, G. R., Martinez, corollary discharge in schizophre- objects versus hands: neural mech-
Desmurget, M., Reilly, K. T., Richard, A., Schroeder, C. E., Javitt, D. nia to auditory hallucinations. Arch. anisms revealed by positron emis-
N., Szathmari, A., Mottolese, C., C., Guilfoyle, D., et al. (2002). Gen. Psychiatry 64, 286. sion tomography. Psychophysiology
and Sirigu, A. (2009). Movement Auditory-somatosensory multi- Hesslow, G. (2002). Conscious thought 35, 151–161.
intention after parietal cortex stim- sensory processing in auditory as simulation of behaviour and Kosslyn, S. M., Ganis, G., and
ulation in humans. Science 324, 811. association cortex: an fMRI study. perception. Trends Cogn. Sci. 6, Thompson, W. L. (2001). Neural
Desmurget, M., and Sirigu, A. (2009). J. Neurophysiol. 88, 540. 242–247. foundations of imagery. Nat. Rev.
A parietal-premotor network for Frith, C. D., Blakemore, S. J., Hickok, G. (2009). Eight problems Neurosci. 2, 635–642.
movement intention and motor and Wolpert, D. M. (2000). for the mirror neuron theory of Kosslyn, S. M., Pascual-Leone, A.,
awareness. Trends Cogn. Sci. 13, Abnormalities in the awareness action understanding in monkeys Felician, O., Camposano, S.,
411–419. and control of action. Philos. Trans. and humans. J. Cogn. Neurosci. 21, Keenan, J., Ganis, G., et al. (1999).
di Pellegrino, G., Fadiga, L., Fogassi, R. Soc. Lond. B Biol. Sci. 355, 1229–1243. The role of area 17 in visual
L., Gallese, V., and Rizzolatti, G. 1771–1788. Hickok, G. (2012). Computational neu- imagery: convergent evidence from
(1992). Understanding motor Fu, K. M. G., Johnston, T. A., Shah, A. roanatomy of speech production. PET and rTMS. Science 284, 167.
events: a neurophysiological study. S., Arnold, L., Smiley, J., Hackett, Nat. Rev. Neurosci. 13, 135–145. Kosslyn, S. M., and Thompson, W. L.
Exp. Brain Res. 91, 176–180. T. A., et al. (2003). Auditory corti- Hickok, G., Houde, J., and Rong, (2003). When is early visual cor-
Djordjevic, J., Zatorre, R., Petrides, cal neurons respond to somatosen- F. (2011). Sensorimotor integra- tex activated during visual men-
M., Boyle, J., and Jones-Gotman, sory stimulation. J. Neurosci. 23, tion in speech processing: computa- tal imagery? Psychol. Bull. 129,
M. (2005). Functional neuroimag- 7510–7515. tional basis and neural organization. 723–746.
ing of odor imagery. Neuroimage 24, Gallese, V., Fadiga, L., Fogassi, L., and Neuron 69, 407–422. Kosslyn, S. M., Thompson, W. L.,
791–801. Rizzolatti, G. (1996). Action recog- Houde, J. F., Nagarajan, S. S., Sekihara, Wraga, M., and Alpert, N. M.
Ehrsson, H. H., Geyer, S., and Naito, E. nition in the premotor cortex. Brain K., and Merzenich, M. M. (2002). (2001). Imagining rotation by
(2003). Imagery of voluntary move- 119, 593. Modulation of the auditory cor- endogenous versus exogenous
ment of fingers, toes, and tongue Gerardin, E., Sirigu, A., Lehericy, S., tex during speech: an MEG study. forces: distinct neural mechanisms.
activates corresponding body-part- Poline, J.-B., Gaymard, B., Marsault, J. Cogn. Neurosci. 14, 1125–1138. Neuroreport 12, 2519.
specific motor representations. C., et al. (2000). Partially overlap- Iacoboni, M., Woods, R. P., Brass, Kraemer, D. J. M., Macrae, C. N.,
J. Neurophysiol. 90, 3304–3316. ping neural networks for real and M., Bekkering, H., Mazziotta, J. C., Green, A. E., and Kelley, W. M.
Eliades, S. J., and Wang, X. (2003). imagined hand movements. Cereb. and Rizzolatti, G. (1999). Cortical (2005). Musical imagery: sound of
Sensory-motor interaction in Cortex 10, 1093–1104. mechanisms of human imitation. silence activates auditory cortex.
the primate auditory cortex dur- Geva, S., Jones, P. S., Crinion, J. Science 286, 2526. Nature 434, 158.
ing self-initiated vocalizations. T., Price, C. J., Baron, J. C., and Ishai, A., Haxby, J. V., and Ungerleider, Lametti, D. R., Nasir, S. M., and
J. Neurophysiol. 89, 2194. Warburton, E. A. (2011). The neural L. G. (2002). Visual imagery of Ostry, D. J. (2012). Sensory prefer-
Eliades, S. J., and Wang, X. (2005). correlates of inner speech defined by famous faces: effects of memory ence in speech production revealed
Dynamics of auditory–vocal inter- voxel-based lesion–symptom map- and attention revealed by fMRI. by simultaneous alteration of audi-
action in monkey auditory cortex. ping. Brain 134, 3071–3082. Neuroimage 17, 1729–1741. tory and somatosensory feedback.
Cereb. Cortex 15, 1510. Goldman, A. I. (1989). Interpretation Ito, T., Tiede, M., and Ostry, D. J. J. Neurosci. 32, 9351–9358.
Eliades, S. J., and Wang, X. (2008). psychologized. Mind Lang. 4, (2009). Somatosensory function in Lotto, A. J., Hickok, G. S., and Holt,
Neural substrates of vocalization 161–185. speech perception. Proc. Natl. Acad. L. L. (2009). Reflections on mir-
feedback monitoring in primate Grush, R. (2004). The emulation theory Sci. U.S.A. 106, 1245. ror neurons and speech perception.
auditory cortex. Nature 453, of representation: motor control, Jeannerod, M. (1994). The represent- Trends Cogn. Sci. 13, 110–114.
1102–1106. imagery, and perception. Behav. ing brain: neural correlates of motor Lotze, M., Montoya, P., Erb, M.,
Ernst, M. O., and Banks, M. S. (2002). Brain Sci. 27, 377–396. intention and imagery. Behav. Brain Hulsmann, E., Flor, H., Klose,
Humans integrate visual and haptic Guenther, F. H., Ghosh, S. S., and Sci. 17, 187–202. U., et al. (1999). Activation of
information in a statistically opti- Tourville, J. A. (2006). Neural Jeannerod, M. (1995). Mental cortical and cerebellar motor
mal fashion. Nature 415, 429–433. modeling and imaging of the imagery in the motor context. areas during executed and imag-
Fetsch, C. R., Turner, A. H., DeAngelis, cortical interactions underlying Neuropsychologia 33, 1419–1432. ined hand movements: an fMRI
G. C., and Angelaki, D. E. (2009). syllable production. Brain Lang. 96, Jeannerod, M. (2001). Neural sim- study. J. Cogn. Neurosci. 11,
Dynamic reweighting of visual and 280–301. ulation of action: a unifying 491–501.
MacIver, K., Lloyd, D., Kelly, S., Moulton, S. T., and Kosslyn, S. M. deficits and their lesion correlates. movement implicates the dynam-
Roberts, N., and Nurmikko, T. (2009). Imagining predictions: Neuropsychologia 42, 1788–1804. ics of internal forward models.
(2008). Phantom limb pain, cortical mental imagery as mental emula- Scheerer, E. (1984). “Motor theories Front. Psychology 1:166. doi:
reorganization and the therapeutic tion. Philos. Trans. R. Soc. B Biol. of cognitive structure: a histori- 10.3389/fpsyg.2010.00166
effect of mental imagery. Brain 131, Sci. 364, 1273–1280. cal review,” in Cognition and Motor Tian, X., Poeppel, D., and Huber, D.
2181. Nikulin, V. V., Hohlefeld, F. U., Processes, eds W. Prinz and A. E. (2011). TopoToolbox: using sen-
Mahon, B. Z., and Caramazza, A. Jacobs, A. M., and Curio, G. Sanders (Berlin: Springer-Verlag), sor topography to calculate psycho-
(2008). A critical look at the embod- (2008). Quasi-movements: a novel 77–98. logically meaningful measures from
ied cognition hypothesis and a new motor-cognitive phenomenon. Schroeder, C. E., Lindsley, R. W., event-related EEG/MEG. Comput.
proposal for grounding concep- Neuropsychologia 46, 727–742. Specht, C., Marcovici, A., Smiley, Intell. Neurosci. 2011, 8.
tual content. J. Physiol. Paris 102, Numminen, J., Salmelin, R., and Hari, J. F., and Javitt, D. C. (2001). Tkach, D., Reimer, J., and Hatsopoulos,
59–70. R. (1999). Subject’s own speech Somatosensory input to auditory N. G. (2007). Congruent activity
Maihöfner, C., Baron, R., DeCol, R., reduces reactivity of the human association cortex in the macaque during action and action observa-
Binder, A., Birklein, F., Deuschl, G., auditory cortex. Neurosci. Lett. 265, monkey. J. Neurophysiol. 85, 1322. tion in motor cortex. J. Neurosci. 27,
et al. (2007). The motor system 119–122. Shepard, R. N., and Cooper, L. A. 13241.
shows adaptive changes in complex O’Craven, K. M., and Kanwisher, N. (1986). Mental Images and Their Tourville, J. A., Reilly, K. J., and
regional pain syndrome. Brain 130, (2000). Mental imagery of faces Transformations. Cambridge, MA: Guenther, F. H. (2008). Neural
2671. and places activates correspond- The MIT Press. mechanisms underlying audi-
Maihöfner, C., Handwerker, H. O., ing stimulus-specific brain regions. Shergill, S. S., Brammer, M. J., Fukuda, tory feedback control of speech.
Neundörfer, B., and Birklein, F. J. Cogn. Neurosci. 12, 1013–1023. R., Williams, S. C. R., Murray, R. Neuroimage 39, 1429–1443.
(2003). Patterns of cortical reor- Papeo, L., Vallesi, A., Isaja, A., and M., and McGuire, P. K. (2003). Tsakiris, M., and Haggard, P. (2005).
ganization in complex regional Rumiati, R. I. (2009). Effects Engagement of brain areas impli- Experimenting with the acting
pain syndrome. Neurology 61, of TMS on different stages of cated in processing inner speech in self. Cogn. Neuropsychol. 22,
1707–1715. motor and non-motor verb pro- people with auditory hallucinations. 387–407.
Martin, R., and Haroldson, S. K. cessing in the primary motor Br. J. Psychiatry 182, 525–531. van Wassenhove, V., Grant, K. W., and
(1979). Effects of five experimental cortex. PLoS ONE 4:e4508. doi: Sirigu, A., Cohen, L., Duhamel, J. R., Poeppel, D. (2005). Visual speech
treatments on stuttering. J. Speech 10.1371/journal.pone.0004508 Pillon, B., Dubois, B., Agid, Y., speeds up the neural processing of
Hear. Res. 22, 132. Poeppel, D., Idsardi, W. J., and van et al. (1995). Congruent unilateral auditory speech. Proc. Natl. Acad.
Max, L., Guenther, F. H., Gracco, Wassenhove, V. (2008). Speech per- impairments for real and imagined Sci. U.S.A. 102, 1181.
V. L., Ghosh, S. S., and Wallace, ception at the interface of neurobi- hand movements. Neuroreport 6, Ventura, M., Nagarajan, S., and Houde,
M. E. (2004). Unstable or insuf- ology and linguistics. Philos. Trans. 997–1001. J. (2009). Speech target modulates
ficiently activated internal models R. Soc. Lond. B Biol. Sci. 363, 1071. Sirigu, A., and Duhamel, J. (2001). speaking induced suppression in
and feedback-biased motor control Price, C. J., Crinion, J. T., and Motor and visual imagery as two auditory cortex. BMC Neurosci.
as sources of dysfluency: a theoret- MacSweeney, M. (2011). A gener- complementary but neurally disso- 10:58. doi: 10.1186/1471-2202-
ical model of stuttering. Contemp. ative model of speech production ciable mental processes. J. Cogn. 10-58
Issues Commun. Sci. Disord. 31, in Broca’s and Wernicke’s areas. Neurosci. 13, 910–919. von Holst, E., and Mittelstaedt, H.
105–122. Front. Psychology 2:237. doi: Sirigu, A., Duhamel, J., Cohen, L., (1950). Daz Reafferenzprinzip.
Mazzola, L., Isnard, J., Peyron, R., and 10.3389/fpsyg.2011.00237 Pillon, B., Dubois, B., and Agid, Wechselwirkungen zwischen
Mauguière, F. (2012). Stimulation Pulvermüller, F., and Fadiga, L. (2010). Y. (1996). The mental representa- Zentralnerven-system und Peri-
of the human cortex and the expe- Active perception: sensorimotor cir- tion of hand movements after pari- pherie. Naturwissenschaften 37,
rience of pain: Wilder Penfield’s cuits as a cortical basis for language. etal cortex damage. Science 273, 467–476.
observations revisited. Brain 135, Nat. Rev. Neurosci. 11, 351–360. 1564–1568. von Holst, E., and Mittelstaedt, H.
631–640. Raffin, E., Mattout, J., Reilly, K. T., and Sommer, I. E. C., Diederen, K. M. J., (1973). The Reafference Principle
McCabe, C., Haigh, R., Halligan, P., Giraux, P. (2012). Disentangling Blom, J. D., Willems, A., Kushan, L., (R. Martin, Trans.). The Behavioral
and Blake, D. (2005). Simulating motor execution from motor Slotema, K., et al. (2008). Auditory Physiology of Animals and Man: The
sensory–motor incongruence in imagery with the phantom limb. verbal hallucinations predomi- Collected Papers of Erich von Holst.
healthy volunteers: implications Brain 135, 582–595. nantly activate the right inferior (Coral Gables, FL: University of
for a cortical model of pain. Ramachandran, V. S., Rogers- frontal area. Brain 131, 3169. Miami Press), 139–173.
Rheumatology 44, 509. Ramachandran, D., and Cobb, Sperry, R. (1950). Neural basis of the von Kriegstein, K., and Giraud, A. L.
McGuire, P., Murray, R., and Shah, S. (1995). Touching the phantom spontaneous optokinetic response (2006). Implicit multisensory asso-
G. (1993). Increased blood flow in limb. Nature 377, 489. produced by visual inversion. ciations influence voice recognition.
Broca’s area during auditory hallu- Rizzolatti, G., and Craighero, L. (2004). J. Comp. Physiol. Psychol. 43, 482. PLoS Biol. 4:e326. doi: 10.1371/jour-
cinations in schizophrenia. Lancet The mirror-neuron system. Annu. Stuart, A., Frazier, C. L., Kalinowski, J., nal.pbio.0040326
342, 703–706. Rev. Neurosci. 27, 169–192. and Vos, P. W. (2008). The effect of Wilson, S. M., Saygin, A. P., Sereno,
Meister, I. G., Krings, T., Foltys, H., Rizzolatti, G., Fadiga, L., Matelli, M., frequency altered feedback on stut- M. I., and Iacoboni, M. (2004).
Boroojerdi, B., Müller, M., Töpper, Bettinardi, V., Paulesu, E., Perani, tering duration and type. J. Speech Listening to speech activates
R., et al. (2004). Playing piano D., et al. (1996). Localization of Lang. Hear. Res. 51, 889. motor areas involved in speech
in the mind: an fMRI study on grasp representations in humans Stuart, A., Kalinowski, J., and Rastatter, production. Nat. Neurosci. 7,
music imagery and performance by PET: 1. Observation versus M. P. (1997). Effect of monaural and 701–702.
in pianists. Cogn. Brain Res. 19, execution. Exp. Brain Res. 111, binaural altered auditory feedback Wolpert, D. M., and Ghahramani,
219–228. 246–252. on stuttering frequency. J. Acoust. Z. (2000). Computational
Morgan, M. L., DeAngelis, G. C., and Rumiati, R. I., Papeo, L., and Corradi- Soc. Am. 101, 3806. principles of movement neu-
Angelaki, D. E. (2008). Multisensory Dell’Acqua, C. (2010). Higher-level Tian, X., and Huber, D. E. (2008). roscience. Nat. Neurosci. 3,
integration in macaque visual cortex motor processes. Ann. N.Y. Acad. Measures of spatial similarity 1212–1217.
depends on cue reliability. Neuron Sci. 1191, 219–241. and response magnitude in MEG Yoo, S. S., Freeman, D. K., McCarthy,
59, 662–673. Saygin, A. P., Wilson, S. M., Dronkers, and scalp EEG. Brain Topogr. 20, J. J. 3rd., and Jolesz, F. A. (2003).
Moseley, G. L. (2006). Graded motor N. F., and Bates, E. (2004). 131–141. Neural substrates of tactile imagery:
imagery for pathologic pain. Action comprehension in apha- Tian, X., and Poeppel, D. (2010). a functional MRI study. Neuroreport
Neurology 67, 2129–2134. sia: linguistic and non-linguistic Mental imagery of speech and 14, 581.
Zatorre, R. J., and Halpern, A. R. (2004). Multisensory cortical Conflict of Interest Statement: The simulation and estimation. Front. Hum.
(2005). Mental concerts: musi- processing of object shape and authors declare that the research Neurosci. 6:314. doi: 10.3389/fnhum.
cal imagery and auditory cortex. its relation to mental imagery. was conducted in the absence of any 2012.00314
Neuron 47, 9–12. Cogn. Affect. Behav. Neurosci. 4, commercial or financial relationships Copyright © 2012 Tian and Poeppel.
Zatorre, R. J., Halpern, A. R., Perry, 251–259. that could be construed as a potential This is an open-access article dis-
D. W., Meyer, E., and Evans, Zheng, Z. Z., Munhall, K. G., conflict of interest. tributed under the terms of the Creative
A. C. (1996). Hearing in the and Johnsrude, I. S. (2010). Commons Attribution License, which
mind’s ear: a PET investigation Functional overlap between Received: 20 April 2012; accepted: 06 permits use, distribution and repro-
of musical imagery and per- regions involved in speech per- November 2012; published online: 28 duction in other forums, provided
ception. J. Cogn. Neurosci. 8, ception and in monitoring one’s November 2012. the original authors and source are
29–46. own voice during speech pro- Citation: Tian X and Poeppel D (2012) credited and subject to any copyright
Zhang, M., Weisser, V. D., Stilla, duction. J. Cogn. Neurosci. 22, Mental imagery of speech: linking motor notices concerning any third-party
R., Prather, S., and Sathian, K. 1770–1781. and perceptual systems through internal graphics etc.
Effect of biomechanical constraints in the hand laterality

judgment task: where does it come from?
Gilles Vannuscorps 1,2*, Agnesa Pillon 1,2 and Michael Andres 2,3
1
Institute of Psychological Sciences and Institute of Neuroscience, Université catholique de Louvain, Louvain-la-Neuve, Belgium
2
Fonds National de la Recherche Scientifique, Brussels, Belgium
3
Faculty of Psychology, Department of Experimental Psychology, Ghent University, Ghent, Belgium
Edited by: Several studies have reported that, when subjects have to judge the laterality of
Joel Pearson, The University of rotated hand drawings, their judgment is automatically influenced by the biomechanical
constraints of the upper limbs. The prominent account for this effect is that, in order
Reviewed by:
to perform the task, subjects mentally rotate their upper limbs toward the position of
Bettina E. Bläsing, Bielefeld
University, Germany the displayed stimulus in a way that is consistent with the biomechanical constraints
Joel Pearson, The University of underlying the actual movement. However, the effect of such biomechanical constraints
New South Wales, Australia was also found in the responses of motor-impaired individuals performing the hand
*Correspondence: laterality judgment (HLJ) task, which seems at odds with the “motor imagery” account for
Gilles Vannuscorps, Institut de
this effect. In this study, we further explored the source of the biomechanical constraint
Recherche en Sciences
Psychologiques, Université effect by assessing the ability of an individual (DC) with a congenital absence of upper
catholique de Louvain, Place du limbs to judge the laterality of rotated hand or foot drawings. We found that DC was
Cardinal Mercier 10, as accurate and fast as control participants in judging the laterality of both hand and
Louvain-la-Neuve 1348, Belgium.
e-mail: gilles.vannuscorps@
foot drawings, without any disadvantage for hands when compared to feet. Furthermore,
uclouvain.be DC’s response latencies (RLs) for hand drawings were influenced by the biomechanical
constraints of hand movements in the same way as control participants’ RLs. These
results suggest that the effect of biomechanical constraints in the HLJ task is not strictly
dependent on “motor imagery” and can arise from the visual processing of body parts
being sensitive to such constraints.
Keywords: motor imagery, motor simulation, biomechanical knowledge, aplasia, body part perception
INTRODUCTION network known for its contribution to the planning and exe-
In the past 20 years, an increased interest has been devoted to the cution of hand movements (Kosslyn et al., 1998; Parsons et al.,
study of motor imagery. Most research has been achieved by the 1998; de Lange et al., 2006). On the basis of this behavioral
means of the hand laterality judgment (HLJ) task (Cooper and and neural evidence, the HLJ task was considered as a privileged
Shepard, 1975; Sekiyama, 1982; Parsons, 1987, 1994). In this task, tool to read out the unconscious and normally covert process of
participants are asked to decide whether drawings of different motor planning (Jeannerod, 1994; Fiorio et al., 2006; de Lange
hand postures rotated with different angles from the upright view et al., 2008; Munzert et al., 2009). In this paper, we report evi-
depict a left or a right limb (see Figure 1). If a hand drawing were dence that calls for a re-examination of the prominent “motor
processed like any other visual object, response latencies (RLs) imagery” account of the effect of biomechanical constraints in the
should increase as a function of the angular disparity between HLJ task.
the hand drawing and its upright view, as evidenced for other 2D The effect of biomechanical constraints on participants’ judg-
or 3D stimuli (Cooper and Shepard, 1975). Instead, it has been ments in the HLJ task is characterized by three features unveiled
shown that the impact of the angular disparity on the speed of by the analysis of the RL pattern (Parsons, 1987; Funk and
HLJs was strongly modulated by the biomechanical limits that Brugger, 2008; Pelgrims et al., 2009). First, the RLs for the hand
constrain the movement of the hand toward the displayed posi- drawings depicted in the various rotation angles (from 0◦ to 360◦
tion (Sekiyama, 1982; Parsons, 1987). This observation was taken in a clockwise direction) from the upright display (0◦ ) are modu-
as evidence that, to solve the task, participants internally simulated by both the specific posture of the depicted hand (e.g., view
late a movement of their own hand and, moreover, that mental of the hand side from the thumb vs. view of the palm from the
imagery of human movements—i.e., motor imagery—relies on wrist) and its laterality (right vs. left hand). This three-way inter-
the same representations and processes as those involved in action action reflects the different biomechanical constraints that limit
planning and/or control (Parsons, 1987, 1994; Jeannerod and the amplitude of the rotation angle of each hand in a given pos-
Frak, 1999; Kosslyn et al., 2001; Nico et al., 2004; Wraga et al., ture. Thus, for instance, the recognition of a side view takes more
2005; de Lange et al., 2006, 2008; Fiorio et al., 2006; Helmich et al., time for right than left hands if the drawing is rotated at 150◦
2007; Munzert et al., 2009). Furthermore, neuroimaging studies clockwise given that such an angle corresponds to the outcome
showed that HLJs induced increased activity in a parieto-frontal of a possible left hand movement but it is almost impossible to
Vannuscorps et al. Biomechanics in hand laterality judgments
directions. The third feature is the significant correlation between

the RL to a given stimulus and its degree of awkwardness that is,
how difficult participants rated it to actually place their own hand
in the displayed position (Parsons, 1987).
There is, however, evidence showing that the effect of biome-
chanical constraints in the HLJ task can be observed even in the
condition of impaired motor planning or execution processes.
After transient disruption of motor-related areas with transcra-
nial magnetic stimulation (TMS), the performance in the HLJ
task was either normal (Sauner et al., 2006) or characterized by
a small RL increase that nevertheless did not hamper the effect
of biomechanical constraints (Ganis et al., 2000; Pelgrims et al.,
2010). Furthermore, studies of patients suffering from motor dis-
orders that prevent normal execution of hand movements such
as congenital hemiparesis (Steenbergen et al., 2007), Parkinson’s
disease (Helmich et al., 2007), dystonia (Fiorio et al., 2006), con-
version paralysis (de Lange et al., 2008), amputation of an upper
limb (Nico et al., 2004), or chronic arm pain (Schwoebel et al.,
2001) showed that these conditions delayed HLJs but, again, with-
out affecting the effect of biomechanical constraints. Finally, a
study using the HLJ task with two individuals suffering from
bilateral upper limb aplasia reported an effect of biomechanical
constraints on RLs in at least one individual (Funk and Brugger,
2008).
Evidence for an effect of biomechanical constraints in the HLJ
task despite impaired motor planning or execution raises the
possibility that the effect does not arise from motor but instead
from visual processes that ensure the perception of human body
parts. Several authors indeed suggested that, when representing
the human body in whole or in parts, the visual system encodes
information about the biomechanical constraints of body part
movements, information that in turn constrains whole body or
body part perception (Marr and Vaina, 1982; Kourtzi and Shiffrar,
1999). Evidence advanced in support of this view includes find-
ings from the apparent motion paradigm. When an object is
displayed sequentially in two different positions, it elicits in par-
ticipants the perception of an apparent motion along the shortest
pathway (Kolers and Pomerantz, 1971). However, the apparent
FIGURE 1 | Line drawings of left and right hands in four postures and
motion induced by two hand postures presented sequentially can
seven rotation angles. In blue: lateral orientations; in green: medial
orientations.
follow a longer pathway when the shortest one is not biome-
chanically possible (Shiffrar and Freyd, 1990, 1993). According to
some authors (Shiffrar and Freyd, 1993; Chatterjee et al., 1996),
achieve with the right hand in this position. Likewise, the palm this effect indicates that implicit perceptual knowledge of how
of a right hand viewed from the wrist, with the thumb point- the body moves impacts how body parts are perceived (but see
ing down toward the right side of the screen (i.e., between 210◦ Stevens et al., 2000, for an alternative account in terms of motor
and 270◦ clockwise) is recognized more slowly than a left hand simulation).
displayed in the same orientation, in line with the fact that it is However, before considering the perceptual hypothesis as an
much more difficult to reach this orientation with the right than alternative account for the effect of biomechanical constraints
with the left hand in this posture. The second feature indexing in the HLJ task, a number of ambiguities that are present in
the effect of biomechanical constraints on participants’ responses the aforementioned studies of motor-impaired or aplasic indi-
is the overall chronometric advantage for judging the laterality of viduals must be addressed. First, one cannot rule out that the
hands oriented in medial positions (stimuli rotated toward the motor-impaired patients in whom the effect of biomechanical
mid-sagittal plane) when compared to lateral positions (stim- constraints was found actually suffered from a deficit affecting
uli rotated away of the mid-sagittal plane). This effect, called processes that take place after the stage where biomechanical
the “Medial Over Lateral Advantage” (MOLA) effect, reflects the constraints influence motor planning and/or execution. Indeed,
impact of the biomechanical constraints of hand movements that motor execution was not totally abolished in these patients. At
make it easier to move one’s hand toward medial than lateral first sight, the effect found in a bilateral aplasic individual (Funk
and Brugger, 2008) should not present this ambiguity since, in as possible whether the drawing corresponded to a right or a
such condition, none of the processes involved in the planning left hand.
and execution of hand movements is functional. Nevertheless, Stimuli were left or right hands, presented according to four
this individual (AZ) reported vivid phantom sensations of her different postures (dorsal view, palm view, side from thumb
missing body parts that include phantom movements corre- view, and palm from wrist view) and at 7 different rotation
sponding to those of normal upper limbs (Funk et al., 2005). angles (upright 0◦ and 30◦ , 90◦ , 150◦ , 210◦ , 270◦ , and 330◦ in a
Thus, one cannot rule out the existence of limb representa- clockwise direction, from Parsons, 1987; see Figure 1). The total
tions in AZ, which could explain her ability to carry out motor number of different stimuli was thus of 56 hands (2 laterality ×
imagery for congenitally absent limbs (Brugger et al., 2000; Funk 4 postures × 7 angles). Ratings of motor awkwardness for each
and Brugger, 2008). Furthermore, and this is also true for the hand drawing were extracted from a previous study (Parsons,
studies with motor-impaired patients, the issue of whether mere 1987), in which judges were asked to position their own hand at
visual familiarity with the various hand positions could have con- the orientation of each stimulus and, afterwards, to estimate the
tributed to the observed pattern of RLs in the HLJ task was not awkwardness of the reached position on a 5-point scale (1 = easy
addressed. What looks like an effect of awkwardness might in fact to place the appropriate limb into the orientation of the stimulus
be an artifact of the differential occurrences of the various hand and 5 = difficult to place the appropriate limb into the orienta-
positions—awkward hand positions are also likely to be less often tion of the stimulus). Ratings of visual familiarity were collected
seen than easy ones. Within this visual familiarity account, the in 25 students of the Université catholique de Louvain (8 males)
RLs in the HLJ task would mainly depend on how often a given who were asked to rate how often they saw a hand in each posture
hand position has been seen in everyday settings, with the more and angle of rotation in everyday life (1 = very unfamiliar and 5 =
frequent hand positions being recognized faster as a right or a left very familiar). DC’s personal visual familiarity was collected sep-
hand than the less frequent ones. arately following the same procedure; it was strongly correlated
In this study, we sought further evidence for the presence of to the students’ ratings [r(56) = 0.53; p < 0.001]. As a control, we
the effect of biomechanical constraints in the HLJ task in the con- also tested foot laterality judgments using stimuli that were found
text of a motor disability to plan and execute hand movements. to evoke implicit activation of the biomechanical constraints of
We presented the HLJ task to DC, a man born without upper foot movements in seminal studies of mental imagery (Parsons,
limbs, as well as to 7 normally limbed control participants, and 1987). Fifty-six drawings of left and right feet were presented at
analyzed their pattern of RLs vis-a-vis the main features indexing the same angle of rotation as hand drawings, according to four
the effect of biomechanical constraints. Because of the congeni- similar postures (sole view, top view, view from inside, and sole
tal disability of DC, who has also never experienced any phantom from heel view).
limb sensation, the present study overcomes the difficulties raised Participants seated in front of a computer screen located at a
by previous studies with motor-impaired patients or aplasic indi- distance of about 60 cm; their feet were lying at rest on the ground
viduals. In the case of DC, no hand motor planning or execution and the hands of control participants were placed palms down on
ability of any kind could be invoked to explain, if any, an effect their knees without visual feedback.
of biomechanical constraints in the HLJ task. Furthermore, we During the experiment, participants performed 5 blocks of 56
examined the effect of biomechanical constraints on the RLs of trials with hand drawings and then 5 blocks of 56 trials with foot
DC and control participants by taking into account the potential drawings. In each block, all postures and rotation angles were
effect of the rated visual familiarity of hand positions. mixed in a different pseudo-randomized order. The first blocks of
hand and foot laterality judgments included a familiarization with
MATERIALS AND METHODS the four postures and 10 practice trials. Within each block, each
PARTICIPANTS trial started with the presentation of a central cross for 200 ms
DC is a 51 year-old man with a Master’s Degree in Psychology. He followed by a hand or foot drawing displayed until a response
presents a congenital bilateral upper limb aplasia (right side: two was recorded. Trials were separated by a blank screen of random
fingers attached to a foreshortened humerus; left side: completely duration between 500 ms to 1000 ms.
aplasic) due to in utero thalidomide exposure. He had no experi- The experiment was controlled with the E-Prime software
ence of prosthesis or phantom limb sensations. His performance (Psychological Software, 2002, Pittsburgh, PA). Stimuli were pre-
was compared with that of 7 right-handed, normally limbed sented on a 15.4 inch laptop screen set at 1024 × 768 pixels and
control participants matched in gender, age (mean age = 53.5), subtended 5◦ of visual angle. During the testing, participants
and educational level. All participants had a normal or corrected were asked to produce a verbal response (“right” or “left”). The
to normal vision and no history of psychiatric or neurologi- RLs corresponded to the post-stimulus onset latency of the sub-
cal disorder. The study was approved by the biomedical ethic ject’s vocalization, whose amplitude was electrically compared
committee of the Cliniques universitaires Saint-Luc (Brussels) to a trip level voltage using a voice key controlled by E-prime.
and all participants gave written informed consent prior to the Malfunctioning of the voice key and response accuracy were
study. monitored on-line by the experimenter.
TASKS AND STIMULI RESULTS

Participants were presented with drawings of a hand outlined Voice key failures (0% and 0.6% of the data in DC and con-
in black on a white background and asked to decide as fast trols, respectively), trials with RLs deviating more than 2 standard
deviations from the mean RL within each participant (5.14% and direction) as between-item factors. For the analysis of DC’s per-
4.43% of the data in DC and controls, respectively), and trials formance, the data were inverse transformed to fulfill the criteria
with errors were discarded from RL analyses. of homoscedasticity and normality. The results replicated the sig-
nificant three-way interaction between LATERALITY, ANGLE, and
GENERAL ANALYSIS OF DC’S AND CONTROLS’ PERFORMANCE POSTURE , not only in control participants [F(18, 108) = 3.77, p <
First of all, we performed Crawford and Howell’s (1998) mod- 0.001] but also in DC [F(18, 202) = 2.69, p < 0.001]. This three-
ified t-tests to test whether DC’s performance (accuracy and way interaction showed that the effect of the angle of rotation on
speed) in hand and foot laterality judgments was impaired in RLs was not symmetric for left and right hands, with the angle
comparison to the control group’s performance. In hand judg- associated to the maximal increase varying as a function of hand
ments, no significant difference was observed between the perfor- posture.
mance of DC (correct responses: 97%; mean RL: 1237 ms) and In order to further exemplify the effect of biomechanical con-
control participants [mean % correct responses ± SD: 91% ± straints, we decomposed the three-way interaction as a function
4%, modified t(6) = 1.56, p > 0.1; mean RL ± SD: 1277 ms ± of hand posture. In control participants, the log transformed data
345 ms, modified t(6) = −0.11, p > 0.9]. The analysis of foot were analyzed separately for each hand posture using repeated
judgments showed a similar pattern, with a non-significant dif- measure ANOVAs with SUBJECT as a random factor and LATER-
ference in accuracy or RLs between DC (correct responses: ALITY and ANGLE as within-subject factors. The inverse trans-
97%; mean RL: 1512 ms) and controls [mean % of correct formed data of DC were entered in similar ANOVAs with ITEM
responses ± SD: 90% ± 6%, modified t(6) = 1.13, p > 0.3; as a random factor and LATERALITY and ANGLE as between-
mean RL ± SD: 1446 ms ± 386 ms, modified t(6) = 16, p > 0.8]. item factors. In control participants, a significant LATERALITY by
Second, Crawford and Garthwaite’s (2005) Revised Standardized ANGLE interaction was found for the palm from the wrist view
Difference Test (RSDT) was applied to test whether the differ- [F(6, 36) = 4.32, p < 0.01] and for the palm view [F(6, 36) = 7.68,
ence between hand and foot laterality judgments in DC deviated p < 0.001]. A near significant interaction effect was observed
from the difference observed between hand and foot judgments for the side from the thumb view [F(6, 36) = 2.15, p = 0.07]
in the control group. The results showed that DC’s differ- but, for the dorsal view, no significant interaction was found
ence in performance between hand and foot drawings was not [F(6, 36) = 1.38, p > 0.2]. The performance of DC also revealed
significantly different from that found in the control partici- a significant LATERALITY by ANGLE interaction for the palm
pants, either in accuracy [RSDT: t(6) = 0.31, p > 0.7] or speed from wrist view [F(6, 51) = 3.98, p < 0.01] as well as for the side
[RSDT: t(6) = 0.4, p > 0.7]. Third, independent samples t-tests from the thumb view [F(6, 56) = 2.5, p < 0.05], but not for the
were performed in order to compare the RLs across posture, palm [F(6, 41) = 1.49, p > 0.2] or dorsal view [F(6, 54) < 1]. In
laterality, and angle of rotation for foot and hand judgments Figure 2, we represented the pattern of RLs of control partici-
in DC and controls, respectively. The results indicated that pants (upper panel) and DC (lower panel) for the two postures
both DC [t(67.76) = −2.99, p < 0.01] and control participants that showed a significant LATERALITY by ANGLE interaction in
[t(84.1) = −4.29, p < 0.01] were significantly faster for hand than DC after decomposition of the three-way interaction. The Figure
for foot judgments. clearly illustrates that the angle of rotation had a different effect
on RLs depending on hand laterality. In both controls (Figure 2A)
EFFECT OF BIOMECHANICAL CONSTRAINTS ON DC’s AND CONTROLS’ and DC (Figure 2C), the side view of a left hand led to maxi-
RLs mal RLs at angle 210◦ , whereas the slowest responses for the side
Having shown that the performance of DC in HLJs was within the view of a right hand were observed at angle 150◦ . In contrast,
normal range, we looked for the presence of the three behavioral the palm from the wrist views showing the longest RLs ranged
features that were classically reported as evidence for an effect of from angles 90◦ to 150◦ for the left hand and from angles 210◦
biomechanical constraints in the HLJ task. to 270◦ for the right hand (Figures 2B,D). The comparison of
First, we investigated the presence of a three-way interaction Figures 2B and 2D showed that maximal RLs for the palm from
between LATERALITY, ANGLE, and POSTURE in the chronometric the wrist views were not observed exactly at the same angles in
data gathered for all participants. To do so, the RLs of con- controls and DC, with a RL curve skewed to angle 150◦ for DC.
trol participants were entered in a repeated measure analysis of Except for this slight difference, the angles associated to maxi-
variance (ANOVA) with SUBJECT as the random factor and LAT- mal RLs in DC and controls reflect the most difficult positions
ERALITY (left vs. right), POSTURE (dorsal view vs. palm view vs. to reach while adopting the displayed posture with one’s left or
palm from the wrist view vs. side from the thumb view), and right hand (see the awkwardness estimates provided by Parsons,
ANGLE with respect to the upright view (0◦ –330◦ , in a clockwise 1987).
direction) as within-subject factors. The data of control par- DC did not show the expected pattern of asymmetric RL
ticipants required a log transformation to satisfy the ANOVA’s curves for the left and right hands viewed from the dorsal or
homoscedasticity and normality assumptions. In order to explore from the palm view. However, the absence of LATERALITY by
the effects of biomechanical constraints in DC’s judgments, we ANGLE interaction for these two postures was already pointed
performed an ANOVA with ITEM as the random factor and LAT- out in previous experiments with normally limbed participants
ERALITY (left vs. right), POSTURE (dorsal view vs. palm view vs. (Parsons, 1987; see for discussion ter Horst et al., 2010). Likewise,
palm from the wrist view vs. side from the thumb view), and in our control participants, the effect of angular disparity was
ANGLE with respect to the upright view (0◦ –330◦ , in a clockwise not modulated by hand laterality for dorsal views and a look at
FIGURE 2 | Mean response latency for left (blue) and right (red) hand drawings rotated clockwise by steps of 60◦ viewed from the side (A,C) and from
the wrist (B,D). The pattern of response latency in control participants and DC are shown in the upper (A,B) and lower (C,D) panels, respectively.
the individual data for the palm views revealed that the expected
interaction was observed in only 3 out of 7 participants.
Second, in order to test for the presence of the MOLA effect,
we recoded the trials according to the “medial” or “lateral” ori-
entation of the displayed hand posture with respect to the body
mid-sagittal plane (cf. Figure 1: medial orientations are displayed
in green and lateral orientations in blue). In that way, the anal-
yses were performed on the mean RLs calculated for each 24
medial and each 24 lateral hand displays (2 hands × 3 angles ×
4 postures). Figure 3 shows the mean RLs associated to medial
and lateral orientations in DC and every control participant.
Independent samples t-tests revealed that both control partic-
ipants [medial: 1191 ms ± 134 ms; lateral: 1345 ms ± 139 ms; FIGURE 3 | Mean response latency and standard deviation (vertical
t(46) = 3.89, p < 0.001] and DC [medial: 1205 ms ± 274 ms; lat- bars) are shown as a function of hand orientation (lateral vs. medial).
eral 1378 ms ± 335; t(46) = 1.95, p = 0.057] responded faster The data for the lateral and medial orientation were obtained by pooling the
to hand drawings in medial than in lateral orientation, which data from all postures and rotation angles corresponding to the lateral (blue)
and medial (green) positions depicted in Figure 1. Individual data (C =
mimicked the effect of biomechanical constraints on actual hand Control; CTRs = mean of controls) are aligned in ascending order on the
movements. Furthermore, RSDT indicated that the MOLA effect X axis as a function of the size of the medial over lateral advantage.
did not significantly differ in size between DC and controls
[RSDT: t(6) = 0.13, p > 0.9].
Third, we calculated the correlation between the mean RL analyses showed that the RLs of both controls [r(56) = 0.55,
obtained for each of the 56 items and the estimates of motor awk- p < 0.001] and DC [r(56) = 0.52, p < 0.001] were significantly
wardness collected independently for the same items (Parsons, correlated to the motor awkwardness of the hand drawing. A
1987). The correlation plots are displayed in Figures 4A,B. The final set of analyses was performed in order to partial out the
FIGURE 4 | Correlation plots between the mean response latency of controls (left panels) and DC (right panels) for each hand stimulus in relation to
rated awkwardness (A,B) and visual familiarity (C,D).
respective influence of motor awkwardness and visual familiar- the performance of DC, a bilateral aplasic individual, with the
ity on RLs (see Figures 4C,D). We conducted stepwise regression performance of control participants in the HLJ task. First, we
and partial correlation analyses between RLs and, respectively, looked at the overall performance and found that DC was as fast
ratings of motor awkwardness and visual familiarity. The regres- and accurate as the control participants. Second, we found that
sion analysis performed in control participants [F(1, 54) = 23.94, (1) DC’s RLs were influenced by the laterality, the angle, and
p < 0.001, R2 = 0.31] showed that the motor awkwardness the posture of hand stimuli, in a way that mirrors the biome-
associated to a given hand drawing was the best predictor of the chanical limits imposed by each posture on left and right hand
observed RLs (β = 0.55, t = 4.89, p < 0.001). The contribution movements; (2) DC’s RLs showed a chronometric advantage
of visual familiarity was not significant (β = −0.04, t = −0.35, of medial over lateral hand orientations, reflecting the differ-
p > 0.7). Likewise, the regression of DC’s RLs [F(1, 54) = 19.87, ence of movement amplitude allowed by these two orientations;
p < 0.001, R2 = 0.27] revealed a significant effect of motor awk- and (3) DC’s RLs were strongly predicted by the motor awk-
wardness (β = 0.52, t = 4.46, p < 0.001) in the context of a wardness of the stimuli but not by their visual familiarity. To
non-significant contribution of visual familiarity (β = 0.01, t = sum up, DC showed the three behavioral features that were
0.11, p > 0.9). Similar results were observed when DC’s own rat- classically reported as evidence for an effect of biomechani-
ings of visual familiarity were taken into account in the regression cal constraints in the HLJ task. These effects were qualitatively
equation [F(1, 54) = 19.87, p < 0.001, R2 = 0.27]. Furthermore, and quantitatively comparable to the effects observed in con-
partial correlations showed that the correlation between RLs and trol participants and they cannot be explained by the differ-
awkwardness estimates remained significant in controls [r(53) = ential occurrences of the various hand positions in everyday
0.54, p < 0.001] and DC [r(53) = 0.51, p < 0.001] after control- life.
ling for the part of variance explained by the effect of visual
familiarity. DISCUSSION
The effect of biomechanical constraints in the HLJ task is com-
SUMMARY OF RESULTS monly assumed to reflect a process of motor simulation anchored
The goal of this study was to test whether the effect of biome- in the same processes and representations as those involved
chanical constraints in the HLJ task is strictly dependent on hand in actual action planning and execution (Parsons, 1987, 1994;
motor planning and execution abilities. To do so, we compared Jeannerod and Frak, 1999; Kosslyn et al., 2001; Nico et al., 2004;
Wraga et al., 2005; de Lange et al., 2006, 2008; Fiorio et al., 2006; by the different body parts (Marr and Vaina, 1982; Shiffrar and
Helmich et al., 2007; Munzert et al., 2009). In this paper, we found Freyd, 1990, 1993; Kourtzi and Shiffrar, 1999). Such information
that a person born without upper limbs was as accurate and fast might have an adaptive value for humans because it facilitates
as a group of control participants in performing the HLJ task the anticipation of the outcome of movements performed by oth-
and that his RLs in this task were significantly influenced by the ers (Kourtzi and Shiffrar, 1999). The type of body representation
biomechanical constraints of upper limb movements, just like the we propose to explain the results of DC should not be confused
RLs of normally limbed participants. These findings show that with the “body schema” because this representation refers specifi-
the effect of biomechanical constraints in the HLJ task is not cally to one’s own body (Corradi-Dell’Acqua and Tessari, 2010; de
strictly dependent on representations and processes involved in Vignemont, 2010). It is also different from the “body structural
the planning and execution of hand movements. Given his total description” (i.e., a visuospatial representation of body parts)
lack of motor experience with upper limb movements, and also of and the “body image” (i.e., a conceptual representation of the
phantom limb experience, DC is not endowed with such motor body) because none of these representations include knowledge
representations and processes and could therefore not rely on of the biomechanical constraints of the body (Sirigu et al., 1991;
them to perform the task. Schwoebel and Coslett, 2005).
Two kinds of accounts for the effect of biomechanical con- The role of visual processes in the effect of biomechanical con-
straints in DC can be dismissed. First, this effect is not an artifact straints in the perception of body parts was already emphasized
of the visual familiarity of the various hand positions. We found by Brugger and colleagues in order to explain the influence of
that DC’s RLs were better predicted by the degree of motor awk- such constraints on HLJs (Brugger et al., 2000; Funk and Brugger,
wardness associated to each hand drawing than by their visual 2008) and apparent motion perception (Funk et al., 2005) in
familiarity. Furthermore, the correlation between his RLs and an bilateral aplasic individual, AZ, who experienced phantom
motor awkwardness estimates remained significant even after limb movement sensations. It should be noted, however, that
removing the influence of visual familiarity on the data, which this proposal deviates from the idea that biomechanical knowl-
indicated that the influence of motor awkwardness and visual edge is an intrinsic component of visual perception. In Brugger
familiarity does not fully overlap in HLJs. Second, our data allow and colleagues’ proposal, visual experience is assumed to acti-
us to rule out that the effect is due to DC performing the HLJ task vate pre-existing limb representations common to both action
by mentally rotating the representation of his feet. Seminal studies observation and execution, thereby allowing AZ to engage in
in fact showed that the motor awkwardness estimates associated a process of motor imagery in the HLJ task. Our finding that
to certain foot positions correlate with those gathered for the biomechanical constraints also affect the performance of a bilat-
homologue hand positions (see Tables 3 and 4 in Parsons, 1987). eral aplasic individual without phantom limb sensations in the
However, DC was 275 ms faster for hand than foot judgments. HLJ task makes it unnecessary to assume that the role of visual
This advantage of hand over foot responses even reached an aver- experience is mediated by processes involved in action planning
age of 583 ms for the side views that showed the typical effect of or execution.
biomechanical constraints1. In their study, Funk and Brugger (2008) also presented the
Our findings thus provide strong evidence for the presence HLJ task to a bilateral aplasic individual, CL, who did not expe-
of an effect of biomechanical constraints in the HLJ task in a rience phantom limb sensations, like DC, but, contrary to the
condition that totally prevents the planning and execution of results we reported here, they found no evidence for an effect
hand movements. Uncovering the source of this effect in such of biomechanical constraints in CL’s response pattern. These dis-
condition was beyond the scope of this study and we have no crepant results in the HLJ task deserve some methodological
direct evidence that speaks to this issue. Nevertheless, the find- considerations. While performing the task, CL was influenced
ing that knowledge of biomechanical constraints was implicitly by the rotation angle but not by the biomechanical constraints,
and automatically recruited in DC’s HLJs is consistent with the as evidenced by an absence of interaction between hand later-
view that such knowledge is an intrinsic component of body part ality and rotation angle. However, CL was tested with a short
visual perception processes. These perceptual processes would version of the HLJ task including only two postures (palm
provide us with a representation of the human body that takes vs. back) and four rotation angles (from 0◦ to 270◦ by steps
into account information about the range of movement allowed of 90◦ ). Previous research in healthy participants showed that
minor modifications in the stimulus set, such as the reduc-
1 Several authors (e.g., Tomasino et al., 2003; Steenbergen et al., 2007; Daprati tion of the number of rotation axes, can suppress the effect of
et al., 2010; ter Horst et al., 2010) argued that, when motor simulation is not biomechanical constraints (ter Horst et al., 2010). Furthermore,
possible (e.g., aplasia or severe motor deficit), alternative strategies may be laterality judgment for hands viewed from the back and the
used to perform the HLJ task. These strategies imply, for instance, mental palm are less regularly associated with an effect of the biome-
rotation of the visual stimulus (Tomasino et al., 2003), visual analysis based chanical constraints, even in normally limbed participants (e.g.,
on non-motor features of the hands such as the number of visible fingers Parsons, 1987; this study). It is therefore unclear whether the
or the position of the thumb and/or pinkie (Daprati et al., 2010), or third- effect of biomechanical constraints in CL’s performance was
person visual imagery of the action “as if someone else performed the action”
(Steenbergen et al., 2007). It is important to note, however, that all these visu-
not observed because he had no motor experience or because
ally based strategies predict no effect of biomechanical constraints on response the stimulus set allowed him to base his responses on strate-
latencies. Thus, any account for DC’s pattern in the HLJ task in terms of such gic processes that do not make use of biomechanical knowledge.
alternative strategies would be inappropriate. This observation underlines the need to assess the influence of
biomechanical knowledge on HLJs with an extended set of hand task. In the meanwhile, the HLJ task should not be considered
drawings. anymore as an unambiguous window on the covert stages of
Finally, our findings call for re-examining the motor imagery motor control and motor planning in normal or motor-impaired
account for the effect of biomechanical constraints in nor- individuals.
mally limbed individuals performing HLJs. Obviously, the effect
observed in DC and controls could be driven by distinct kinds ACKNOWLEDGMENTS
of representations and processes, i.e., visual vs. motor processes, We are indebted to DC and the control participants for their
both being sensitive to the biomechanics of the human body. motivation during the experiments carried out in this study. We
Nevertheless, we find it unlikely that an implicit and automatic are also very grateful to Lawrence Parsons who kindly provided
access to biomechanics, such as the one revealed in DC’s lat- us with the hand and foot drawings used in this study. Agnesa
erality judgments, would be totally suspended when normal Pillon is a senior researcher, Gilles Vannuscorps is a research
subjects perform the task. Our findings thus encourage fur- fellow, and Michael Andres was a post-doctoral researcher at
ther empirical studies to entertain the hypothesis that visual in the Fonds National de la Recherche Scientifique (FNRS). This
addition to motor processes, or even visual processes alone, con- research was supported by the FNRS grant no. 1.5.192.10F to
tribute to the effect of biomechanical constraints in the HLJ Agnesa Pillon.
REFERENCES motor imagery: an fMRI study. apparent motion. J. Exp. Psychol. 87, Pelgrims, B., Andres, M., and Olivier,
Brugger, P., Kollias, S. S., Müri, R. Neuroimage 33, 609–617. 99–108. E. (2009). Double dissocia-
M., Crelier, G., Hepp-Reymond, de Lange, F. P., Roelofs, K., and Toni, Kosslyn, S. M., DiGirolamo, G. J., tion between motor and visual
M. C., and Regard, M. (2000). I. (2008). Motor imagery: a window Thompson, W. L., and Alpert, N. M. imagery in the posterior pari-
Beyond re-membering: phantom into the mechanisms and alterations (1998). Mental rotation of objects etal cortex. Cereb. Cortex 19,
sensations of congenitally absent of the motor system. Cortex 44, versus hands: neural mechanisms 2298–2307.
limbs. Proc. Natl. Acad. Sci. U.S.A. 494–506. revealed by positron emission Pelgrims, B., Michaux, N., Olivier,
97, 6167–6172. de Vignemont, F. (2010). Body tomography. Psychophysiology 35, E., and Andres, M. (2010).
Chatterjee, S. H., Freyd, J. J., and schema and body image: pros 151–161. Contribution of the primary
Shiffrar, M. (1996). Configural and cons. Neuropsychologia 48, Kosslyn, S. M., Ganis, G., and motor cortex to motor imagery:
processing in the perception of 669–680. Thompson, W. L. (2001). Neural a subthreshold TMS study. Hum.
apparent biological motion. J. Exp. Fiorio, M., Tinazzi, M., and Aglioti, foundations of imagery. Nat. Rev. Brain Mapp. 32, 1802–1810.
Psychol. Hum. Percept. Perform. 22, S. M. (2006). Selective impairment Neurosci. 2, 635–642. Sauner, D., Bestmann, S., Siebner,
916–929. of hand mental rotation in patients Kourtzi, Z., and Shiffrar, M. (1999). H. R., and Rothwell, J. C. (2006).
Cooper, L. A., and Shepard, R. with focal hand dystonia. Brain 129, Dynamic representations of human No evidence for a substantial
N. (1975). Mental transforma- 47–54. body movement. Perception 28, involvement of primary motor
tion in the identification of Funk, M., and Brugger, P. (2008). 49–62. hand area in handedness judge-
left and right hands. J. Exp. Mental rotation of congenitally Marr, D., and Vaina, L. (1982). ments: a transcranial magnetic
Psychol. Hum. Percept. Perform. 1, absent hands. J. Int. Neuropsychol. Representation and recognition of stimulation study. Eur. J. Neurosci.
48–56. Soc. 14, 81–89. the movements of shapes. Proc. R. 23, 2215–2224.
Corradi-Dell’Acqua, C., and Tessari, Funk, M., Shiffrar, M., and Brugger, P. Soc. Lond. B Biol. Sci. 214, 501–524. Schwoebel, J., and Coslett, H. B.
A. (2010). Is the body in the eye (2005). Hand movement observa- Munzert, J., Lorey, B., and Zentgraf, (2005). Evidence for multiple,
of the beholder? Visual process- tion by individuals born without K. (2009). Cognitive motor pro- distinct representations of the
ing of bodies in individuals with hands: phantom limb expe- cesses: the role of motor imagery human body. J. Cogn. Neurosci. 17,
anomalous anatomical sensory and rience constrains visual limb in the study of motor repre- 543–553.
motor features. Neuropychologia 48, perception. Exp. Brain Res. 164, sentations. Brain Res. Rev. 60, Schwoebel, J., Friedman, R., Duda, N.,
689–702. 341–346. 306–326. and Coslett, H. B. (2001). Pain
Crawford, J. R., and Garthwaite, P. Ganis, G., Keenan, J. P., Kosslyn, S. Nico, D., Daprati, E., Rigal, F., Parsons, and the body schema: evidence for
H. (2005). Testing for suspected M., and Pascual-Leone, A. (2000). L., and Sirigu, A. (2004). Left peripheral effects on mental repre-
impairments and dissociations Transcranial magnetic stimulation and right hand recognition in sentations of movement. Brain 124,
in single-case studies in neu- of primary motor cortex affects upper limb amputees. Brain 127, 2098–2104.
ropsychology: evaluation of mental rotation. Cereb. Cortex 10, 120–132. Sekiyama, K. (1982). Kinesthetic
alternatives using monte carlo 175–180. Parsons, L. M. (1987). Imagined aspects of mental representations
simulations and revised tests for Helmich, R. C., de Lange, F. P., Bloem, spatial transformations of one’s in the identification of left and
dissociations. Neuropsychology 19, B. R., and Toni, I. (2007). Cerebral hands and feet. Cogn. Psychol. 19, right hands. Percept. Psychophys. 32,
318–331. compensation during motor 178–241. 89–95.
Crawford, J. R., and Howell, D. C. imagery in Parkinson’s disease. Parsons, L. M. (1994). Temporal Shiffrar, M., and Freyd, J. J. (1990).
(1998). Comparing an individ- Neuropsychologia 45, 2201–2215. and kinematic properties of Apparent motion of the human
ual’s test score against norms Jeannerod, M. (1994). The represent- motor behavior reflected in men- body. Psychol. Sci. 1, 257–264.
derived from small samples. Clin. ing brain: neural correlates of motor tally simulated action. J. Exp. Shiffrar, M., and Freyd, J. J. (1993).
Neuropsychol. 12, 482–486. intention and imagery. Behav. Brain Psychol. Hum. Percept. Perform. 20, Timing and apparent motion
Daprati, E., Nico, D., Duval, S., and Sci. 17, 187–245. 709–730. path choice with human body
Lacquaniti, F. (2010). Different Jeannerod, M., and Frak, V. (1999). Parsons, L. M., Gabrieli, J. D., Phelps, photographs. Psychol. Sci. 4,
motor imagery modes follow- Mental imaging of motor activity in E. A., and Gazzaniga, M. S. (1998). 379–384.
ing brain damage. Cortex 46, humans. Curr. Opin. Neurobiol. 9, Cerebrally lateralized mental Sirigu, A., Grafman, J., Bressler,
1016–1030. 735–739. representations of hand shape K., and Sunderland, T. (1991).
de Lange, F. P., Helmich, R. C., and Kolers, P. A., and Pomerantz, J. and movement. J. Neurosci. 18, Multiple representations con-
Toni, I. (2006). Posture influences R. (1971). Figural change in 6539–6548. tribute to body knowledge
processing. Evidence from a ter Horst, A. C., van Lier, R., and versus objects: an fMRI study. Citation: Vannuscorps G, Pillon A and
case of autotopagnosia. Brain 114, Steenbergen, B. (2010). Mental Neuropsychologia 43, 1351–1361. Andres M (2012) Effect of biomechan-
629–642. rotation task of hands: dif- ical constraints in the hand laterality
Steenbergen, B., van Nimwegen, M., ferential influence number of judgment task: where does it come
and Crajé, C. (2007). Solving a rotational axes. Exp. Brain Res. 203, Conflict of Interest Statement: The from? Front. Hum. Neurosci. 6:299. doi:
mental rotation task in congenital 347–354. authors declare that the research 10.3389/fnhum.2012.00299
hemiparesis: motor imagery versus Tomasino, B., Rumiati, R. I., and was conducted in the absence of any Copyright © 2012 Vannuscorps, Pillon
visual imagery. Neuropsychologia 45, Umilta, C. A. (2003). Selective commercial or financial relationships and Andres. This is an open-access arti-
3324–3328. deficit of motor imagery as tapped that could be construed as a potential cle distributed under the terms of the
Stevens, J. A., Fonlupt, P., Shiffrar, by a left-right decision of visually conflict of interest. Creative Commons Attribution License,
M. A., and Decety, J. (2000). New presented hands. Brain Cogn. 53, which permits use, distribution and
aspects of motion perception: selec- 376–380. reproduction in other forums, provided
tive neural encoding of apparent Wraga, M., Shephard, J. M., Church, Received: 04 May 2012; accepted: 14 the original authors and source are cred-
human movements. Neuroreport 11, J. A., Inati, S., and Kosslyn, S. M. October 2012; published online: 01 ited and subject to any copyright notices
109–115. (2005). Imagined rotations of self November 2012. concerning any third-party graphics etc.
doi: 10.3389/fpsyg.2012.00284
Understanding immersivity: image generation and

transformation processes in 3D immersive environments
Maria Kozhevnikov 1,2 * and Rupali P. Dhond 2
1
Psychology Department, National University of Singapore, Singapore
2
Department of Radiology, Martinos Center for Biomedical Imaging, Harvard Medical School, Charlestown, MA, USA
Edited by: Most research on three-dimensional (3D) visual-spatial processing has been conducted
using traditional non-immersive 2D displays. Here we investigated how individuals gen-
erate and transform mental images within 3D immersive (3DI) virtual environments, in
Reviewed by:
Natasha Sigala, University of Sussex, which the viewers perceive themselves as being surrounded by a 3D world. In Experiment
UK 1, we compared participants’ performance on the Shepard and Metzler (1971) mental
Adriane E. Seiffert, Vanderbilt rotation (MR) task across the following three types of visual presentation environments;
University, USA
traditional 2D non-immersive (2DNI), 3D non-immersive (3DNI – anaglyphic glasses), and
*Correspondence:
3DI (head mounted display with position and head orientation tracking). In Experiment 2,
Maria Kozhevnikov , Department of
Radiology, Martinos Center for we examined how the use of different backgrounds affected MR processes within the
Biomedical Imaging, Harvard Medical 3DI environment. In Experiment 3, we compared electroencephalogram data recorded
School, 149 Thirteenth Street, Suite while participants were mentally rotating visual-spatial images presented in 3DI vs. 2DNI
2301, Charlestown, MA 02129, USA.
environments. Overall, the findings of the three experiments suggest that visual-spatial
e-mail: mkozhevn@
nmr.mgh.harvard.edu processing is different in immersive and non-immersive environments, and that immersive
environments may require different image encoding and transformation strategies than the
two other non-immersive environments. Specifically, in a non-immersive environment, par-
ticipants may utilize a scene-based frame of reference and allocentric encoding whereas
immersive environments may encourage the use of a viewer-centered frame of reference
and egocentric encoding. These findings also suggest that MR performed in laboratory
conditions using a traditional 2D computer screen may not reflect spatial processing as it
would occur in the real world.
Keywords: mental rotation, immersivity, three-dimensional immersive virtual environments
INTRODUCTION in. Second, unlike non-immersive environments where a scene

Our ability to generate and transform three-dimensional (3D) is fixed on a 2D computer screen, 3DI involves image updating
visual-spatial images is important not only for our every-day achieved by position and head orientation tracking. Although little
activities (locomotion, navigation) but also for a variety of profes- is known about cognitive processes and neural dynamics under-
sional activities, such as architecture, air traffic control, and tele- lying image encoding and transformation in 3DI environments,
robotics. Difficulties of studying visual-spatial cognition within researchers have speculated that immersivity would differentially
real world environments, where controlling the experimental affect selection of a spatial frame of reference (i.e., spatial coordi-
stimuli and recording participants’ behavior is often impossi- nate system) during object encoding processes (Kozhevnikov and
ble, have led researchers to increasingly employ 3D immersive Garcia, 2011).
(3DI) virtual environments (Chance et al., 1998; Klatzky et al., Two different spatial frames of reference, environmental and
1998; Loomis et al., 1999; Tarr and Warren, 2002; Macuga et al., viewer-centered, can be used for encoding and transforming visual-
2007; Kozhevnikov and Garcia, 2011). Specifically, 3DI technology spatial images. An environmental frame may involve the “perma-
allows one to create a complex immersive environment of high nent environment” which is bound by standard orthogonal planes,
ecological validity, in which participants are presented with and i.e., the floor, walls, ceiling, and perceived direction of gravity or the
manipulate a variety of 3D stimuli under controlled conditions. local “scene-based” spatial environment where the target object’s
An immersive virtual environment involves computer simula- components are encoded allocentrically in relation to another
tion of a 3D space and a human computer-interaction within that object, i.e., table-top, blackboard, computer screen, etc. In contrast
space (Cockayne and Darken, 2004). There are two major charac- to environmental frames of reference, the viewer-centered frame is
teristics of 3DI environments that distinguish them from non- egocentric, that is, it defines object configurations and orientations
immersive 2D non-immersive (2DNI) and 3D non-immersive relative to the viewer’s gaze and it includes an embedded retinal
(3DNI) environments. First, 3DI involves egocentric navigation coordinate system. In the case of imagined spatial transformations
(the user is surrounded by the environment) rather than exocen- such as mental rotation (MR), the prevailing hypothesis is that
tric navigation where the user is outside the environment, looking individuals rely more upon an environmental, scene-based, rather

Kozhevnikov and Dhond Understanding immersivity
than a viewer-centered frame of reference (Corballis et al., 1976,

1978; Rock, 1986; Hinton and Parsons, 1988; Palmer, 1989; Pani
and Dupree, 1994). For example, Corballis et al. tested normal-
mirror discriminations of rotated alphanumeric characters when
participants’ heads or bodies were either aligned with the gravita-
tional vertical or misaligned by up to 60˚. The results showed that
the participants made their judgments by rotating the characters
to the gravitational vertical (Y axis) rather than using a viewer-
centered (head-centered or retina-centered) reference frame. Fur-
thermore, Hinton and Parsons (1988) reported that while mentally
FIGURE 1 | Example of the MR test trial stimulus with (A) showing the
rotating two shapes positioned on a table into congruence, partic- subject’s view of the stimulus and (B) showing the three principle axes
ipants often rotated one shape until it had the same relationship of rotation, X, Y, and Z, used in the current study. Note that this
to the table-top (and room) as the other shape (thus achieving coordinate frame differs from the one normally used in computer graphics
scene-based alignment), even though this produced quite different in which the positive Z -direction is perpendicular to the plane of the display,
retinal images. Thus, it appears that the orientation of the viewer pointing toward the viewer.
is defined relative to the scene, rather than the orientation of the

scene being defined relative to the viewer. This lends support for
theories suggesting that the representation of spatial relationships ease of rotating images around different axes support scene-based
is established primarily in terms of scene-based reference systems. encoding, during which the observers rely largely on representa-
Additional evidence for primacy of scene-based reference tions containing more “structural” information (e.g., information
frames comes from experiments (e.g., Parsons, 1987, 1995) com- about spatial relations among the elements of the object and their
paring the speed of MR of classical Shepard and Metzler’s (1971) orientations with respect to the scene in which the objects lie)
3D forms around different axes (see Figure 1A). MR around differ- rather than on retina-based 2D representations of visual-spatial
ent axes places different demands on the transformation processes, images.
and results in different brain activity (Gauthier et al., 2002). Rota- One limitation of previous studies on MR is that they have been
tion in the picture plane preserves the feasibility of all the features conducted using traditional non-immersive environments, where
of a shape, but perturbs the top-bottom relations between fea- the stimuli were presented on a 2D computer screen or another flat
tures. Rotation in depth around the vertical axis alters side-to-side surface (e.g., a table-top), which defines a fixed local frame of ref-
relationships between features and the visibility of features, some erence. This limited and fixed field of view (FOV) may encourage
coming into view and others becoming occluded. Rotation in the use of a more structural scene-based encoding, during which
depth around a horizontal axis is the most demanding rotation; it the parts of the 3D image are encoded in relation to the sides
alters top-bottom relations between features and feature visibility. of the computer screen or another salient object in the environ-
Interestingly, it has been consistently found that participants men- ment. However, because 3DI environments enclose an individual
tally rotate shapes in the depth plane just as fast as or even faster within the scene and allow images to be updated with respect to the
than in the picture plane (Shepard and Metzler, 1971; Parsons, observer’s head orientation, egocentric, viewer-centered encoding
1987, 1995). If participants were in fact rotating viewer-centered may predominate.
2D retina-based visual representations, the depth rotation would The primary goal of the current research was to examine
take longer than rotation in the picture plane since rotation in how individuals process visual-spatial information (specifically
depth would have to carry out additional foreshortening and hid- encode and rotate 3D images) and what spatial frames of ref-
den line removal operations, not required during picture plane erence they rely upon in 3DI virtual environments vs. conven-
rotation. tional non-immersive displays. In our first experiment, in order
Shepard and Metzler (1971) were the first to interpret sim- to control the effect of “three-dimensionality” vs. “immersiv-
ilar slopes for rotation in depth and in the picture plane to ity,” we compared participants’ performance on the Shepard and
indicate that latency was a function of the angle of rotation in Metzler (1971) MR task across the following three types of envi-
three dimensions, not two, as in a retinal projection (for addi- ronments; traditional 2DNI, 3DNI (anaglyphic glasses), and 3DI
tional discussion see Pinker, 1988). In order to investigate this [head mounted display (HMD) with position and head orien-
further, Parsons (1987) conducted an extensive experimental study tation tracking]. In the second experiment, we compared how
examining the rates of imagined rotation not only around three participants encode and transform visual-spatial images in differ-
principal axes of the observer’s reference frame, but also around ent 3DI environments with different backgrounds where shapes
diagonal axes lying within one of the principal planes (frontal, were embedded in a realistic scene vs. in a rectangular frame.
midsagittal, or horizontal) and around “skew” axes not lying in Furthermore, if the neurocognitive correlates of visual-spatial
any of the principal planes. The findings indicated that the rota- imagery are affected by immersivity of visual presentation envi-
tion around different axes, including rotation in depth around ronment, this should be evidenced in the underlying temporal
a horizontal axis perpendicular to the line of sight (Z axis, see dynamic and/or spatial distribution of the electroencephalogram
Figure 1B) were as fast as or even faster than rotations in the (EEG) response. Thus, in the third experiment, EEG was recorded
picture plane (rotations around the axis defined by the line of while participants performed the MR task in 3DI and 2DNI
sight, X -axis in this study). Parsons concluded that this equal environments.
FIGURE 2 | Three different viewing environments (A) 3DI, which includes HMD with position tracking, (B) 3DNI with anaglyphic glasses to present a
stereo picture of three-dimensional spatial forms, and (C) 2D monocular viewing environment.
EXPERIMENT 1 rotation of user’s head was captured by a digital compass mounted

MATERIALS AND METHODS on the back of the HMD.
Participants In the 3DNI environment, the shapes were presented to the par-
Fourteen volunteers (eight males and eight females, average ticipant on a computer screen. Stereoscopic depth was provided
age = 21.5) participated in the study for monetary compensation. by means of anaglyphic glasses (Figure 2B). In the 2DNI envi-
The study was approved by George Mason University (Fairfax, ronment, the shapes were presented for on a standard computer
VA, USA) as well as by The Partners Human Research Commit- screen (Figure 2C).
tee (PHRC, MA, USA) and informed consent was obtained from The retinal image size of the stimuli was kept constant across
all subjects. Participants were asked about their ability to perceive all the environments (computed as ratio of image size over the
stereoscopic images prior to the start of the experiment, and only participant’s distance to the screen). The Vizard Virtual Reality
those who did report difficulty with stereopsis were included. Toolkit v. 3.0 (WorldViz, 2007) was used to create the scenes and
to record the dependent variables (latency and accuracy).
Materials and design Before beginning the MR trials, participants listened to verbal
Each participant completed the MR task – a computerized adap- instructions while viewing example trials in each environment.
tation of Shepard and Metzler’s (1971) task – in three different Eight practice trials were given to ensure participants’ compre-
viewing environments: 3DI, 3DNI, and 2DNI. For each trial, par- hension of the instructions and that they were using a MR strategy
ticipants viewed two spatial figures, one of which was rotated (as opposed to a verbal or analytical strategy). If a response to a
relative to the position of the other (Figure 1A). Participants were practice trial was incorrect, the participants were asked to explain
to imagine rotating one figure to determine whether or not it how they solved the task in order to ensure the use of a rota-
matched the other figure and to indicate whether they thought the tion strategy (i.e., rather than verbal or analytical strategy). In
figures were the same or different by pressing a left (same) or right 3DI, to familiarize the participants with immersive virtual real-
(different) button on a remote control device. Participants were ity, there was also an exploratory phase prior to the practice trials
asked to respond as quickly and as accurately as possible. Twelve in which the participants were given general instructions about
rotation angles were used: 20, 30, 40, 60, 80, 90, 100, 120, 140, 150, virtual reality and the use of the remote control device (about
160, and 180˚. The figures were rotated around three spatial axes: 7–10 min). During the practice and test phases, the participants
line of sight (X ), vertical (Y ), and horizontal (Z ) corresponding remained seated in the chair, but were allowed to move and rotate
to rotations parallel with the frontal (YZ ), horizontal (XZ ), and their head to view 3D Shepard and Metzler shapes. The partici-
midsagittal (XY ) anatomical planes, respectively (Figure 1B). The pants were also given similar time to familiarize themselves with
test included: 12 trial groups for the 12 rotation angles, 3 trial pairs the shapes in the 3DNI and 2DNI environment, and were also
for the 3 axes, and each pair had 1 trial with matching figures and allowed to move and rotate their head to view Shepard and Metzler
1 trial with different figures; thus, there were 72 (12 × 3 × 2) trials shapes.
in total.
In the 3DI virtual environment, the shapes were presented to RESULTS
the participant through an nVisor SX60 (by Nvis, Inc.) HMD Descriptive statistics for performance in the three environments
(Figure 2A). The HMD has a 44˚ horizontal by 34˚ vertical FOV are given in Table 1. Outlier response times (RTs; i.e., RTs ± 2.5
with a display resolution of 1280 × 1024 and under 15% geometric SD from a participant’s mean) were deleted (a total of 2.59% of all
distortion. During the experiment, participants sat on a chair in trials). All simple main effects were examined using the Bonfer-
the center of the room, wearing the HMD to view“virtual”Shepard roni correction procedure. Two participants that performed below
and Metzler images in front of them. Sensors on the HMD enabled chance level were not included in the analysis, thus the final analysis
real-time simulation in which any movement of the subject’s head was performed on 12 participants only.
immediately caused a corresponding change to the image rendered Response accuracy (proportion correct) and RT for correct
in the HMD. The participant’s head position was tracked by four responses were assessed as a function of the rotation axis (X,
cameras located in each corner of the experimental room and sen- Y, and Z ) and environment (3DI, 3DNI, and 2DNI). Data were
sible to an infrared light mounted on the top of the HMD. The analyzed using a 3 (axis) × 3 (environment) repeated measures

ANOVA with a General Linear Model (GLM). The effect of envi- Thus, our central finding is that in 3DI, the RT of rotation
ronment was marginally significant [F (2,22) = 2.9, p = 0.040] and differed between X and Z axes (Z was slower) and that rotation
as pairwise comparison showed, the accuracy in 3DNI and 3DI around the Y axis was faster than Z but not faster than X rotations.
environments was slightly less than in 2DNI (p = 0.08). There was In contrast, RT patterns for 2DNI and 3DNI environments were
a significant main effect of axis [F (2,22) = 19.83, p < 0.001] where similar to those found in previous MR studies (i.e., Y rotations are
Y axis rotations were more accurate than X and Z axis rotations faster than X and Z and X and Z are similar).
(ps < 0.01). The interaction was not significant (F < 1). Overall,
the accuracy level was relatively high for all the environments and
all axes, with the proportion correct ranging from 0.84 to 0.97. Rate of rotation as a function of axis and environment. RT as
Given the high rate of accuracy, indicating that ceiling perfor- a function of rotation angles (i.e., orientation differences between
mance was reached for some rotations, we focused our remaining two Shepard and Metzler shapes) around X, Y, and Z axes for 3DI,
analyses on the RTs. 3DNI, and 2DNI environments respectively are shown in Figure 4.
With respect to RT, there was a significant effect of axis
[F (2,22) = 15.40, p < 0.001] with Y axis rotations being the fastest
(ps < 0.05), see Figure 3. There was no significant effect of envi-
ronment (F < 1), however, there was a significant interaction
between axis and test environment [F (4,44) = 6.45, p < 0.001].
Analysis of simple main effects revealed that, RT for rotation
around the Y axis was significantly faster than either around X (all
ps < 0.05) or Z (all ps < 0.05) for 3DNI and 2DNI environments,
consistently with previous studies (Shepard and Metzler, 1971;
Parsons, 1987, 1995). However, rotations around X and Z axes
were similar (p = 0.98 and 0.79 for 2DNI and 3DNI respectively).
Interestingly the opposite occurred for MR in the 3DI environ-
ment. In 3DI, rotation around Z was significantly longer than X
(p = 0.001) or Y (p = 0.01), while rotations around X and Y were
similar (p = 0.97).
Table 1 | Descriptive statistics for three versions of the MR test in

2DNI, 3DNI, and 3DI.
Test Proportion SD RT (s) SD

correct
2D 0.90 0.07 5.33 1.02

3D non-immersive 0.86 0.10 5.47 1.64
3D immersive 0.87 0.09 5.42 1.46
FIGURE 3 | Response time as a function of axis of rotation and viewing

environment (2DNI, 3DNI, and 3DI). Error bars represent standard error FIGURE 4 | Response time as a function of angle and axis of rotation in
means. (A) 2DNI, (B) 3DNI, and (C) 3DI environments.
The range of rotation angles was from 20 to 160; 180˚ was omit- in which the object is presented. The fact that there was equiv-
ted due to participant’s reports that for this particular angle, they alent performance in 2D and 3DNI environments suggests that
did not rotate shapes mentally, but only scanned two images for depth information per se, which is provided in a 3DNI environ-
mirror-reversed symmetry. ment is insufficient to encourage the use of viewer-centered frame
The slopes of the best-fit linear RT-Rotation Angle functions of reference.
for each axis and each environment (representing rates of rotation
around different axes in different environments) were computed EXPERIMENT 2
and are presented in Table 2. One of possible limitations of Experiment 1 is that, in the 3DI
A repeated measures ANOVA of slopes of best-fit linear regres- environment, the Shepard and Metzler shapes were presented to
sion equations of RT on Rotation Angle show a significant effect participants on a non-realistic “empty” background lacking any
of axis on the slope [F (2,22) = 51.34, p < 0.001] and a sig- points of reference (such as ceilings, walls, other objects), which
nificant interaction between environment and axis of rotation would usually be present in a real scene. Thus, viewer-centered
[F (4,44) = 3.38, p < 0.05], while the effect of environment is not encoding observed in 3DI could be due not to the immersiv-
significant. For 3DI, the rate of rotation around Z was more ity of the environment, but rather due to the lack of any other
than 1.5 times slower than around X (p < 0.05). In both 3DNI objects – except the observers themselves – in relation to which
and 2DNI, the rate of rotation around X and Z did not dif- Shepard and Metzler shapes could have been encoded. In Exper-
fer. Across the environments, the rate of rotation around X was iment 2, the participants in a 3DI condition were presented with
significantly faster for 3DI than for 2DNI (p < 0.05), whereas Shepard and Metzler forms embedded in a realistic scene (city).
the rate of rotation around Z was significantly slower for 3DI In addition, we added a second condition in which the partici-
than for either 3DNI or 2DNI (ps < 0.05). There were no sig- pants viewed Shepard and Metzler forms embedded in a virtual
nificant differences in the rate of rotation around the Y axis rectangular-shaped frame within the 3DI environment. This was
across environment, and the rate of rotation around Y seems done to examine whether the fixed frame around objects in a 3DI
to be the one of the fastest rotations. This is consistent with environment induces scene-based encoding similar to a computer
the findings of previous investigators (Rock and Leaman, 1963; screen in the real world.
Attneave and Olson, 1967; Parsons, 1987; Corballis, 1988) who
argued that rotation around Y, a “gravitational vertical” axis, is MATERIALS AND METHODS
the most common of all rotations in our ecology, so that the fast Participants
rate of rotation around it may result from our extraexperimental Twenty-six volunteers (10 males and 16 females, average age = 20)
familiarity. recruited by advertisement participated in the study for mone-
In summary, the results of Experiment 1 show that the rate tary compensation. The study was approved by George Mason
of MR about the horizontal axis (Z axis) in 3DI (and only 3DI) University (Fairfax, VA, USA) as well as by The Partners Human
was significantly slower than the rate of rotation about the line Research Committee (PHRC, MA, USA) and the informed con-
of sight (X -axis). This finding suggests that in the 3DI environ- sent was obtained from all subjects. Participants who reported
ment the participants were encoding and rotating 2D retina-based difficulty with stereopsis were excluded from participation.
visual representations in relation to a viewer-centered frame of
reference since only then would depth rotation take longer than Materials and design
rotation in the picture plane, due to the involvement of additional Each participant completed the MR task used in Experiment 1
foreshortening and hidden line removal transformations. In con- in three different viewing environments: 3DIC (3DI City) envi-
trast, in 2DNI and 3DNI environments, the rates of MR around ronment, where Shepard and Metzler forms were embedded in a
the X and Z axes were not different, consistent with previous realistic scene of a city (Figure 5A), 3DF (3D Frame), where Shep-
findings for MR using 2D traditional computer displays (Shep- ard and Metzler forms where embedded in a rectangular frame
ard and Metzler, 1971; Parsons, 1987). Thus, in non-immersive (Figure 5B), and a 2DNI environment. The order of environments
environments, participants seem to generate visual representa- was counterbalanced. The experimental procedure was similar to
tions containing more allocentric information such as information that described in Experiment 1.
about spatial relations among the elements of the object and their
orientations with respect to the scene (i.e., the computer screen)
Table 2 | Mean regression slopes of RT-Rotation angle function (s/˚).
Environment Axis of rotation
X Y Z
3DI 0.028 0.014 0.043

FIGURE 5 | Two different viewing environments (A) 3DIC (3DI City)
3DNI 0.029 0.013 0.031
environment, (B) 3DF (3D Frame) environment.
2DNI 0.032 0.016 0.036

RESULTS square-shaped frame that enclosed the stimuli. The square-shaped

Descriptive statistics for proportion correct and RT correct for frame is similar to the sides of the computer screen in non-
2DNI, 3DF, and 3DIC environments are given in Table 3. Outlier immersive environments, suggesting that whenever a person is
RTs (i.e., RTs ± 2.5 SD from a participant’s mean) were deleted (a observing a scene from the “outside” (e.g., a scene is defined by the
total of 3.84% of all trials). One participant that performed below frame or computer screen), it encourages the use of “scene-based”
chance level was not included in the analysis. encoding.
The overall accuracy (proportion correct) ranged from 0.75 to
0.81. For accuracy, a 3 (axis) × 3 (environment) repeated mea- EXPERIMENT 3
sures ANOVA revealed a significant effect of axis [F (2,48) = 5.38, Previous EEG research suggests that MR task involves four sequen-
p < 0.01], where Z axis rotations were significantly less accurate tial cognitive stages which may also differentially modulate frontal
than either Y or X (ps < 0.05). There was a significant effect of and posterior brain (Desrocher et al., 1995). The first stage at
environment F (2,48) = 13.59, p < 0.001]. Participants were sig- ∼200–300 ms post-stimulus is independent of the object’s angular
nificantly less accurate in the 3DIC environment than in either the disparity and involves early sensory processing and simple stimu-
2DNI or 3DIF environments (ps < 0.01), while 2DNI and 3DIF did lus evaluation. Subsequently, at ∼300–400 ms a pre-rotation “set-
not differ among themselves. Also, there was a significant interac- up” stage involves evaluation of object orientation and rotation
tion between axis and environment [F (4,96) = 26.04, p < 0.001]. strategy selection. Third, is the act of MR at ∼400–800 ms post-
Accuracy for rotation around the Y axis was higher than for rota- stimulus which is followed by response selection and execution
tions around X in 2DNI (p < 0.01), but there was no differences from ∼1000 ms onward.
between the accuracy of X and Y rotation in 3DF and 3DIC. Object encoding with respect to a specific frame of reference
Figure 6 presents RT against viewing environment for each occurs prior to the actual process of MR. Thus, the selection of
axis. For RT, there was a significant effect of axis [F (2,48) = 35.10, a frame of reference should begin in the earliest cognitive stages
p < 0.001], where Y axis rotations were significantly faster than between ∼200 and 400 ms post-stimulus. Furthermore, the results
Z (p < 0.01) or X (p = 0.05). There was no significant effect of Experiments 1 and 2 demonstrate that selection of a frame
of environment [F (2,48) = 3.85, p = 0.18]. However, there was of reference is determined by the viewing environment. Thus,
a significant interaction between axis and test environment we hypothesized that when performing a MR task in 2DNI vs.
[F (4,96) = 6.18, p < 0.001]. Examination of simple main effects 3DI, brain response differences should be largest at early sen-
revealed that rotation around the Y axis was faster than around X sory and/or pre-rotation“set-up”stages occurring at ∼200–400 ms
in 2DNI (p < 0.001) and in 3DIF (p = 0.07). In addition, rotation post-stimulus.
around Y was faster than around Z in both 2DNI and 3DF (all
ps < 0.001). However, rotations around X and Z axes were similar
(p = 0.14 and 0.24 for 2DNI and 3DF respectively).
Participants
In contrast, for 3DIC, rotation around Z was significantly
longer than around X or Y (ps < 0.001), while rotations around Eight undergraduate psychology students (four males and four
X and Y were similar (p = 0.78). Thus, our central finding is that females) from the National University of Singapore (age between
in 3DIC the RT of rotation differed between X and Z axes (Z 19 and 25 years) participated in the study for monetary reimburse-
was slower) and that rotation around the Y axis was faster than Z ment. The study was approved by National University of Singapore
but not faster than around X. This pattern is similar to what we committee and informed consent was obtained from all subjects.
reported for the 3DI environment in Experiment 1. In contrast,
RT patterns for 2DNI and 3DF are similar to those found reported
for 2DNI environment in Experiment 1.
In conclusion, the findings of Experiment 2 suggest that indi-
viduals encode and rotate visual-spatial representations in relation
to a viewer-centered frame only in 3DI environments in which
the viewer is enclosed in the scene. Presumably, the participants
used a viewer-centered reference frame because they perceived
themselves to be “inside” the city scene and a scene-based ref-
erence frame when they perceived themselves to be “outside” the
Table 3 | Descriptive statistics for the MR test in 2DNI, 3DF, and 3DIC
(N = 25).
Environment Proportion SD RT correct (s) SD

correct
2DNI 0.81 0.09 5.88 1.76 FIGURE 6 | Response time as a function of axis of rotation and viewing
environment (2DNI, 3DIF, and 3DIC). Error bars represent standard error
3DF 0.76 0.09 6.19 1.87
means.
3DIC 0.75 0.08 6.32 1.76
Participants who reported difficulty with stereopsis were excluded trend for greater negativity prior to ∼250 ms but, larger positivity
from participation. at latencies >350 ms (Figures 8C–E). This effect was signifi-
cant for Z -rotations which were more negative at ∼270–300 ms
Design and analysis post-stimulus for MR task in the 3DI environment.
Electroencephalogram was recorded while subjects completed the The results of Experiment 3 demonstrate that 2DNI and 3DI
MR task in 2DNI and 3DI viewing environments. The order of environments do evoke differential parietal ERP response. The
environments was counterbalanced and, in general, the procedures early latency of these ERP differences further supports our hypoth-
were similar to the first two experiments except as follows. EEG esis that 3DI and 2DNI environments differ at the level of shape
was recorded using a 256-channel HydroCel Geodesic Sensor Net encoding with respect to selection of a frame of reference. Our
(Electrical Geodesics, Inc.). Signals were amplified using the EGI finding of significantly larger ERP differences for 3DI vs. 2DNI
NetAmps 300 amplifier. The signal was sampled at 250 Hz and environment during Z axis rotations is consistent with our behav-
bandpass filtered online at 1.0−100 Hz. For the 3DI condition, the ioral results from Experiments 1 and 2. Specifically, the selection
HMD was placed directly on top of the sensor net (Figure 7). of frame of reference (viewer-centered vs. scene-based) primarily
In order to assure an adequate number of trials for averaging, affects MR in horizontal depth (Z axis rotation).
participants were administered 2 blocks of 72 randomly ordered
trials in the 3DI condition (overall 144 trials), and then another 2 DISCUSSION
blocks of 72 trials in the 2DNI condition. The results of this study suggest that cognitive processing in a 3DI
Data preprocessing and analysis were performed using environment differs from that occurring in 2D and 3DNI envi-
a combination of EEGLAB (Delorme and Makeig, 2004), ronments. Furthermore, only immersive environments seem to
Neuromag software (Elekta, Stockholm) and MNE Software encourage individuals to use egocentric spatial reference frames in
(http://www.martinos.org/mne/). Raw EEG signals were first low- visual encoding and transformation tasks.
pass filtered at 40 Hz to eliminate 50 Hz electrical noise generated In Experiment 1 relative depth information provided by the
by the 3DI headset. Eye blink artifacts were then removed using the 3DNI environment was insufficient to encourage the use of a
Signal Space Projection method provided within the open source viewer-centered frame of reference. In 2DNI and 3DNI environ-
MNE toolbox. The resultant cleaned raw data was used to create ments parity judgments were fastest when objects were rotated
ERP averages from −200 ms pre-stimulus to 800 ms post-stimulus. around the Y axis, while RTs for X and Z axis rotations were sim-
Baseline was defined as −150 to 0 ms. ilar. This suggests that the objects’ components were encoded in
terms of “vertical” and “horizontal” relations with regard to the
RESULTS sides of the computer screen. In particular, rotation in horizontal
All participants demonstrated clear centroparietal responses while depth plane might be relatively easy because it does not alter the
performing the MR task in the 2DNI and also the 3DI environment orientation of the “sides” of the object with respect to the “left” and
(Figure 8). In the 2DNI environment, parietal ERPs were highly “right” sides of the computer screen (see also Parsons, 1987, for a
similar for shapes rotated around the X and Z axes but more nega- similar discussion). This also suggests that cognition in the 2DNI
tive for Y axis rotations from ∼250 ms onward (Figure 8A). In the and 3DNI environments is scene-based and might be atypical of
3DI environment X, Z, and Y rotation demonstrated increasing human interactions with large-scale, real environments.
negativities at ∼350 ms (Figure 8B). When comparing rotations In contrast, our results for a 3DI environment were unique
between 3DI and 2DNI environments, rotations in 3DI showed a and demonstrated that viewers employed an egocentric frame of
reference during MR. Specifically, the rate of rotation in hori-
zontal depth (around Z ) was significantly slower than that in the
picture plane (around X ). At the same time, the rate of rotation
in the picture plane (around X ) was faster in immersive envi-
ronments compared to non-immersive environments, which is
expected for rotation in a plane where no object components are
occluded. Furthermore, when comparing between environments,
rotation in the picture plane (around X ) was the fastest in 3DI
while rate of rotation in horizontal depth (around Z ) was the
slowest in 3DI. This suggests that subjects were in fact rotating a
depth 2D retina-based object representations in 3DI environments
and experiencing difficulties with foreshortening and occlusion.
Furthermore, the results of Experiment 2 demonstrated that
the simple use of 3DI technology is not sufficient to promote the
use of viewer-based frames of reference. When presentation is in
3DI and target objects are embedded within a “fixed frame,” the
observer relies on scene-based encoding. This is similar to the effect
of viewing in 2DNI where the fixed borders are defined by the sides
FIGURE 7 | Electroencephalogram recording in 3DI virtual reality
of a computer screen. In this case, scene-based encoding is most
environment.
efficient because the observer is free to move around without need

FIGURE 8 | Electroencephalograms for Mental Rotation under 2DNI and to X rotations at ∼350 ms. (C–E) When comparing rotations between 2DNI
3DI conditions: Evoked potential (µV) peaks were largest in and 3DI environments 3DI rotations demonstrated a trend for greater
centroparietal electrodes regardless of visual environment. (A) 2DNI negativity prior to 350 ms but slightly larger positivity at longer latencies
ERPs were most negative for Y axis rotations from ∼250 ms onward. (B) 3DI >350 ms. The largest differences were between Z -rotations which were more
ERP’s demonstrated increasing negativities for Z and Y rotation with respect positive for 3DI at ∼270–300 ms, (∗) with paired t -test p < 0.03.
to mentally update the position of every object within the scene. Importantly, our study is the first attempt to understand
In contrast, within 3DI environments (without frame embedding) immersivity from a cognitive neuroscience perspective. Currently,
as well as in the large-scale real world, the positions of objects rela- there is no clear understanding of what “immersivity” means in
tive to an external frame may be constantly changing thus making cognitive terms. Most commonly used terms and definitions are
viewer-centered encoding more efficient. merely descriptive, such as “perceiving oneself to be enveloped by,
Finally, most previous EEG/ERP experiments on mental included in, and interacting with an environment” (Witmer and
imagery have been performed only in 2DNI environments. The Singer, 1998, p. 227), and often confounded with such terms as
results of Experiment 3 demonstrate that 2DNI and 3DI envi- “immersion” and “presence” describing the “extent to which the
ronments do evoke differential parietal ERP responses, and that human operator loses his or her awareness of being present at
ERPs were more negative at ∼270–300 ms post-stimulus for MR the site and instead feels present in the artificial environment”
in the 3DI vs. 2DNI environment. One interpretation is that this (Durlach and Mavor, 1995, p. 22). Similarly, one problem in
early modulation of ERP activity marks viewer-centered vs. scene- research aimed at understanding cognitive processing differences
based orienting in preparation for subsequent MR from 400 ms between virtual and real environments, is that they do not clearly
onward. However, this early modulation may in addition involve define “immersivity.” For example, Perani et al. (2001) reported
other processes such as spatial attention and simple shape evalua- that the right inferior parietal cortex (IPC) was activated when
tion. Thus, future ERP research should evaluate the contribution subjects observed real hand actions but not when observing hand
of these component processes which may affect early stages of MR movements in a 3D virtual environment or on a 2D display. Perani
task performance in different visual environments. Also as Experi- et al. suggested that within virtual environments, the right parietal
ment 3 was a preliminary study, much work remains to be done to system may not provide sufficient information for the computa-
map the neural markers of brain processing differences between tion of the spatial reference frame as it is used in the real world.
2DNI vs. 3DI environments. This may include the use of structural However, what Perani et al. called an immersive environment was
MRI for anatomical localization of ERPs as well as the use of more the condition in which participants laid in a PET scanner with
ecologically valid task designs. their heads position fixed, and black curtains were arranged so
that the participants were able to focus only on what was shown differences in encoding and transformation processes in immer-
behind a single rectangular opening in front of them. We should sive vs. non-immersive environments. In fact, the findings of this
note that this type of environment does not meet what we con- research explain the results of previous studies that show no trans-
sider to be the essential requirement of “immersivity,” namely fer from training in 2D non-environments to immersive virtual
viewing the scene from the “inside.” We believe that an immer- environments. For instance, Pausch et al. (1997) reported that
sive 3D environment, in which a viewer is surrounded by the immersive prior practice with conventional 2D displays in visual
environment and no clear borders are present is necessary to search tasks impaired performance in immersive virtual environ-
encourage the use of an egocentric (viewer-centered) frame of ments. The researchers suggested that using desktop graphics to
reference. train users for real world search tasks may not be efficient. The
Our findings have implications for future studies on spatial current study explains this finding by pointing out that the encod-
transformations of mental images and the design of testing envi- ing of spatial relations and cognitive strategies applied to perform
ronments. They show that the results of the previous experiments visual/spatial transformations in these two types of environments
on MR, performed in laboratory conditions using a traditional are different. We suggest that 3DI environments with a variety of
2D computer screen, might be limited in that they may not reflect simulated 3D stimuli will provide the most efficient environment
the MR patterns that would be measured in a natural, 3D environ- for training egocentric visual-spatial skills that will generalize and
ment. In addition to its theoretical implications, this research could transfer to real world tasks.
be of considerable interest from an applied perspective; specifically
for the design of training and learning environments. Although 3D ACKNOWLEDGMENTS
environments might be more attractive to the user, the results of This research was supported by the National University of Singa-
the current research show that there will probably be no signifi- pore (Singapore) and US Office of Naval Research (N00014-10-1-
cant differences between encoding and spatial transformation of 0638, Kozhevnikov) and the National Institutes of Health NCCAM
images under 2DNI and 3DNI conditions. On the other hand, a (K01AT004481, Dhond). We thank Olesya Blazhenkova and David
3DI environment can provide a unique and possibly more realis- Schloerb for fruitful discussion. We thank Jodie Royan for her help
tic learning environment. In particular, a 3DI environment should in conducting Experiment 1, and Debbie Morton in conducting
provide advantage to those tasks that benefit from encoding from Experiment 3. We also thank (in alphabetical order): Andrey Gor-
an egocentric frame of reference (e.g., navigation, wayfinding, bunov and Remi Tache for programming 3D virtual reality testing
laparoscopic surgery, and telerobotics). In general, using desktop environments; Sheraz Khan and Emily Ruzich for assisting with
graphics to train users for real world egocentric spatial tasks might EEG data analysis; as well as Klaus Gramann and Florian Friemel
not be effective, and may actually be counterproductive due to the for EEG data analysis advice.
REFERENCES for analysis of single-trial EEG virtual environments,” in Collabo- Parsons, L. M. (1995). Inability to rea-
Attneave, R., and Olson, R. K. (1967). dynamics. J. Neurosci. Methods 134, rative Design in Virtual Environ- son about an object’s orientation
Discriminability of stimuli varying 9–21. ments (Intelligent Systems, Control using an axis and angle of rotation.
in physical and retinal orientation. J. Desrocher, M. E., Smith, M. L., and Tay- and Automation: Science and Engi- J. Exp. Psychol. Hum. Percept. 21,
Exp. Psychol. 74, 149–157. lor, M. J. (1995). Stimulus and sex neering) Vol. 48, eds X. Wang and 1259–1277.
Chance, S., Gaunet, F., Beall, A., differences in performance of men- J. Jen-Hung (Dordrecht: Springer), Pausch, R., Proffitt, D., and Williams,
and Loomis, J. (1998). Locomotion tal rotation: evidence from event- 17–26. G. (1997). “Quantifying immer-
mode affects the updating of objects related potentials. Brain Cogn. 28, Loomis, J., Blascovich, J., and Beall, A. sion in virtual reality,” in Proceed-
encountered during travel: the con- 14–38. (1999). Immersive virtual environ- ings of ACM SIGGRAPH ‘97 Con-
tribution of vestibular and propri- Durlach, N. I., and Mavor, A. S. ment technology as a basic research ference on Computer Graphics, Los
oceptive inputs to path integration. (1995). Virtual Reality: Scientific tool in psychology. Behav. Res. Meth- Angeles.
Presence (Camb.) 7, 168–178. and Technological Challenges. Wash- ods Instrum. Comput. 31, 557–564. Perani, D., Fazio, F., Borghese, N. A.,
Cockayne, W., and Darken, R. (2004). ington, DC: National Academy Macuga, K. L., Beall, A. C., Kelly, J. Tettamanti, M., Ferrari, S., Decety,
“The application of human abil- Press. W., Smith, R. S., and Loomis, J. J., and Gilardi, M. C. (2001). Dif-
ity requirements to virtual environ- Gauthier, I., Hayward, W. G., Tarr, M. (2007). Changing lanes: iner- ferent brain correlates for watch-
ment interface design and evalua- M. J., Anderson, A. W., Skud- tial cues and explicit path informa- ing real and virtual hand actions.
tion,”in The Handbook of Task Analy- larski, P., and Gore, J. C. (2002). tion facilitate steering performance Neuroimage 14, 749–758.
sis for Human-Computer Interaction, BOLD activity during mental when visual feedback is removed. Pinker, S. (1988). “A computational the-
eds D. Diaper and N. Stanton (Mah- rotation and viewpoint-dependent Exp. Brain Res. 178, 141–150. ory of the mental imagery medium,”
wah: Lawrence Erlbaum Associates), object recognition. Neuron 34, Palmer, S. E. (1989). “Reference frames in Cognitive and Neuropsychological
401–421. 161–171. in the perception of shape and orien- Approaches to Mental Imagery, eds
Corballis, M. C. (1988). Recognition of Hinton, G. E., and Parsons, L. M. tation,” in Object Perception: Struc- M. Denis, J. Engelkamp, and J. T. E.
disoriented shapes. Psychol. Rev. 95, (1988). Scene-based and viewer- ture and Process, eds B. Shepp and S. Richardson (Amsterdam: Martinus
115–123. centered representations for com- Ballesteros (Hillsdale, NJ: Lawrence Nijhoff), 17–36.
Corballis, M. C., Anuza, T., and Blake, paring shapes. Cognition 30, 1–35. Erlbaum Associates), 121–163. Rock, I. (1986). “The description and
L. (1978). Tachistoscopic perception Klatzky, R., Loomis, J., Beall, A., Chance, Pani, J. R., and Dupree, D. (1994). Spa- analysis of object and event per-
under head tilt. Percept. Psychophys. S., and Golledge, R. (1998). Spatial tial reference systems in the compre- ception,” in Handbook of Perception
24, 274–284. updating of self-position and ori- hension of rotational motion. Per- and Human Performance, Chap. 33,
Corballis, M. C., Zbrodoff, J., and entation during real, imagined, and ception 23, 929–946. eds K. Boff, L. Kaufman, and J.
Roldan, C. E. (1976). What’s up in virtual locomotion. Psychol. Sci. 9, Parsons, L. M. (1987). Visual discrim- Thomas (New York: John Wiley),
mental rotation? Percept. Psychophys. 293–298. ination of abstract mirror-reflected 1–71.
19, 525–530. Kozhevnikov, M., and Garcia, A. (2011). three-dimensional objects at many Rock, I., and Leaman, R. (1963). An
Delorme, A., and Makeig, S. (2004). “Visual-spatial learning and train- orientations. Percept. Psychophys. 42, experimental analysis of visual sym-
EEGLAB: an open source toolbox ing in collaborative design in 49–59. metry. Acta Psychol. 21, 171–183.

Shepard, R. N., and Metzler, J. questionnaire. Presence (Camb.) 7, solely the responsibility of the authors This article was submitted to Frontiers in
(1971). Mental rotation of three- 225–240. and does not necessarily represent the Perception Science, a specialty of Frontiers
dimensional objects. Science 171, WorldViz. (2007). Vizard VR Toolkit R3. official views of our funding agencies. in Psychology.
701–703. Available at: http://www.worldviz. Copyright © 2012 Kozhevnikov and
Tarr, M. J., and Warren, W. H. com/products/vizard4/index.html Received: 12 April 2012; accepted: 23 July Dhond. This is an open-access article dis-
(2002). Virtual reality in 2012; published online: 10 August 2012. tributed under the terms of the Creative
behavioral neuroscience and Conflict of Interest Statement: The Citation: Kozhevnikov M and Dhond Commons Attribution License, which
beyond. Nat. Neurosci. 5(Suppl.), research was conducted in the absence RP (2012) Understanding immersiv- permits use, distribution and reproduc-
1089–1092. of any commercial or financial rela- ity: image generation and transformation in other forums, provided the original
Witmer, B. G., and Singer, M. J. tionships that could be construed as tion processes in 3D immersive envi- authors and source are credited and sub-
(1998). Measuring presence in a potential conflict of interest. Fur- ronments. Front. Psychology 3:284. doi: ject to any copyright notices concerning
virtual environments: a presence thermore, the content of this article is 10.3389/fpsyg.2012.00284 any third-party graphics etc.
doi: 10.3389/fpsyg.2012.00315
Abacus in the brain: a longitudinal functional MRI study of

a skilled abacus user with a right hemispheric lesion
Satoshi Tanaka 1 *, Keiko Seki 2 , Takashi Hanakawa 3 , Madoka Harada 4 , Sho K. Sugawara 5,6 , Norihiro
Sadato 5,6 , Katsumi Watanabe 7 and Manabu Honda 3
1
Center for Fostering Young and Innovative Researchers, Nagoya Institute of Technology, Nagoya, Japan
2
Graduate School of Health Sciences, Kobe University, Kobe, Japan
3
Department of Functional Brain Research, National Institute of Neuroscience, Kodaira, Japan
4
Department of Rehabilitation Medicine, Eisei Hospital, Hachioji, Japan
5
Division of Cerebral Integration, National Institute for Physiological Sciences, Okazaki, Japan
6
School of Life Sciences, The Graduate University for Advanced Studies (SOKENDAI), Hayama, Japan
7
Research Center for Advanced Technology and Science, The University of Tokyo, Komaba, Japan
Edited by: The abacus, a traditional physical calculation device, is still widely used in Asian coun-
Joel Pearson, The University of
tries. Previous behavioral work has shown that skilled abacus users perform rapid and
precise mental arithmetic by manipulating a mental representation of an abacus, which is
Reviewed by:
David Barner, University of California based on visual imagery. However, its neurophysiological basis remains unclear. Here, we
San Diego, USA report the case of a patient who was a good abacus user, but transiently lost her “mental
Feiyan Chen, Zhejiang University, abacus” and superior arithmetic performance after a stroke owing to a right hemispheric
China
lesion including the dorsal premotor cortex (PMd) and inferior parietal lobule (IPL). Func-
*Correspondence:
tional magnetic resonance imaging experiments were conducted 6 and 13 months after
Satoshi Tanaka, Center for Fostering
Young and Innovative Researchers, her stroke. In the mental calculation task, her brain activity was shifted from the language-
Nagoya Institute of Technology, related areas, including Broca’s area and the left dorsolateral prefrontal and IPLs, to the
Gokiso-cho Showa-Ku, Nagoya visuospatial-related brain areas including the left superior parietal lobule (SPL), according
466-8555, Japan.
to the recovery of her arithmetic abilities. In the digit memory task, activities in the bilateral
e-mail: tanaka.satoshi@nitech.ac.jp
SPL, and right visual association cortex were also observed after recovery.The shift of brain
activities was consistent with her subjective report that she was able to shift the calcula-
tion strategy from linguistic to visuospatial as her mental abacus became stable again. In a
behavioral experiment using an interference paradigm, a visual presentation of an abacus
picture, but not a human face picture, interfered with the performance of her digit memory,
confirming her use of the mental abacus after recovery. This is the first case report on the
impairment of the mental abacus by a brain lesion and on recovery-related brain activity.
We named this rare case “abacus-based acalculia.” Together with previous neuroimaging
studies, the present result suggests an important role for the PMd and parietal cortex in
the superior arithmetic ability of abacus users.
Keywords: acalculia, arithmetic, calculation, expertise, imagery, memory, plasticity, stroke
INTRODUCTION 2008; Frank and Barner, 2012). These works have demonstrated
To perform complex calculations, most people rely on physical examples of the role of mental imagery in mental arithmetic
devices such as pencil and paper, mechanical calculators, and operations.
more recently digital computers. One such device is an abacus, Several behavioral and neuroimaging studies have attempted to
which is still widely used in Asian countries. The abacus is a sim- examine the neural correlates of the calculation strategy employed
ple device of beads and rods, and numbers are represented by by abacus users (Hatta and Ikeda, 1988; Tanaka et al., 2002, 2008;
the spatial locations of beads (Figure 1). Skilled abacus users can Hanakawa et al., 2003; Chen et al., 2006; Wu et al., 2009; Hu et al.,
calculate accurate answers to mathematical problems extremely 2011; Ku et al., 2012). For example, recent neuroimaging studies
rapidly. Interestingly, however, abacus users not only manipulate have reported activation in the bilateral dorsal premotor cortex
the tool skillfully in its physical form but also gain the ability (PMd) and inferior and superior parietal lobule (IPL and SPL,
to mentally calculate extraordinarily large numbers, often more respectively) during mental calculation and digit memory tasks
than 10 digits at the expert level, with unusual speed and accu- in abacus users (Tanaka et al., 2002; Hanakawa et al., 2003; Chen
racy (Hatano et al., 1977). Psychological studies have shown that a et al., 2006; Wu et al., 2009; Ku et al., 2012). However, there have
non-linguistic strategy using visual imagery of the abacus (a“men- been no neuropsychological studies that report deficits in mental
tal abacus”) underlies this unusual calculation ability (Hatano abacus ability after focal brain injury. Therefore, the causal rela-
et al., 1977, 1987; Hatano and Osawa, 1983; Stigler, 1984; Hatta tionship between mental abacus ability and region-specific brain
et al., 1989; Hishitani, 1990; Hanakawa et al., 2004; Tanaka et al., structures remains unclear.

Tanaka et al. Loss of brain abacus
without the damaged node, possibly because of functional reorga-

A 1 2 3 4 5 6 B 3 0 7 9 2 8 nization within the remote intact nodes (Frost et al., 2003; Fridman
et al., 2004; Dancause et al., 2006). Thus, we hypothesized that the
intact PMd, IPL, and/or SPL would be active with the recovery of
mental abacus ability.
In the present study, functional MRI experiments were con-
ducted 6 and 13 months after her stroke and brain activity between
the two sessions was compared in order to test this hypothesis.
In addition, a behavioral experiment using dual-task interfer-
FIGURE 1 | Illustrations of an abacus. An abacus is a rectangular wooden
ence paradigms was conducted to confirm her use of the mental
calculator based on the decimal system. Each vertical rod has five sliding imagery of an abacus on a digit memory task 13 months after her
beads, one above and four below a middle horizontal bar. Numbers are stroke.
represented by the configurations of the beads. A bead above the bar is
equal to five when it is pushed down, and each of the four beads below is
equal to one when pushed up. For example, the left figure (A) represents
123,456 and the right figure (B) represents 307,928. CASE REPORT
The patient was a 57-year old left handed female. She had worked
as a professor in a national university before the stroke. She had a
Here, we report the case of a patient who was a well-experienced Ph.D. degree in medicine and had worked as a scientist in the field
abacus user but had impaired mental arithmetic performance of neuropsychology for more than 25 years. She had published
based on her mental abacus strategy due to a stroke. Her knowl- more than 20 international peer-reviewed papers. She had also
edge of basic arithmetic facts and her knowledge and operation of engaged in rehabilitative medicine as a speech-language-hearing
a physical abacus were intact. Only performance in mental calcu- therapist for more than 25 years.
lation and digit memory tasks based on the mental abacus strategy She started her abacus training at an abacus school when she
was transiently impaired after the lesion. When we met her for the was an elementary-school child, and had trained in physical and
first time, she said “I lost my abacus in the brain.” mental abacus operation for 3 years. We speculated that she was an
The first purpose of the present study was to localize the lesion excellent and skilled abacus user owing to the fact that she became a
areas using high-resolution structural magnetic resonance imag- finalist at a domestic abacus competition in Japan in two successive
ing (MRI) with a 3T MRI scanner. We hypothesized that the years, although her training period was relatively shorter compared
lesion areas should include the PMd and/or parietal regions that with the grand experts who participated in our previous functional
were dominantly activated during the mental calculation and digit MRI studies (Tanaka et al., 2002; Hanakawa et al., 2003). After she
memory tasks in the previous functional MRI studies of abacus finished her abacus training, she kept using abacus-based mental
users (Tanaka et al., 2002; Hanakawa et al., 2003; Chen et al., 2006; calculation and mnemonic strategies in everyday activities for a
Ku et al., 2012). long period and did not lose her ability. In fact, she reported that
The second purpose of the present study was to examine the her forward digit span was around 12 before the stroke episode.
changes of brain activity with the recovery of mental abacus ability. This was far beyond the average score for her age group.
Several neuroimaging studies have reported changes of brain activ- In July 2009, she suffered from a right hemispheric infarct in
ities with recovery from motor, attentional, or language deficits the territory of the anterior and middle cerebral arteries. When a
after stroke (Ward et al., 2003; Fridman et al., 2004; Corbetta et al., therapist tested her digit span during a clinical neuropsychological
2005; Price and Crinion, 2005; Heiss and Thiel, 2006). However, evaluation in a hospital approximately 2 months after her stroke,
recovery-related changes in brain activity from deficits in arith- she noticed that she was not able to use the mental abacus strategy
metic ability, especially in the non-linguistic aspects of arithmetic for the digit span test. She was not able to generate vivid mental
operation, remain totally unknown. imagery of an abacus and the image of the abacus was very frag-
We hypothesized that the patient would change her strategy for ile. Detailed structural MRI scans were obtained in January 2010.
mental calculation and digit memory from verbal to visuospatial Functional MRI scans were conducted at two different periods, the
with stroke recovery. Therefore, her brain activity during mental first in January 2010 and the second in August 2010.
calculation would shift from language-related to visuospatial-
related brain regions after recovery. As mentioned above, previous NEUROPSYCHOLOGICAL EVALUATION
imaging studies have revealed dominant activation in the bilateral Neuropsychological evaluations were conducted approximately
PMd, IPL, and SPL during mental calculation in abacus experts 1 month after stroke onset. Her score on Raven’s Standard Pro-
(Tanaka et al., 2002; Hanakawa et al., 2003). Neuroanatomical gressive Matrices was in the average range (33/36). Similarly, her
studies have shown that the PMd and parietal cortex have dense IQ measured by Kohs Block Design Test was also in the aver-
neuroanatomical connections (Wise et al., 1997; Luppino et al., age range (108). The Standard Language Test for Aphasia (SLTA;
1999; Wise and Murray, 2000). Thus, the PMd, IPL, and SPL may Hasegawa et al., 1984), which has been widely used in Japan, did
work as a functional network during abacus-based mental calcu- not detect any impairments of language. However, clinical obser-
lation. Damage in one node may induce transient impairment of vation detected mild impairments of her speech production: her
mental abacus ability. However, it is possible that the other intact prosody was impaired and speed of speech was slow with small
nodes in the functional network could gain the ability to work volume. Clinical observation immediately after her stroke detected
unilateral visual neglect. For motor function, the patient showed

a severe paralysis in the left upper limb and mild paralysis in the 10
left lower limb.
9
ARITHMETIC ABILITY
After her stroke onset, her arithmetic ability was not impaired 8
according to the neuropsychological evaluation. She was able to
perform four basic arithmetic operations without any problem. In
Span
7
fact, she was able to answer all arithmetic problems correctly in
the SLTA. In addition, her long-term memory of digits was also
intact because she correctly remembered the numbers of her bank 6
accounts and airplane mileage accounts. However, she noticed that
she was not able to generate visual imagery of a mental abacus, 5
which had been easily generated before the stroke, when a neu-
ropsychologist tested her maximum digit span 2 months after her 4
stroke. Before the stroke, she used to use the mental abacus strategy 1 2 3 4 5 6 7 8 9 10 11 12 13
especially when she calculated and memorized larger sequences of Time after stroke onset (month)
digits, because the visuospatial strategy, rather than a phonological
strategy, was useful in coding a larger number of digits (Hatano
et al., 1977; Hatano and Osawa, 1983). Due to the impairment of
visual imagery after her stroke, she used the phonological strategy
instead. She was able to perform four basic arithmetic operations FIGURE 2 | Behavioral performance of maximum digit and alphabet
correctly although she felt that her arithmetic ability had declined span tasks. Maximum digit spans were higher compared with alphabet
after her stroke. span, indicating that her superior performance was digit memory-specific.
Her maximum forward digit span (white circle), as well as forward and
Six months after her stroke, just before the first functional MRI backward alphabet spans (white and black triangles), was unchanged
session, we evaluated her knowledge of basic arithmetic facts, as across the entire experimental period. However, the maximum backward
well as her knowledge, and operation of a physical abacus. These digit span (black circle) was improved over time after her stroke.
aspects were all intact. However, she still felt that it was difficult to
generate a vivid visual image of a mental abacus. She reported that
she was not able to perform mental calculations and memorize The first and second functional MRI sessions were conducted
digit sequences based on the mental abacus strategy because her 6 months (January 2010) and 13 months (August 2010) after her
mental abacus was fragile. However, 13 months after her stroke, stroke onset, respectively. The difference of the brain activities
she reported that her capacity for visual imagery of a mental aba- between the two sessions was compared. Structural MRI scans
cus had recovered. At that time, she participated in the second were obtained in January 2010. In addition, the patient partici-
functional MRI session. pated in a behavioral experiment after the second functional MRI
Figure 2 shows her behavioral performance of maximum digit session in order to examine whether the patient would use abacus-
and alphabet span tasks. Forward digit span and forward and back- based mental calculation and digit memory strategies in these tasks
ward alphabet spans were all unchanged across the experimental (Experiment 2).
period. In contrast, backward digit span improved over time after
her stroke. Her backward digit span 13 months after her stroke EXPERIMENT 1
was eight and almost equal to her forward digit span. It has been Behavioral task in functional MRI experiment
reported that abacus experts reproduce a series of digits in back- For the functional MRI experiment, the patient performed mental
ward order almost as well as in the forward order, because both calculation and digit memory tasks that were used in our previ-
require experts to read off the digits from visuospatial mental rep- ous functional MRI studies of abacus experts (Tanaka et al., 2002;
resentation of an abacus (Hatano and Osawa, 1983). Therefore, Hanakawa et al., 2003). Before the functional MRI experiment, she
nearly identical maximum digit spans both backward and forward practiced these tasks outside the scanner to become familiar with
might be interpreted as evidence that she used her mental abacus the tasks. Presentation software (Neurobehavioral Systems Inc.,
13 months after her stroke. In fact, she reported that she was able Albany, CA, USA) was used for the visual stimulus presentation
to use the mental abacus strategy for the backward digit span task and to record her responses. Stimuli were presented on a screen
13 months after her stroke. using a liquid crystal display projector, and she viewed the screen
though a mirror.
EXPERIMENTAL PROCEDURE For the mental calculation task, white digit stimuli were pre-
The patient gave written, informed consent before the experi- sented for 1.5 s with inter-stimulus intervals of 2 s on the center
ments, which were approved by the local ethics committee of the of a screen (Figure 3A, Hanakawa et al., 2003). Digit stimuli were
National Institute for Neuroscience. presented 10 times during each trial. The patient was asked to men-
The patient participated in two functional MRI sessions of tally add the presented series of digits without moving her fingers.
the mental calculation and digit memory tasks (Experiment 1). After the presentation of these digit stimuli, a red digit stimulus

Imaging data acquisition and analysis

A The functional MRI experiment was conducted using a 3.0-T
MRI scanner (MAGNETOM Trio, Siemens, Erlangen, Germany).
45
16 ....... 13 .......... 675? Functional images were acquired using a T2∗ -weighted echo
planar imaging sequence (TR/TE/FA/FOV/voxel size/slice num-
ber = 3000 ms/30 ms/90˚/192 mm/3.0 mm × 3.0 mm × 3.0 mm/46
axial slices for the mental calculation task, and 2000 ms/40 ms/80˚/
192 mm/3.0 mm × 3.0 mm × 4.0 mm/25 axial for the digit mem-
B ory task). A total of 143 and 205 functional images on each mental
calculation and digit memory task were collected during each ses-
+ ................... 5372693? sion. The first three and five images of each task were discarded
from data analysis to allow for the stabilization of the magneti-
zation. Eighty-three images were obtained on each verbal fluency
and hand grip task and the first three images were discarded. A
high-resolution structural T1 image was acquired using a Magne-
tization Prepared Rapid Acquisition in Gradient Echo (MPRAGE)
FIGURE 3 | (A) Schematic illustration of the mental calculation task. The
sequence.
patient was asked to add a series of numbers mentally that were visually
presented on the computer screen. (B) Schematic illustration of the digit SPM8 software (Wellcome Department of Cognitive Neurol-
memory task. She was asked to retain the target sequence of digits during ogy, London, UK) was used for image processing and analysis.
the delay period. The T1 image was spatially normalized to fit a Montreal Neuro-
logical Institute (MNI) template (Evans et al., 1993). The damaged
regions were masked to reduce the influence from non-brain or
was presented for 3 s. She was asked to judge whether the addition lesioned tissue (Brett et al., 2001). For functional images, the data
answer in her mind and the test digit stimuli were the same or were first realigned to the mean functional images in order to
different, by pressing one of the response buttons with the right reduce the effect of head motion. These images were then normal-
fingers. After each trial, there was an 18-s inter-trial interval (ITI) ized to the MNI template, with the same parameter obtained for
in which the patient simply watched the white fixation cross pre- T1 normalization. Then, the images were spatially smoothed using
sented at the center of the screen (visual fixation condition). She an isotropic Gaussian kernel of 6-mm full-width half maximum
performed additional tasks with single-digit and two-digit num- (FWHM).
bers. The experimental session consisted of five trials for each task
in an alternate order. Statistical analysis
For the digit memory task, a delayed match-to-sample task Statistical analysis of the time course data at each voxel was con-
using a digit sequence as the stimulus was employed (Figure 3B, ducted with a general linear model in order to identify voxels that
Tanaka et al., 2002). A target digit sequence was presented on showed task-specific and session-specific signal changes (Friston
the center of a screen for 3 s. The length of the digit sequence et al., 1994). The brain activities in the mental calculation and digit
was a five digit number, which was two digits shorter than her memory tasks were analyzed separately.
digit span memory capacity measured before the first functional For the mental calculation task, one-digit and two-digit calcu-
MRI session. After a 15-s delay period, during which only a fix- lation tasks were separately modeled as regressors on each session
ation cross appeared on the screen, a test sequence of digits was with boxcar functions convolved with a hemodynamic response
presented for 3 s. She was asked to judge whether the target and function. For the digit memory task, the presentations of the target
test sequences were the same or different, by pressing one of the and test sequences, and the delay period, were separately modeled
response buttons. Following these behavioral events, there was on each session using three boxcar functions convolved with a
a 17-s visual fixation. The experimental session consisted of 10 hemodynamic response function. For the verbal fluency and hand
trials. grip task, the task period was modeled using three boxcar func-
The patient also participated in functional MRI experiments of tions convolved with a hemodynamic response function. In all
verbal fluency and hand grip tasks 13 months after the stroke. tasks, head-movement parameters were also included as regressors
These experiments were conducted to ascertain whether the of no interest.
region-specific brain activity during arithmetic tasks 13 months To test hypotheses about regionally specific task-effects or
after the stroke would be task-specific or not. In the verbal fluency session-effects, the estimates for each model parameter were com-
task, the subject was asked to generate in her mind as many words pared with the linear contrasts. The resulting set of voxel values
as possible from an indicated category (such as names of sports or constituted a statistical parametric map of the t statistic, SPM{t }.
fruits) during a 24-s trial. After each trial, there was a 24-s visual In all tasks, the statistical threshold was set at p < 0.001 at the voxel
fixation condition. The task and fixation condition were alternately level. Control for multiple comparisons was achieved at the clus-
performed 10 times. In the hand grip task, the patient was asked to ter level with Gaussian random field theory either in the whole
make the hand grip movement with her paretic hand every 2 s dur- brain (p corr < 0.05) or the small volume around the coordi-
ing a 24-s period. The hand grip task and visual fixation condition nates of the regions of interest (ROIs) based on the published
were alternately performed 10 times. papers (p svc < 0.05). On the basis of previous works on abacus
experts (Tanaka et al., 2002; Hanakawa et al., 2003), spherical ROIs EXPERIMENT 1: FUNCTIONAL MRI EXPERIMENT
(r = 8 mm) were created at the peak voxel in the bilateral SPL Mental calculation task
(left x = −18, y = −66, z = 60; right x = 14, y = −66, z = 64 at The patient responded correctly in all trials of the calculation
MNI coordinate), left IPL (x = −46, y = −40, z = 54), left PMd tasks in both functional MRI sessions. Figure 5A shows brain
(x = −32, y = −6, z = 52), and Broca’s area (x = −50, y = 10, activity associated with one- and two-digit mental calculation
z = 26). tasks relative to the visual fixation condition (see Table A1 in
Appendix online). In one-digit mental calculations, brain activ-
EXPERIMENT 2 ity was generally lateralized to the left hemisphere both 6 and
Behavioral evaluation in mental abacus use 13 months after her stroke. In contrast, brain activity in two-
A behavioral experiment using interference paradigms was con- digit mental calculations was observed bilaterally both 6 and
ducted to examine whether the patient would utilize the mental 13 months after her stroke. These brain regions include the mid-
abacus strategy on a digit memory task 13 months after her stroke dle frontal gyrus, pre- and postcentral gyrus, SPL, middle and
(Figure 10A). The behavioral paradigm was based on Hatta et al. superior occipital gyrus, inferior temporal gyrus, and cerebellum.
(1989). She performed a delayed digit recall task. First, a tar- This activity was not observed in the damaged regions of the
get digit sequence was presented on the computer screen for right hemisphere. When brain activities during one- and two-digit
3 s. The length of the target digit sequence was eight, which was mental calculation tasks were directly compared, significant brain
one-digit shorter than her maximum digit span memory capac- activity in the left middle frontal gyrus was observed 6 months
ity. After a 15-s retention interval, she was asked to recall and after her stroke (Figure 5B). In contrast, significant activity was
report the digit sequence orally. There were three experimen- observed in the bilateral SPL, right middle frontal gyrus, post-
tal conditions which differed according to the types of visual central gyrus, and middle occipital gyrus 13 months after her
distractors. Pictures of abacus figures, human faces, or gray rec- stroke.
tangles were presented on the center of the screen during the To investigate the time-specific brain activities, her whole brain
retention interval. Each distractor stimulus was presented for 1 s activities between 6 and 13 months after her stroke were directly
with 0.5 s inter-stimulus intervals. She performed 15 trials for compared. A previous study has revealed that the region-specific
each distractor condition. We hypothesized that if she utilized brain activities in abacus users were more evident in the mental
a mental abacus for the digit memory task, the presentation of calculation task with a higher cognitive demand (Hanakawa et al.,
the pictures of abacus figures would interfere with task perfor- 2003). Therefore, the brain activities in the two-digit addition task
mance more than the presentation of the human faces and gray between 6 and 13 months after her stroke were compared in the
rectangles. analysis.
The results are shown in Figure 6. The left hemispheric cor-
RESULTS tical activities including Broca’s area (peak coordinate x = −48,
STRUCTURAL MRI y = 8, z = 8; t = 4.73, cluster size = 227 voxels, p corr < 0.05),
The T1-weighted MRI showed a right fronto-parietal lesion, the left dorsolateral prefrontal cortex (DLPFC, x = −48, y = 38,
involving the posterior parts of the inferior and superior frontal z = 30; t = 4.81, cluster size = 118 voxels, p corr < 0.05), and
gyrus, anterior insula, anterior cingulate gyrus, pre and post cen- IPL (x = −44, y = −50, z = 54; t = 4.38, cluster size = 118 vox-
tral gyrus, and supramarginal gyrus (Figure 4). These lesioned els, p corr < 0.05) were significantly greater at 6 months com-
areas included the right PMd and IPL, which were domi- pared with 13 months after the stoke (Figure 6A). These brain
nantly activated during the mental calculation and digit mem- regions were repeatedly activated in many language- related cog-
ory tasks in the previous functional MRI studies of abacus nitive tasks (Paulesu et al., 1993; Fiez et al., 1996; Smith et al.,
experts (Tanaka et al., 2002; Hanakawa et al., 2003; Chen et al., 1998). In contrast, activity in the left SPL (x = −20, y = −66,
2006; Ku et al., 2012). The lesion was not observed in the left z = 66; t = 3.60, cluster size = 10, p svc < 0.05) was significantly
hemisphere. greater at 13 months compared with 6 months after her stroke
Left Right
FIGURE 4 | T1-weighted structural MRI of the patient. The lesion was observed in the fronto-parietal cortex, including the posterior parts of the inferior and
superior frontal gyrus, anterior insula, anterior cingulate gyrus, pre and post central gyrus, and supramarginal gyrus. No lesion was observed in the left
hemisphere.

A 6 months after stroke 13 months after stroke
One Digit >

visual fixation
Two Digit >

visual fixation
Two >
One Digit
FIGURE 5 | (A) Active areas during the mental calculation task relative to indicates the lesion area. Brain activity is overlayed on the spatially normalized
those during the visual fixation condition 6 and 13 months after her stroke. rendering images (voxel level, p < 0.001 uncorrected for multiple comparison;
(B) Active areas during the two-digit mental calculation task relative to those cluster level p < 0.05 FWE). The detailed coordinates and statistical values are
during the one-digit task 6 and 13 months after her stroke. The blue region listed in Table A1 in Appendix online.
A
3
contrast estimate at
[−44, −50, 54]
0
one two one two
Session 1 Session 2
FIGURE 6 | (A) Active areas during the mental calculation task-specific level, p < 0.001 uncorrected for multiple comparison; cluster level
to 6 months after her stroke. These areas include Broca’s area, the left p < 0.05 FWE). In each bar graph, the vertical axis indicates contrast
dorsolateral prefrontal cortex and the inferior parietal cortex. (B) Active estimated values relative to the baseline, whereas the horizontal axis
areas specific to 13 months after her stroke. Only the left superior indicates the type of task. One and two in the figure indicates one-digit
parietal cortex was significantly activated on this comparison. The blue and two-digit addition task, respectively. Blue and orange color bar
color on the rendering structural image indicates the damaged cortical indicates session 1 and 2, respectively. The error bars represent
area. Only statistically significantly different activity is shown (voxel standard deviations across scans.
(Figure 6B). Activity in the left SPL was observed in the pre- calculation strategy from a phonological – based to a mental aba-
vious functional imaging studies of mental calculation tasks in cus – based strategy according to her level of recovery from the
abacus experts (Hanakawa et al., 2003; Chen et al., 2006; Wu stroke.
et al., 2009). These functional MRI results were very consistent
with the patient’s subjective report that she was able to shift the Digit memory task
The patient correctly answered all trials in both functional MRI
sessions. The present analysis of the digit memory task focuses on
6 months after stroke the brain activities associated with memory retention and thus
the brain activities only during the delay period are reported.
Figure 7 shows brain activity associated with the delay interval
period during the digit memory tasks relative to the visual fixation
condition (see Table A2 in Appendix online). Overall, brain activ-
ity was left lateralized 6 months after her stroke, whereas bilateral
activation was observed 13 months after her stroke. These brain
regions include the inferior and middle frontal gyrus, insula, sup-
plementary motor area, IPL, SPL, cuneus, fusiform gyrus, inferior
temporal gyrus, and cerebellum.
13 months after stroke A direct comparison of the brain activities observed dur-
ing the delay period between the two sessions is shown in
Figure 8. No brain regions were observed that showed sig-
nificant regional-specific activities at 6 months compared with
those at 13 months after her stroke. In contrast, activities in
the bilateral SPL (left x = −18, y = −64, z = 64; t = 6.36, cluster
size = 223, p corr < 0.05; right x = 20, y = −48, z = 70; t = 4.93,
cluster size = 132, p corr < 0.05) and the right visual association
FIGURE 7 | Active areas during the delay period on the digit memory
cortex (x = 36, y = −80, z = −14; t = 6.77, cluster size = 529, p
task relative to these during the visual fixation condition 6 and corr < 0.05) were significantly greater at 13 months compared
13 months after her stroke. The blue region indicates the lesion area. Brain with 6 months after her stroke. The bilateral activities in the SPL
activity is overlayed on the spatially normalized rendering images (voxel during the delay period were observed in the previous functional
level, p < 0.001 uncorrected for multiple comparison; cluster level p < 0.05
MRI study of abacus experts (Tanaka et al., 2002). Thus, the result
FWE). The detailed coordinates and statistical values are listed in Table A2
in Appendix online. suggests that the visuospatial strategy of mental abacus represen-
tation might be more dominantly used in the digit memory task,
FIGURE 8 | Active areas during the retention period in the digit memory lobule and visual association cortex were significantly activated. Statistical
task specific to 13 months after her stroke. The bilateral superior parietal threshold was the same as with the mental calculation task.

the same as in the mental calculation task, at 13 months after performance, suggesting her use of a mental abacus on the
her stroke. Again, this was consistent with the patient’s subjec- digit memory task and mental calculations 13 months after her
tive report that she was able to utilize the mental abacus strategy stroke.
13 months after her stroke.
DISCUSSION
Verbal fluency and hand grip tasks This is the first case report on the impairment of mental abacus
Figure 9 shows the results of verbal fluency and hand grip tasks. ability by a brain lesion and on recovery-related brain activity. The
There was significant task-specific activity mainly in the left patient’s knowledge and operation of basic arithmetic facts and of
DLPFC for the verbal fluency task and in the right primary motor a physical abacus were all intact. Her impairment of arithmetic
cortex for the left hand grip task, respectively. In contrast, in both ability was specific to mental calculation and digit memory only
tasks, the left SPL, which was dominantly activated during her based on the mental abacus strategy. Therefore, we consider that
mental calculation and digit memory tasks, was not significantly this would be a specific case of spatial acalculia (Hécaen et al.,
activated compared with the visual fixation condition. These find- 1961; Hartje, 1987; Granà et al., 2006). This is a quite rare case and
ings suggest that activation in the SPL was specific to mental we have named this “abacus-based acalculia.”
calculation and digit memory tasks 13 months after the stroke. The results of the present study show that brain activity dur-
ing mental calculation at 13 months after her stroke was observed
EXPERIMENT 2 more in an area implicated in visuospatial working memory
Behavioral experiment (Jonides et al., 1993; Mellet et al., 1996; Courtney et al., 1998a,b;
The number of correctly answered trials was 12 for the Rowe et al., 2001; Tanaka et al., 2005; Oshio et al., 2010), whereas
human face and gray rectangle conditions, compared with at 6 months after her stroke, brain activity was more predominant
6 for the abacus picture condition (Figure 10B). Therefore, in the left hemisphere in areas related to verbal working mem-
the number of the correct trials in the abacus picture con- ory (Paulesu et al., 1993; Fiez et al., 1996; Smith et al., 1998).
dition was clearly fewer than that in the other two distrac- Brain activity at 13 months after her stroke was observed in the
tor conditions. This result showed that the presentation of left SPL, whereas that at 6 months after her stroke was observed
pictures of abacus figures interfered with the patient’s task in Broca’s area and the left DLPFC and IPL. This shift of region-
specific brain activities is consistent with her subjective report
that she was able to shift her calculation strategy from a verbal
A 2
to a visuospatial strategy according to the level of her recovery
from the stroke. In a behavioral experiment using interference
contrast estimate
1
paradigms, a visual presentation of an abacus picture, but not
a human face picture, interfered with her performance of digit
0 memory, confirming her use of the mental abacus 13 months after
her stroke.
−1 The present result is consistent with previous functional imag-
ing studies that reported activation in the SPL during mental
−2 calculation and digit memory tasks in abacus users (Tanaka et al.,
L.DLPFC R.M1 L.SPL 2002; Hanakawa et al., 2003; Chen et al., 2006; Wu et al., 2009). It
is possible that a spatial representation of numbers is developed
B 2
through abacus practice, which involves rulebased visuomotor
processing, and utilized in mental calculation and digit memory
tasks, because it is more efficient to mentally manipulate large
contrast estimate
1
numbers using a spatial representation than a sequentially orga-
nized phonological representation (Hatano et al., 1977; Hatano
0
and Osawa, 1983 Hatano et al., 1987; Hatta et al., 1989; Hishi-
tani, 1990; Tanaka et al., 2008; Frank and Barner, 2012). The
−1 SPL might be a key brain region for such non-verbal visuospatial
representation of numbers.
−2 According to the structural MRI, the lesion area involved the
L.DLPFC R.M1 L.SPL
right fronto-parietal regions. Her impairment of mental aba-
cus ability due to her right hemispheric lesion was consistent
FIGURE 9 | Active areas during verbal fluency and left hand grip tasks
13 months after the stroke. (A) There were significant task-specific
with previous behavioral and neuroimaging studies that indicate
activities in the left dorsolateral prefrontal cortex (DLPFC) for verbal fluency involvement of the right hemisphere in the superior arithmetic
task and (B) right primary motor cortex (M1) for left hand grip task. The left abilities of abacus users (Hatta and Ikeda, 1988; Tanaka et al., 2002;
SPL was not significantly activated. In each bar graph, the vertical axis Hanakawa et al., 2003; Chen et al., 2006; Wu et al., 2009). More
indicates contrast estimated values relative to the visual fixation task,
specifically, her lesion area included the right PMd and IPL, which
whereas the horizontal axis indicates each brain region. The error bars
represent standard deviations across scans. have been repeatedly activated in the previous functional neu-
roimaging studies of abacus users (Tanaka et al., 2002; Hanakawa
FIGURE 10 | (A) A delayed digit recall task using interference paradigms, delayed digit recall task. The number of correctly answered trials was 12
based on Hatta et al. (1989). The patient was asked to recall a target digit when a human face or a gray rectangle were presented as distractors. In
sequence after a 15-s retention interval. Three different types of visual contrast, the number of correctly answered trials was six when the distractor
distractors were presented during the retention interval (pictures of abacus was pictures of abacus figures. This result indicated that the presentation of
figures, human faces, or gray rectangles). (B) Behavioral performance in the abacus figures interfered with the patient’s digit memory performance.
et al., 2003; Chen et al., 2006; Wu et al., 2009). Therefore, the used in the mental calculation task, activation in the bilateral SPL
present study may suggest the functional relevance of these brain was observed.
regions to the mental calculation and digit memory of abacus Regarding the task-specific activity of the SPL, one might argue
users. However, we should be careful about such interpretations that the observed differences in SPL activity among arithmetic and
because the lesion area not only covered the PMd and IPL but also other control tasks (such as verbal fluency and hand grip) might
included relatively large areas of the right frontal and parietal cor- be explained by the difference in task difficulty. However, that
tex. A non-invasive brain stimulation study or neuropsychological would be unlikely because the SPL activity during verbal fluency
study of patients with a more focal brain lesion will clarify this and hand grip tasks was not significantly different compared with
issue. the easiest visual fixation condition in which the subject simply
The activation in the SPL was less evident at 6 months com- watched the fixation on the screen. If the explanation of activity
pared with 13 months after her stroke. This implies that the dam- difference by task difficulty is true, then SPL activity during the
aged regions in the right hemisphere, possibly the PMd and IPL, verbal fluency and hand grip tasks should be greater than during
and the SPL may work as a functional network during abacus- the visual fixation task. Therefore, it is reasonable to consider that
based mental calculation and digit memory. In fact, it is known the SPL activity would be specific for her mental abacus use after
that there is an anatomical and functional connectivity between her stroke recovery.
the premotor and parietal cortex (Wise et al., 1997; Luppino et al., It has been proposed that the human capacity for mathematical
1999; Wise and Murray, 2000; Tanaka et al., 2005; Oshio et al., intuition depends on both linguistic competence and visuospatial
2010). Damage in one cortical node may induce less activity in representations (Dehaene et al., 1999). By a combination of neu-
another cortical node within the functional network. However, ropsychological and neuroimaging techniques, the present finding
13 months after her stroke, the SPL might be able to work without provides evidence for an important role of visual imagery in men-
the damaged brain regions, possibly because of remote cortical tal arithmetic operations and also for its underlying neural corre-
reorganization that may occur within the intact SPL region (Frost lates, the superior parietal cortex. The SPL might be an important
et al., 2003; Fridman et al., 2004; Dancause et al., 2006). cortical structure for non-verbal forms of number representation
In the present study, the significant activity in the SPL was left for calculation. The present finding may contribute to developing
lateralized in the mental calculation task, whereas bilateral acti- our understanding of the relationship between mental imagery
vation was found in the digit memory task. This might be due and mental arithmetic operations.
to differences in task difficulty between the two tasks, based on There are several limitations for this study. First, this is a
her subjective report after the experiment. A previous functional single case study and it is difficult to generalize this finding
MRI study has reported that bilateral SPL activity in abacus users to other populations. Second, the patient was left handed and
was more evident in the tasks with a higher cognitive demand thus it is difficult to discuss the lateralization of brain activa-
(Hanakawa et al., 2003). In fact, if a lower statistical threshold was tion. For this reason, we did not make any conclusions on the

lateralization of brain activity from the present study. Third, the and also the first longitudinal functional MRI study of abacus
results of the behavioral interference task might be explained by users.
a potential difference in difficulty between the distractors, such In conclusion, the present study reports for the first time a
as the difference in the visual complexity of stimuli. Thus, in case of “abacus-based acalculia” caused by a brain lesion. Together
future studies, interference tasks should be matched for difficulty with previous neuroimaging studies, the present result provides
and the subject should be asked to make a behavioral response evidence for an important role of the PMd and parietal cortex in
to the interfering stimuli, to be certain that the subject is actu- the mental calculation and digit memory tasks of abacus users.
ally processing the stimuli. Despite these limitations, however, we
believe that this result has important implications regarding the ACKNOWLEDGMENTS
neural substrates underlying the superior arithmetic ability of aba- This work was supported by grants from the Grants-in-Aid for
cus users, because this is the first neuropsychological case report Scientific Research (KAKENHI) to Satoshi Tanaka (24680061).
REFERENCES short-term maintenance of ver- overall assessment of SLTA results. neural processes in abacus men-
Brett, M., Leff, A. P., Rorden, C., and bal information. J. Neurosci. 16, High. Brain Funct. Res. 4, 638–646. tal addition: an EEG and FMRI
Ashburner, J. (2001). Spatial nor- 808–822. Hatano, G., Amaiwa, S., and Shimizu, case study. PLoS ONE 7, e36410.
malization of brain images with Frank, M. C., and Barner, D. (2012). K. (1987). Formation of a mental doi:10.1371/journal.pone.0036410
focal lesions using cost function Representing exact number visually abacus for computation and its use Luppino, G., Murata, A., Govoni, P.,
masking. Neuroimage 14, 486–500. using mental abacus. J. Exp. Psychol. as a memory devices for digits: a and Matelli, M. (1999). Largely
Chen, F., Hu, Z., Zhao, X., Wang, R., Gen. 141, 134–149. developmental study. Dev. Psychol. segregated parietofrontal connec-
Yang, Z., Wang, X., and Tang, X. Fridman, E. A., Hanakawa, T., Chung, 23, 832–838. tions linking rostral intraparietal
(2006). Neural correlates of serial M., Hummel, F., Leiguarda, R. C., Hatano, G., Miyake, Y., and Bink, M. cortex (areas AIP and VIP) and
abacus mental calculation in chil- and Cohen, L. G. (2004). Reorga- (1977). Performance of expert aba- the ventral premotor cortex (areas
dren: a functional MRI study. Neu- nization of the human ipsilesional cus operators. Cognition 5, 57–71. F5 and F4). Exp. Brain Res. 128,
rosci. Lett. 403, 46–51. premotor cortex after stroke. Brain Hatano, G., and Osawa, K. (1983). Digit 181–187.
Corbetta, M., Kincade, M. J., Lewis, C., 127, 747–758. memory of grand experts in abacus- Mellet, E., Tzourio, N., Crivello, F.,
Snyder, A. Z., and Sapir, A. (2005). Friston, K. J., Holmes, A. P., Wors- derived mental calculation. Cogni- Joliot, M., Denis, M., and Mazoyer, B.
Neural basis and recovery of spatial ley, K. J., Poline, J. P., Frith, C. D., tion 15, 95–110. (1996). Functional anatomy of spa-
attention deficits in spatial neglect. and Frackowiak, R. S. J. (1994). Sta- Hatta, T., Hirose, T., Ikeda, K., and tial mental imagery generated from
Nat. Neurosci. 8, 1603–1610. tistical parametric maps in func- Fukuhara, H. (1989). Digit mem- verbal instructions. J. Neurosci. 16,
Courtney, S. M., Petit, L., Haxby, J. V., tional imaging: a general linear ory of soroban experts: evidence of 6504–6512.
and Ungerleider, L. G. (1998a). The approach. Hum. Brain. Mapp. 2, utilization of mental imagery. Appl. Oshio, R., Tanaka, S., Sadato, N., Sokabe,
role of prefrontal cortex in working 189–210. Cogn. Psychol. 3, 23–33. M., Hanakawa, T., and Honda, M.
memory: examining the contents of Frost, S. B., Barbay, S., Friel, K. M., Hatta, T., and Ikeda, K. (1988). Hemi- (2010). Differential effect of double-
consciousness. Philos. Trans. R. Soc. Plautz, E. J., and Nudo, R. J. spheric specialization of abacus pulse TMS applied to dorsal pre-
Lond. B. Biol. Sci. 353, 1819–1828. (2003). Reorganization of remote experts in mental calculation: evi- motor cortex and precuneus dur-
Courtney, S. M., Petit, L., Maisog, J. cortical regions after ischemic brain dence from the results of time- ing internal operation of visuospa-
M., Ungerkeider, L. G., and Haxby, injury: a potential substrate for sharing tasks. Neuropsychologia 26, tial information. Neuroimage 49,
J. V. (1998b). An area special- stroke recovery. J. Neurophysiol. 89, 877–893. 1108–1115.
ized for spatial working memory in 3205–3214. Hécaen, H., Angerlergues, R., and Paulesu, E., Frith, C. D., and Frackowiak,
human frontal cortex. Science 279, Granà, A., Hofer, R., and Semenza, C. Houiller, S. (1961). Les variétés clin- R. S. (1993). The neural correlates
1347–1351. (2006). Acalculia from a right hemi- iques des acalculies au cours des of the verbal component of working
Dancause, N., Barbay, S., Frost, S. sphere lesion dealing with “where” lésions rétrorolandiques: approche memory. Nature 362, 342–345.
B., Zoubina, E. V., Plautz, E. J., in multiplication procedures. Neu- statistique du problème. Rev. Neurol. Price, C. J., and Crinion, J. (2005). The
Mahnken, J. D., and Nudo, R. J. ropsychologia 14, 2972–2986. (Paris) 105, 85–103. latest on functional imaging studies
(2006). Effects of small ischemic Hanakawa, T., Honda, M., and Hallett, Heiss, W. D., and Thiel, A. (2006). A pro- of aphasic stroke. Curr. Opin. Neurol.
lesions in the primary motor cor- M. (2004). Amodal imagery in ros- posed regional hierarchy in recovery 18, 429–434.
tex on neurophysiological organiza- tral premotor areas. Behav. Brain Sci. of post-stroke aphasia. Brain. Lang. Rowe, J., Toni, I., Josephs, O., Frack-
tion in ventral premotor cortex. J. 27, 406–407. 98, 118–123. owiak, R. S., and Passingham, R.
Neurophysiol. 96, 3506–3511. Hanakawa, T., Honda, M., Okada, T., Hishitani, S. (1990). Imagery expert: E. (2001). The prefrontal cortex:
Dehaene, S., Spelke, E., Pinel, P., Fukuyama, H., and Shibasaki, H. how do expert abacus operators response selection or maintenance
Stanescu, R., and Tsivkin, S. (2003). Neural correlates underly- process imagery? Appl. Cogn. Psy- within eorking memory? Science
(1999). Sources of mathematical ing mental calculation in abacus chol. 4, 33–46. 288, 1656–1660.
thinking: behavioral and brain- experts: a functional magnetic res- Hu, Y., Geng, F., Tao, L., Hu, N., Smith, E. E., Jonides, J., Marshuetz, C.,
imaging evidence. Science 284, onance imaging study. Neuroimage Du, F., Fu, K., and Chen, F. and Koeppe, R. A. (1998). Compo-
970–974. 19, 296–307. (2011). Enhanced white matter nents of verbal working memory:
Evans, A. C., Collins, D. L., Mills, S. Hartje, W. (1987). “The effect of spatial tracts integrity in children with aba- evidence from neuroimaging. Proc.
R., Brown, E. D., Kelly, R. L., and disorders on arithmetical skills,” in cus training. Hum. Brain Mapp. 32, Natl. Acad. Sci. U.S.A. 95, 876–882.
Peters, T. M. (1993). 3D statistical Mathematical disabilities: a cognitive 10–21. Stigler, J. (1984). “Mental abacus”: the
neuroanatomical models from 305 neuropsychological perspective, eds G. Jonides, J., Smith, E. E., Koeppe, R. A., effects of abacus training on Chinese
MRI volumes. IEEE Nucl. Sci. Symp. Deloche, and X. Seron (Hillsdale, NJ: Awh, E., Minoshima, S., and Mintun, children’s mental calculation. Cogn.
Conf. Rec. 108, 1877–1878. Erlbaum), 121–125. M. A. (1993). Spatial working mem- Psychol. 16, 145–176.
Fiez, J. A., Raife, E. A., Balota, D. A., Hasegawa, T., Kishi, H., Shigeno, K., ory in humans as revealed by PET. Tanaka, S., Hanakawa, T., and Honda,
Schwarz, J. P., Raichle, M. E., and Tanemura, J., Kusunoki, K., Kifune, Nature 363, 623–625. M. (2008). Neural substrates under-
Petersen, S. E. (1996). A positron Y., and Yoshida, M. (1984). A study Ku, Y., Hong, B., Zhou, W., Bodner, M., lying cognitive expertise. Brain
emission tomography study of the on aphasia rating scale: a method for and Zhou, Y. D. (2012). Sequential Nerve 60, 257–262.
Tanaka, S., Honda, M., and Sadato, Wise, S. P., Boussaoud, D., Johnson, P. J. Nucl. Med. Mol. Imaging 36, MRI study of a skilled abacus user with a
N. (2005). Modality-specific cogni- B., and Caminiti, R. (1997). Premo- 436–445. right hemispheric lesion. Front. Psychol-
tive function of medial and lateral tor and parietal cortex: corticocor- ogy 3:315. doi: 10.3389/fpsyg.2012.00315
human Brodmann area 6. J. Neu- tical connectivity and combinatorial Conflict of Interest Statement: The This article was submitted to Frontiers in
rosci. 25, 496–501. computations. Annu. Rev. Neurosci. authors declare that the research was Perception Science, a specialty of Frontiers
Tanaka, S., Michimata, C., Kaminaga, T., 20, 25–42. conducted in the absence of any com- in Psychology.
Honda, M., and Sadato, N. (2002). Wise, S. P., and Murray, E. A. (2000). mercial or financial relationships that Copyright © 2012 Tanaka, Seki,
Superior digit memory of aba- Arbitrary associations between could be construed as a potential con- Hanakawa, Harada, Sugawara, Sadato,
cus experts: an event-related func- antecedents and actions. Trends flict of interest. Watanabe and Honda. This is an
tional MRI study. Neuroreport 13, Neurosci. 23, 271–276. open-access article distributed under
2187–2191. Wu, T. H., Chen, C. L., Huang, Received: 27 April 2012; accepted: 10 the terms of the Creative Commons
Ward, N. S., Brown, M. M., Thomp- Y. H., Liu, R. S., Hsieh, J. C., August 2012; published online: 28 August Attribution License, which permits use,
son, A. J., and Frackowiak, R. and Lee, J. J. (2009). Effects of 2012. distribution and reproduction in other
S. (2003). Neural correlates of long-term practice and task com- Citation: Tanaka S, Seki K, Hanakawa forums, provided the original authors
motor recovery after stroke: a lon- plexity on brain activities when T, Harada M, Sugawara SK, Sadato N, and source are credited and subject to
gitudinal fMRI study. Brain 126, performing abacus-based mental Watanabe K and Honda M (2012) Aba- any copyright notices concerning any
2476–2496. calculations: a PET study. Eur. cus in the brain: a longitudinal functional third-party graphics etc.

APPENDIX
Table A1 | Brain activity during mental calculation task.
Cluster size Voxel T MNI coordinate Laterality Anatomy
x y z (mm)
ONE-DIGIT MENTAL CALCULATION >VISUAL FIXATION 6 MONTHS AFTER THE STROKE

952 6.17 −42 −40 68 L Postcentral gyrus
124 5.02 −64 −22 40 L Supramarginal gyrus
108 5.00 14 −72 −44 R Cerebellum (lobule VIIb)
178 4.86 −30 42 −10 L Middle orbital gyrus
176 4.67 −46 0 10 L Superior frontal gyrus
90 4.59 18 62 24 R Superior frontal gyrus
163 4.19 −22 −4 74 L Cerebellum (lobule VIIb)
TWO-DIGIT MENTAL CALCULATION >VISUAL FIXATION 6 MONTHS AFTER THE STROKE
497 6.15 26 −72 −18 R Cerebellum
312 4.91 −22 −98 10 L Middle occipital gyrus
1015 5.42 42 −38 62 R Postcentral gyrus
134 4.28 −46 −64 −8 L Inferior temporal gyrus
427 4.12 34 44 34 R Middle frontal gyrus
TWO-DIGIT > ONE-DIGIT MENTAL CALCULATION 6 MONTHS AFTER THE STROKE
290 6.42 −44 52 10 L Middle frontal gyrus
ONE-DIGIT MENTAL CALCULATION >VISUAL FIXATION 13 MONTHS AFTER THE STROKE
175 4.70 24 −70 −12 R Fusiform gyrus
TWO-DIGIT MENTAL CALCULATION >VISUAL FIXATION 13 MONTHS AFTER THE STROKE
588 7.91 18 −56 78 R Superior parietal lobule
1417 7.66 −20 −66 64 L Superior parietal lobule
320 6.80 −50 −4 58 L Precentral gyrus
581 6.42 −2 −48 −14 L Cerebellum (vermis)
547 5.43 −22 −90 12 L Middle occipital gyrus
266 5.73 −60 2 38 L Precentral gyrus
337 4.55 22 −98 10 R Superior occipital gyrus
TWO-DIGIT > ONE-DIGIT MENTAL CALCULATION 13 MONTHS AFTER THE STROKE
461 6.04 −18 −64 64 L Superior parietal lobule
508 4.31 30 −84 10 R Middle occipital gyrus
Table A2 | Brain activity during digit memory task.
Cluster size (number of voxel) T value MNI coordinates Laterality Anatomy
x y z (mm)
DELAY PERIOD DURING DIGIT MEMORY TASK >VISUAL FIXATION 6 MONTHS AFTER STROKE
7021 10.15 −62 −20 38 L Supramarginal gyrus
656 7.97 26 −70 −20 R Cerebellum (lobule VI)
368 6.08 −38 34 −4 L Inferior frontal gyrus
162 5.42 −2 12 58 L SMA
215 4.68 −48 −54 −12 R Inferior temporal gyrus
166 4.75 46 52 4 L Middle frontal gyrus
178 4.7 −28 14 10 L Insula
DELAY PERIOD DURING DIGIT MEMORY TASK >VISUAL FIXATION 13 MONTHS AFTER STROKE
10518 18.9 −42 −44 56 L Inferior parietal lobule
3206 17.87 24 −72 −16 R Fusiform gyrus
495 10.04 −40 −62 −2 L Middle temporal gyrus
946 8.67 −44 42 22 L Middle frontal gyrus
581 7.75 −26 −66 −24 L Cerebellum (lobule VI)
1552 7.62 2 −88 20 R Cuneus
98 3.88 52 −8 20 R Rolandic operculum
MNI coordinates (x, y, z) and statistical t-values at the peak anatomical voxel and size of cluster (number of voxels) are listed (P < 0.001 uncorrected for multiple
comparisons at voxel level, p < 0.05 corrected for multiple comparisons with Gaussian random field theory at cluster level). L indicates the left hemisphere whereas
R indicates the right hemisphere.

Mental Imagery

Uploaded by

Copyright:

Available Formats

Mental Imagery

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Mental Imagery

Uploaded by

Copyright:

Available Formats

What is Frontiers and what does it aim to achieve?

What is Frontiers and what does it aim to achieve?

What are Frontiers Research Topics?

What are Frontiers Research Topics?

MENTAL IMAGERY

PSYCHOLOGY HUMAN NEUROSCIENCE

July 2013 | Mental Imagery | 1

Our ability to be conscious of the world

July 2013 | Mental Imagery | 2

July 2013 | Mental Imagery | 3

July 2013 | Mental Imagery | 4

www.frontiersin.org April 2013 | Volume 4 | Article 198 | 5

Hemispheric differences within the fronto-parietal network

www.frontiersin.org June 2012 | Volume 3 | Article 214 | 6

Frontiers in Psychology | Perception Science June 2012 | Volume 3 | Article 214 | 7

www.frontiersin.org June 2012 | Volume 3 | Article 214 | 8

Frontiers in Psychology | Perception Science June 2012 | Volume 3 | Article 214 | 9

www.frontiersin.org June 2012 | Volume 3 | Article 214 | 10

Frontiers in Psychology | Perception Science June 2012 | Volume 3 | Article 214 | 11

www.frontiersin.org June 2012 | Volume 3 | Article 214 | 12

Frontiers in Psychology | Perception Science June 2012 | Volume 3 | Article 214 | 13

www.frontiersin.org June 2012 | Volume 3 | Article 214 | 14

Frontiers in Psychology | Perception Science June 2012 | Volume 3 | Article 214 | 15

Unmasking the Perky effect: spatial extent of image

www.frontiersin.org August 2012 | Volume 3 | Article 296 | 16

Frontiers in Psychology | Perception Science August 2012 | Volume 3 | Article 296 | 17

www.frontiersin.org August 2012 | Volume 3 | Article 296 | 18

RESULTS AND DISCUSSION

Frontiers in Psychology | Perception Science August 2012 | Volume 3 | Article 296 | 19

MATERIALS AND METHODS EXPERIMENT 3

www.frontiersin.org August 2012 | Volume 3 | Article 296 | 20

Frontiers in Psychology | Perception Science August 2012 | Volume 3 | Article 296 | 21

www.frontiersin.org August 2012 | Volume 3 | Article 296 | 22

Training visual imagery: improvements of metacognition,

www.frontiersin.org July 2012 | Volume 3 | Article 224 | 23

Frontiers in Psychology | Perception Science July 2012 | Volume 3 | Article 224 | 24

Cue Imagery Subjective ratings Rivalry 0.8

www.frontiersin.org July 2012 | Volume 3 | Article 224 | 25

Frontiers in Psychology | Perception Science July 2012 | Volume 3 | Article 224 | 26

negligible influence on participant’s reports of rivalry dominance,

www.frontiersin.org July 2012 | Volume 3 | Article 224 | 27

0.6 0.6 0.6

0.4 0.4 0.4

0.2 0.2 0.2

0.8 0.8 0.8

0.6 0.6 0.6

0.4 0.4 0.4

0.2 0.2 0.2

Frontiers in Psychology | Perception Science July 2012 | Volume 3 | Article 224 | 28

0.65 vividness were not better at evaluating their imagery strength

Proportion Aroc effort

0.6 day 1 0.6

www.frontiersin.org July 2012 | Volume 3 | Article 224 | 29

Frontiers in Psychology | Perception Science July 2012 | Volume 3 | Article 224 | 30

www.frontiersin.org July 2012 | Volume 3 | Article 224 | 31

Frontiers in Psychology | Perception Science July 2012 | Volume 3 | Article 224 | 32

www.frontiersin.org July 2012 | Volume 3 | Article 224 | 33

New percepts via mental imagery?

www.frontiersin.org October 2012 | Volume 3 | Article 360 | 34

Frontiers in Psychology | Perception Science October 2012 | Volume 3 | Article 360 | 35

k = k (n−k )! with n! = 123. . . n. From Herzog and Esfeld (2009).