Previous studies have shown that spike-timing-dependent plasticity (STDP) can be used in spiking ... more Previous studies have shown that spike-timing-dependent plasticity (STDP) can be used in spiking neural networks (SNN) to extract visual features of low or intermediate complexity in an unsupervised manner. These studies, however, used relatively shallow architectures, and only one layer was trainable. Another line of research has demonstrated - using rate-based neural networks trained with back-propagation - that having many layers increases the recognition robustness, an approach known as deep learning. We thus designed a deep SNN, comprising several convolutional (trainable with STDP) and pooling layers. We used a temporal coding scheme where the most strongly activated neurons fire first, and less activated neurons fire later or not at all. The network was exposed to natural images. Thanks to STDP, neurons progressively learned features corresponding to prototypical patterns that were both salient and frequent. Only a few tens of examples per category were required and no label ...
The primate visual system has inspired the development of deep artificial neural networks, which ... more The primate visual system has inspired the development of deep artificial neural networks, which have revolutionized the computer vision domain. Yet these networks are much less energy-efficient than their biological counterparts, and they are typically trained with backpropagation, which is extremely data-hungry. To address these limitations, we used a deep convolutional spiking neural network (DCSNN) and a latency-coding scheme. We trained it using a combination of spike-timingdependent plasticity (STDP) for the lower layers and reward-modulated STDP (R-STDP) for the higher ones. In short, with R-STDP a correct (resp. incorrect) decision leads to STDP (resp. anti-STDP). This approach led to an accuracy of 97.2% on MNIST, without requiring an external classifier. In addition, we demonstrated that R-STDP extracts features that are diagnostic for the task at hand, and discards the other ones, whereas STDP extracts any feature that repeats. Finally, our ap-* Corresponding author.
Numerous theories of neural processing, often motivated by experimental observations, have explor... more Numerous theories of neural processing, often motivated by experimental observations, have explored the computational properties of neural codes based on the absolute or relative timing of spikes in spike trains. Spiking neuron models and theories however, as well as their experimental counterparts, have generally been limited to the simulation or observation of isolated neurons, isolated spike trains, or reduced neural populations. Such theories would therefore seem inappropriate to capture the properties of a neural code relying on temporal spike patterns distributed across large neuronal populations. Here we report a range of computer simulations and theoretical considerations that were designed to explore the possibilities of one such code and its relevance for visual processing. In a unified framework where the relation between stimulus saliency and spike relative timing plays the central role, we describe how the ventral stream of the visual system could process natural input scenes and extract meaningful information, both rapidly and reliably. The first wave of spikes generated in the retina in response to a visual stimulation carries information explicitly in its spatio-temporal structure: the most salient information is represented by the first spikes over the population. This spike wave, propagating through a hierarchy of visual areas, is regenerated at each processing stage, where its temporal structure can be modified by (i) the selectivity of the cortical neurons, (ii) lateral interactions and (iii) top-down attentional influences from higher order cortical areas. The resulting model could account for the remarkable efficiency and rapidity of processing observed in the primate visual system.
Accurate binocular depth perception requires information about both stereopsis (relative depth) a... more Accurate binocular depth perception requires information about both stereopsis (relative depth) and distance (absolute depth). It is unclear how these two types of information are integrated in the visual system. In alert, behaving monkeys the responsiveness of a large majority of neurons in the primary visual cortex (area V1) was modulated by the viewing distance. This phenomenon affected particularly disparity-related activity and background activity and was not dependent on the pattern of retinal stimulation. Therefore, extraretinal factors, probably related to ocular vergence or accommodation, or both, can affect processing early in the visual pathway. Such modulations could be useful for (i) judging true distance and (ii) scaling retinal disparity to give information about three-dimensional shape.
Experiments investigating the mechanisms involved in visual processing often fail to separate low... more Experiments investigating the mechanisms involved in visual processing often fail to separate low-level encoding mechanisms from higher-level behaviorally relevant ones. Using an alternating dual-task event-related potential (ERP) experimental paradigm (animals or vehicles categorization) where targets of one task are intermixed among distractors of the other, we show that visual categorization of a natural scene involves different mechanisms with different time courses: a perceptual, task-independent mechanism, followed by a task-related, category-independent process. Although average ERP responses reflect the visual category of the stimulus shortly after visual processing has begun (e.g. 75-80 msec), this difference is not correlated with the subject's behavior until 150 msec poststimulus.
Page 1. 1 BIOLOGICAL CONSTRAINTS ON CONNECTIONIST MODELLING Simon J. Thorpe and Michel Imbert, In... more Page 1. 1 BIOLOGICAL CONSTRAINTS ON CONNECTIONIST MODELLING Simon J. Thorpe and Michel Imbert, Institut des Neurosciences, Departement des Neurosciences de la Vision, Université Pierre et Marie Curie, 9, Quai St. Bernard, 75005, Paris, FRANCE. ABSTRACT ...
To investigate the importance of feedback loops in visual information processing, we have analyze... more To investigate the importance of feedback loops in visual information processing, we have analyzed the dynamic aspects of neuronal responses to oriented gratings in cortical area V1 of the awake primate. If recurrent feedback is important in generating orientation selectivity, the initial part of the neuronal response should be relatively poorly selective, and full orientation selectivity should only appear after a delay. Thus, by examining the dynamics of the neuronal responses it should be possible to assess the importance of feedback processes in the development of orientation selectivity. The results were base on a sample of 259 cells recorded in two monkeys, of which 89% were visually responsive. Of these, approximately two-thirds were orientation selective. Response latency varied considerably between neurons, ranging from a minimum of 41 ms to over 150 ms, although most had latencies of 50–70 ms. Orientation tuning (defined as the bandwidth at half-height) ranged from 16 deg ...
Finding ones way to an unknown destination, navigating complex routes, finding desired inanimate ... more Finding ones way to an unknown destination, navigating complex routes, finding desired inanimate objects; these are all tasks that can be challenging for the visually impaired. The project NAVIG (Navigation Assisted by artificial VIsion and GNSS) is directed towards increasing the autonomy of visually impaired users in known and unknown environments, exterior and interior, large scale and small scale, through a combination of a Global Navigation Satellite System (GNSS) and rapid visual recognition with which the precise position of the user can be determined. Relying on geographical databases and visually identified objects, the user is guided to their desired destination through spatialized audio rendering, always maintained in the head-centered reference frame. This paper presents the overall project design and architecture of the NAVIG system.
In this paper I propose a very radical reform of the taxation system, in which a single flat rate... more In this paper I propose a very radical reform of the taxation system, in which a single flat rate financial transaction tax (FTT) is used to replace the vast majority of existing taxes (including VAT, income tax, taxes on profits...). Existing economic data indicates that a flat rate FTT of 1% would generate far more revenue that is currently generated by all existing taxes, and would allow governments to rapidly repay debts and restore programs of public expenditure as well as allowing resources to be allocated to globally important challenges such as third world development, climate change and health issues.
Previous studies have shown that spike-timing-dependent plasticity (STDP) can be used in spiking ... more Previous studies have shown that spike-timing-dependent plasticity (STDP) can be used in spiking neural networks (SNN) to extract visual features of low or intermediate complexity in an unsupervised manner. These studies, however, used relatively shallow architectures, and only one layer was trainable. Another line of research has demonstrated – using rate-based neural networks trained with back-propagation – that having many layers increases the recognition robustness, an approach known as deep learning. We thus designed a deep SNN, comprising several convolutional (trainable with STDP) and pooling layers.We used a temporal coding scheme where the most strongly activated neurons fire first, and less activated neurons fire later or not at all. The network was exposed to natural images. Thanks to STDP, neurons progressively learned features corresponding to prototypical patterns thatwere both salient and frequent. Only a few tens of examples per category were required and no label wa...
The primate visual system has inspired the development of deep artificial neural networks, which ... more The primate visual system has inspired the development of deep artificial neural networks, which have revolutionized the computer vision domain. Yet these networks are much less energy-efficient than their biological counterparts, and they are typically trained with backpropagation, which is extremely data-hungry. To address these limitations, we used a deep convolutional spiking neural network (DCSNN) and a latency-coding scheme. We trained it using a combination of spike-timing-dependent plasticity (STDP) for the lowest layers and reward-modulated STDP (R-STDP) for the highest ones. In short, with R-STDP a correct (resp. incorrect) decision leads to STDP (resp. anti-STDP). This approach led to an accuracy of 97.2% on MNIST, without requiring an external classifier. In addition, we demonstrated that R-STDP extracts features that are diagnostic for the task at hand, and discards the other ones, whereas STDP extracts any feature that repeats. Finally, our approach is biologically pla...
Previous studies have shown that spike-timing-dependent plasticity (STDP) can be used in spiking ... more Previous studies have shown that spike-timing-dependent plasticity (STDP) can be used in spiking neural networks (SNN) to extract visual features of low or intermediate complexity in an unsupervised manner. These studies, however, used relatively shallow architectures, and only one layer was trainable. Another line of research has demonstrated - using rate-based neural networks trained with back-propagation - that having many layers increases the recognition robustness, an approach known as deep learning. We thus designed a deep SNN, comprising several convolutional (trainable with STDP) and pooling layers. We used a temporal coding scheme where the most strongly activated neurons fire first, and less activated neurons fire later or not at all. The network was exposed to natural images. Thanks to STDP, neurons progressively learned features corresponding to prototypical patterns that were both salient and frequent. Only a few tens of examples per category were required and no label ...
The primate visual system has inspired the development of deep artificial neural networks, which ... more The primate visual system has inspired the development of deep artificial neural networks, which have revolutionized the computer vision domain. Yet these networks are much less energy-efficient than their biological counterparts, and they are typically trained with backpropagation, which is extremely data-hungry. To address these limitations, we used a deep convolutional spiking neural network (DCSNN) and a latency-coding scheme. We trained it using a combination of spike-timingdependent plasticity (STDP) for the lower layers and reward-modulated STDP (R-STDP) for the higher ones. In short, with R-STDP a correct (resp. incorrect) decision leads to STDP (resp. anti-STDP). This approach led to an accuracy of 97.2% on MNIST, without requiring an external classifier. In addition, we demonstrated that R-STDP extracts features that are diagnostic for the task at hand, and discards the other ones, whereas STDP extracts any feature that repeats. Finally, our ap-* Corresponding author.
Numerous theories of neural processing, often motivated by experimental observations, have explor... more Numerous theories of neural processing, often motivated by experimental observations, have explored the computational properties of neural codes based on the absolute or relative timing of spikes in spike trains. Spiking neuron models and theories however, as well as their experimental counterparts, have generally been limited to the simulation or observation of isolated neurons, isolated spike trains, or reduced neural populations. Such theories would therefore seem inappropriate to capture the properties of a neural code relying on temporal spike patterns distributed across large neuronal populations. Here we report a range of computer simulations and theoretical considerations that were designed to explore the possibilities of one such code and its relevance for visual processing. In a unified framework where the relation between stimulus saliency and spike relative timing plays the central role, we describe how the ventral stream of the visual system could process natural input scenes and extract meaningful information, both rapidly and reliably. The first wave of spikes generated in the retina in response to a visual stimulation carries information explicitly in its spatio-temporal structure: the most salient information is represented by the first spikes over the population. This spike wave, propagating through a hierarchy of visual areas, is regenerated at each processing stage, where its temporal structure can be modified by (i) the selectivity of the cortical neurons, (ii) lateral interactions and (iii) top-down attentional influences from higher order cortical areas. The resulting model could account for the remarkable efficiency and rapidity of processing observed in the primate visual system.
Accurate binocular depth perception requires information about both stereopsis (relative depth) a... more Accurate binocular depth perception requires information about both stereopsis (relative depth) and distance (absolute depth). It is unclear how these two types of information are integrated in the visual system. In alert, behaving monkeys the responsiveness of a large majority of neurons in the primary visual cortex (area V1) was modulated by the viewing distance. This phenomenon affected particularly disparity-related activity and background activity and was not dependent on the pattern of retinal stimulation. Therefore, extraretinal factors, probably related to ocular vergence or accommodation, or both, can affect processing early in the visual pathway. Such modulations could be useful for (i) judging true distance and (ii) scaling retinal disparity to give information about three-dimensional shape.
Experiments investigating the mechanisms involved in visual processing often fail to separate low... more Experiments investigating the mechanisms involved in visual processing often fail to separate low-level encoding mechanisms from higher-level behaviorally relevant ones. Using an alternating dual-task event-related potential (ERP) experimental paradigm (animals or vehicles categorization) where targets of one task are intermixed among distractors of the other, we show that visual categorization of a natural scene involves different mechanisms with different time courses: a perceptual, task-independent mechanism, followed by a task-related, category-independent process. Although average ERP responses reflect the visual category of the stimulus shortly after visual processing has begun (e.g. 75-80 msec), this difference is not correlated with the subject's behavior until 150 msec poststimulus.
Page 1. 1 BIOLOGICAL CONSTRAINTS ON CONNECTIONIST MODELLING Simon J. Thorpe and Michel Imbert, In... more Page 1. 1 BIOLOGICAL CONSTRAINTS ON CONNECTIONIST MODELLING Simon J. Thorpe and Michel Imbert, Institut des Neurosciences, Departement des Neurosciences de la Vision, Université Pierre et Marie Curie, 9, Quai St. Bernard, 75005, Paris, FRANCE. ABSTRACT ...
To investigate the importance of feedback loops in visual information processing, we have analyze... more To investigate the importance of feedback loops in visual information processing, we have analyzed the dynamic aspects of neuronal responses to oriented gratings in cortical area V1 of the awake primate. If recurrent feedback is important in generating orientation selectivity, the initial part of the neuronal response should be relatively poorly selective, and full orientation selectivity should only appear after a delay. Thus, by examining the dynamics of the neuronal responses it should be possible to assess the importance of feedback processes in the development of orientation selectivity. The results were base on a sample of 259 cells recorded in two monkeys, of which 89% were visually responsive. Of these, approximately two-thirds were orientation selective. Response latency varied considerably between neurons, ranging from a minimum of 41 ms to over 150 ms, although most had latencies of 50–70 ms. Orientation tuning (defined as the bandwidth at half-height) ranged from 16 deg ...
Finding ones way to an unknown destination, navigating complex routes, finding desired inanimate ... more Finding ones way to an unknown destination, navigating complex routes, finding desired inanimate objects; these are all tasks that can be challenging for the visually impaired. The project NAVIG (Navigation Assisted by artificial VIsion and GNSS) is directed towards increasing the autonomy of visually impaired users in known and unknown environments, exterior and interior, large scale and small scale, through a combination of a Global Navigation Satellite System (GNSS) and rapid visual recognition with which the precise position of the user can be determined. Relying on geographical databases and visually identified objects, the user is guided to their desired destination through spatialized audio rendering, always maintained in the head-centered reference frame. This paper presents the overall project design and architecture of the NAVIG system.
In this paper I propose a very radical reform of the taxation system, in which a single flat rate... more In this paper I propose a very radical reform of the taxation system, in which a single flat rate financial transaction tax (FTT) is used to replace the vast majority of existing taxes (including VAT, income tax, taxes on profits...). Existing economic data indicates that a flat rate FTT of 1% would generate far more revenue that is currently generated by all existing taxes, and would allow governments to rapidly repay debts and restore programs of public expenditure as well as allowing resources to be allocated to globally important challenges such as third world development, climate change and health issues.
Previous studies have shown that spike-timing-dependent plasticity (STDP) can be used in spiking ... more Previous studies have shown that spike-timing-dependent plasticity (STDP) can be used in spiking neural networks (SNN) to extract visual features of low or intermediate complexity in an unsupervised manner. These studies, however, used relatively shallow architectures, and only one layer was trainable. Another line of research has demonstrated – using rate-based neural networks trained with back-propagation – that having many layers increases the recognition robustness, an approach known as deep learning. We thus designed a deep SNN, comprising several convolutional (trainable with STDP) and pooling layers.We used a temporal coding scheme where the most strongly activated neurons fire first, and less activated neurons fire later or not at all. The network was exposed to natural images. Thanks to STDP, neurons progressively learned features corresponding to prototypical patterns thatwere both salient and frequent. Only a few tens of examples per category were required and no label wa...
The primate visual system has inspired the development of deep artificial neural networks, which ... more The primate visual system has inspired the development of deep artificial neural networks, which have revolutionized the computer vision domain. Yet these networks are much less energy-efficient than their biological counterparts, and they are typically trained with backpropagation, which is extremely data-hungry. To address these limitations, we used a deep convolutional spiking neural network (DCSNN) and a latency-coding scheme. We trained it using a combination of spike-timing-dependent plasticity (STDP) for the lowest layers and reward-modulated STDP (R-STDP) for the highest ones. In short, with R-STDP a correct (resp. incorrect) decision leads to STDP (resp. anti-STDP). This approach led to an accuracy of 97.2% on MNIST, without requiring an external classifier. In addition, we demonstrated that R-STDP extracts features that are diagnostic for the task at hand, and discards the other ones, whereas STDP extracts any feature that repeats. Finally, our approach is biologically pla...
Uploads
Papers by Simon Thorpe