In silico discovery of representational relationships across visual cortex

Here we provide the code to reproduce all results from the paper:
"In silico discovery of representational relationships across visual cortex".
Alessandro T. Gifford, Maya A. Jastrzębowska, Johannes J.D. Singer, Radoslaw M. Cichy

📄 Paper abstract

Human vision is mediated by a complex interconnected network of cortical brain areas that jointly represent visual information. While these areas are increasingly understood in isolation, their representational relationships remain elusive. Here we developed relational neural control (RNC), and used it to investigate the representational relationships for univariate and multivariate fMRI responses of areas across visual cortex. Through RNC we generated and explored in silico fMRI responses for large amounts of images, discovering controlling images that align or disentangle responses across areas, thus indicating their shared or unique representational content. This revealed a typical network-level configuration of representational relationships in which shared or unique representational content varied based on cortical distance, categorical selectivity, and position within the visual hierarchy. Closing the empirical cycle, we validated the in silico discoveries on in vivo fMRI responses from independent subjects. Together, this reveals how visual areas jointly represent the world as an interconnected network.

🌓 RNC variants

We developed two RNC variants:

• Univariate RNC controls univariate fMRI responses (i.e., responses averaged over all voxels within an ROI), thus exploring representational relationships for visual information encoded in the strongest activation trends common across all ROI voxels.

• Multivariate RNC controls multivariate fMRI responses (i.e., population response of all voxels within a ROI), thus exploring representational relationships for visual information encoded in the multi-voxel response patterns.

🚀 RNC tutorials

We created interactive tutorials where you can implement univariate and multivariate RNC on in silico fMRI responses of 23 areas spanning the entire visual cortex for ~150,000 naturalistic images: 73,000 images from the Natural Scenes Dataset (Allen et al., 2022); 50,000 images from the ImageNet 2012 Challenge (Russakovsky et al., 2015); 26,107 images from the THINGS Database (Hebart et al., 2019).

These tutorials are available on either Google Colab (univariate RNC, multivariate RNC) or Jupyter Notebook (univariate RNC, multivariate RNC).

♻️ Reproducibility

⚙️ Installation

This repository contains code to reproduce all paper's results.

To run the code, you first need to install the libraries in the requirements.txt file within an Anaconda environment. Here, we guide you through the installation steps.

First, create an Anaconda environment with the correct Python version:

conda create -n rnc_env python=3.9

Next, download the requirements.txt file, navigate with your terminal to the download directory, and activate the Anaconda environment previously created with:

source activate rnc_env

Now you can install the libraries with:

pip install -r requirements.txt

Finally, you also need to install the NEST Python package (version 0.3.7) with:

pip install -U git+https://github.com/gifale95/NEST.git@0.3.7

🧰 Data

To run the code you will need to download the following:

• The Neural Encoding Simulation Toolkit (https://github.com/gifale95/NEST).

• The 73,000 stimulus images from the Natural Scenes Dataset (https://naturalscenesdataset.org/).

• The 50,000 ILSVRC-2012 validation images (https://www.image-net.org/challenges/LSVRC/2012/index.php).

• The 26,107 images from THINGS (https://osf.io/jum2f/).

• The in vivo fMRI responses for the controlling images collected in this project (https://openneuro.org/datasets/ds005503).

• The Visual Illusion Reconstruction Dataset (https://figshare.com/articles/dataset/Reconstructing_visual_illusory_experiences_from_human_brain_activity/23590302).

📦 Code description

• 00_generate_insilico_fmri_responses: Generate the in silico fMRI responses later used by RNC.
• 01_in_silico_fmri_encoding_accuracy: Compute the encoding model's prediciton accuracy, and perform a noise analysis on the in silico fMRI responses.
• 02_univariate_rnc: Apply univariate RNC on the in silico fMRI responses.
• 03_generative_univariate_rnc: Apply generative univariate RNC on the in silico fMRI responses.
• 04_multivariate_rnc: Apply multivariate RNC on the in silico fMRI responses.
• 05_multivariate_rnc_retinotopy: Perform the retinotopy analysis on the in silico fMRI resposnes for the V1 vs. V4 multivariate RNC controlling images.
• 06_rnc_categorical_slectivity: Apply the univariate and multivariate RNC categorical selectivity analysis on in silico fMRI responses for high-level visual areas.
• 07_multidimensional_scaling: Apply multidimensional scaling on the in silico fMRI responses for RNC's controlling images.
• 08_in_vivo_validation: Analyze the in vivo fMRI responses for the V1 vs. V4 univariate and multivariate RNC controlling images.

❗ Issues

If you experience problems with the code, please get in touch with Ale (alessandro.gifford@gmail.com), or submit an issue.

📜 Citation

If you use any of our data or code, please cite:

• Gifford AT, Jastrzębowska M, Singer JJD, Cichy RM. 2024. In silico discovery of representational relationships across visual cortex. arXiv preprint, arXiv:2411.10872. DOI: https://doi.org/10.48550/arXiv.2411.10872
• Gifford AT, Bersch D, Roig G, Cichy RM. 2025. The Neural Encoding Simulation Toolkit. In preparation. https://github.com/gifale95/NEST