Summary
+Executing BIDS-Apps leveraging DataLad-controlled datasets +within containers can be tricky. +In particular, one of our general recommendations involves mounting +or binding folders into the container in read-only mode, which +will disallow DataLad from writing to the dataset tree. +Similarly, and depending on the specific runtime settings of the +container framework, DataLad may encounter issues with file ownership too. +This section guides users through ensuring smooth execution of +BIDS-Apps on DataLad/Git-annex-managed datasets.
+Apps may be able to identify if the input dataset is handled with +DataLad or git-annex, and pull down linked data that has not +been fetched yet. +One example of one such application is MRIQC, and all the examples +on this documentation page will refer to it.
+When executing MRIQC within Docker on a DataLad dataset +(for instance, installed from OpenNeuro), +we will need to ensure the following settings are observed:
+Check ReproNim if the suggestions here did not work
+The actions suggested here are expected to work in most circumstances,
+but your system may have specific circumstances that require additional
+or alternative approaches.
+For instance, the ReproNim project maintains
+ReproNim/containers, a
+DataLad dataset with ready-to-use Singularity images for released BIDS Apps, NeuroDesktop applications,
+and other containers.
+Its README.md guides through an approach via that dataset with built-in execution helper taking care about bind-mounts,
+proxying critical Git configuration and potentially executing Singularity images via Docker (e.g., under OSX).
In the particular case of MRIQC, please consider updating (if necessary)
+and fetching the required data before execution and then
+add the --no-datalad-get argument to workaround issues with
+DataLad.
If the execution uid does not match the uid of the user who installed +the DataLad dataset, we will likely encounter the following error +with relatively recent +Git versions (+2.35.2):
+datalad.runner.exception.CommandError: CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false -c annex.merge-annex-branches=false annex find --not --in . --json --json-error-messages -c annex.dotfiles=true -- sub-0001/func/sub-0001_task-restingstate_acq-mb3_bold.nii.gz sub-0002/func/sub-0002_task-emomatching_acq-seq_bold.nii.gz sub-0002/func/sub-0002_task-restingstate_acq-mb3_bold.nii.gz sub-0001/func/sub-0001_task-emomatching_acq-seq_bold.nii.gz sub-0001/func/sub-0001_task-faces_acq-mb3_bold.nii.gz sub-0001/dwi/sub-0001_dwi.nii.gz sub-0002/func/sub-0002_task-workingmemory_acq-seq_bold.nii.gz sub-0001/anat/sub-0001_T1w.nii.gz sub-0002/anat/sub-0002_T1w.nii.gz sub-0001/func/sub-0001_task-gstroop_acq-seq_bold.nii.gz sub-0002/func/sub-0002_task-faces_acq-mb3_bold.nii.gz sub-0002/func/sub-0002_task-anticipation_acq-seq_bold.nii.gz sub-0002/dwi/sub-0002_dwi.nii.gz sub-0001/func/sub-0001_task-anticipation_acq-seq_bold.nii.gz sub-0001/func/sub-0001_task-workingmemory_acq-seq_bold.nii.gz sub-0002/func/sub-0002_task-gstroop_acq-seq_bold.nii.gz' failed with exitcode 1 under /data [info keys: stdout_json] [err: 'git-annex: Git refuses to operate in this repository, probably because it is owned by someone else.
+
+To add an exception for this directory, call:
+git config --global --add safe.directory /data
+
+git-annex: automatic initialization failed due to above problems']
+Confusingly, following the suggestion from DataLad
+(just propagated from Git) of executing
+git config --global --add safe.directory /data will not work in this
+case, because this line must be executed within the container.
+However, containers are transient and the setting this configuration
+on Git will not be propagated between executions unless advanced
+actions are taken (such as mounting a HOME folder with the necessary settings).
Instead, we can override the default user executing within the container
+(which is root, or uid = 0).
+Let's update the last example in the previous
+Docker execution section:
$ docker run -ti --rm \
+ -v $HOME/ds002785:/data:ro \
+ -v $HOME/ds002785/derivatives:/out \
+ -v $HOME/tmp/ds002785-workdir:/work \
+ -u $(id -u):$(id -g) \ # set execution uid:gid
+ nipreps/mriqc:<latest-version> \
+ \
+ /data /out/mriqc-<latest-version> \
+ participant \
+ -w /work
+The above command line will ensure MRIQC to be executed with the current +uid and gid, which will match the filesystem's permissions if the dataset +was installed with the same user.
+Match uid and gid with those corresponding to the user who installed the dataset
+When different users are to install the dataset and
+execute the application, Docker must be executed with the
+uid and gid corresponding to the user who installed the dataset.
+The uid corresponding to a given username (for instance janedoe)
+can be obtained as follows:
getent passwd "janedoe" | cut -f 3 -d ":"
+and her gid:
+getent passwd "janedoe" | cut -f 4 -d ":"
+If the dataset is protected with read-only permissions, then MRIQC
+will hit the following error
+(see nipreps/mriqc#1363):
get(error): sub-0001/func/sub-0001_task-restingstate_acq-mb3_bold.nii.gz (file) [git-annex: .git/annex/tmp: createDirectory: permission denied (Read-only file system)]
+action summary:
+ get (error: 1)
+Traceback (most recent call last):
+ File "/opt/conda/bin/mriqc", line 8, in <module>
+ sys.exit(main())
+ ^^^^^^
+ File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 43, in main
+ parse_args(argv)
+ File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/parser.py", line 658, in parse_args
+ initialize_meta_and_data()
+ File "/opt/conda/lib/python3.11/site-packages/mriqc/utils/misc.py", line 447, in initialize_meta_and_data
+ _datalad_get(dataset)
+ File "/opt/conda/lib/python3.11/site-packages/mriqc/utils/misc.py", line 282, in _datalad_get
+ return get(
+ ^^^^
+ File "/opt/conda/lib/python3.11/site-packages/datalad/interface/base.py", line 773, in eval_func
+ return return_func(*args, **kwargs)
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ File "/opt/conda/lib/python3.11/site-packages/datalad/interface/base.py", line 763, in return_func
+ results = list(results)
+ ^^^^^^^^^^^^^
+ File "/opt/conda/lib/python3.11/site-packages/datalad_next/patches/interface_utils.py", line 287, in _execute_command_
+ raise IncompleteResultsError(
+datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
+[{'action': 'get',
+ 'annexkey': 'MD5E-s76037251--344f061a3165c71e36b98ad1649c3c8c.nii.gz',
+ 'error_message': 'git-annex: .git/annex/tmp: createDirectory: permission '
+ 'denied (Read-only file system)',
+ 'path': '/data/sub-0001/func/sub-0001_task-restingstate_acq-mb3_bold.nii.gz',
+ 'refds': '/data',
+ 'status': 'error',
+ 'type': 'file'}]
+This error indicates that the container is executed with +the appropriate uid and gid pair. +In this case, we will need to ensure DataLad can write +to the dataset installation when obtaining new data. +This is easily achieved by removing the read-only parameters of the +mount option:
+$ docker run -ti --rm \
+ -v $HOME/ds002785:/data \ # mount data WITHOUT :ro
+ -v $HOME/ds002785/derivatives:/out \
+ -v $HOME/tmp/ds002785-workdir:/work \
+ -u $(id -u):$(id -g) \ # set execution uid:gid
+ nipreps/mriqc:<latest-version> \
+ \
+ /data /out/mriqc-<latest-version> \
+ participant \
+ -w /work
+In the case of Singularity and Apptainer, ensuring the uid that +executes the container involves using user namespace mappings. +Therefore, you will need to contact your system administrator to figure +out a convenient solution to the problem.
+Since most of Singularity/Apptainer deployments automatically bind
+the user's $HOME directory, DataLad's suggested direction may
+work:
git config --global --add safe.directory <path-to-dataset-in-host>
+Allowing the container to write on the dataset's tree is straightforward
+and homologous to Docker, by removing the :ro setting in the binding
+option (-B).
Summary
+Here, we describe how to run NiPreps with Docker containers. +To illustrate the process, we will show the execution of fMRIPrep, but these guidelines extend to any other end-user NiPrep.
+Probably, the most popular framework to execute containers is Docker.
+If you are to run a NiPrep on your PC/laptop, this is the RECOMMENDED way of execution.
+Please make sure you follow the
+Docker Engine's installation instructions.
+You can check your installation running their hello-world image:
$ docker run --rm hello-world
+If you have a functional installation, then you should obtain the following output:
+Hello from Docker!
+This message shows that your installation appears to be working correctly.
+
+To generate this message, Docker took the following steps:
+ 1. The Docker client contacted the Docker daemon.
+ 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
+ (amd64)
+ 3. The Docker daemon created a new container from that image which runs the
+ executable that produces the output you are currently reading.
+ 4. The Docker daemon streamed that output to the Docker client, which sent it
+ to your terminal.
+
+To try something more ambitious, you can run an Ubuntu container with:
+ $ docker run -it ubuntu bash
+
+Share images, automate workflows, and more with a free Docker ID:
+ https://hub.docker.com/
+
+For more examples and ideas, visit:
+ https://docs.docker.com/get-started/
+After checking your Docker Engine is capable of running Docker +images, you are ready to pull your first NiPreps container image.
+Troubleshooting
+If you encounter issues while executing a containerized application, +it is critical to identify where the fault is sourced. +For issues emerging from the Docker Engine, please read the +corresponding troubleshooting guidelines. +Once verified the problem is not related to the container system, +then follow the specific application debugging guidelines.
+Fix Docker Desktop startup issue on macOS
+Due to a recent issue +affecting Docker Desktop versions 4.29 through 4.36, +the application may fail to start. +If affected, please follow the +official guidelines to resolve this issue.
+For every new version of the particular NiPreps application that is released, a corresponding Docker image is generated. +The Docker image becomes a container when the execution engine loads the image and adds an extra layer that makes it runnable. In order to run NiPreps' Docker images, the Docker Engine must be installed.
+ + +Taking fMRIPrep to illustrate the usage, first you might want to make sure of the exact version of the tool to be used:
+$ docker pull nipreps/fmriprep:<latest-version>
+You can run NiPreps interacting directly with the Docker Engine via the docker run interface.
Some NiPreps include a lightweight wrapper script for convenience.
+That is the case of fMRIPrep and its fmriprep-docker wrapper.
+Before starting, make sure you have the wrapper installed.
+When you run fmriprep-docker, it will generate a Docker command line for you, print it out for reporting purposes, and then execute it without further action needed, e.g.:
$ fmriprep-docker /path/to/data/dir /path/to/output/dir participant
+RUNNING: docker run --rm -it -v /path/to/data/dir:/data:ro \
+ -v /path/to_output/dir:/out nipreps/fmriprep:20.2.2 \
+ /data /out participant
+...
+fmriprep-docker implements the unified command-line interface of BIDS Apps, and automatically translates directories into Docker mount points for you.
We have published a step-by-step tutorial illustrating how to run fmriprep-docker.
+This tutorial also provides valuable troubleshooting insights and advice on what to do after
+fMRIPrep has run.
If you need a finer control over the container execution, or you feel comfortable with the Docker Engine, avoiding the extra software layer of the wrapper might be a good decision.
+Containers are confined in a sandbox, so they can't access the data on the host
+unless explicitly enabled.
+The Docker Engine provides mounting filesystems into the container with the -v argument and the following syntax:
+-v some/path/in/host:/absolute/path/within/container:ro,
+where the trailing :ro specifies that the mount is read-only.
+The mount permissions modifiers can be omitted, which means the mount
+will have read-write permissions.
Docker for Windows requires enabling Shared Drives
+On Windows installations, the -v argument will not work
+by default because it is necessary to enable shared drives.
+Please check on this Stackoverflow post how to enable them.
In general, you'll want to at least provide two mount-points: +one set in read-only mode for the input data and one read/write +to store the outputs:
+$ docker run -ti --rm \
+ -v path/to/data:/data:ro \ # read-only, for data
+ -v path/to/output:/out \ # read-write, for outputs
+ nipreps/fmriprep:<latest-version> \
+ /data /out/out \
+ participant
+We recommend mounting a work (or scratch) directory for intermediate workflow results.
+This is particularly useful for debugging or reusing pre-cached intermediate results,
+but can also be useful to control where these (large) directories get created,
+as the default location for files created inside a docker container may not have sufficient space.
+In the case of NiPreps, we typically inform the BIDS Apps
+to override the work directory by setting the -w/--work-dir
+argument (please note that this is not defined by the BIDS Apps
+specifications and it may change across applications):
$ docker run -ti --rm \
+ -v path/to/data:/data:ro \
+ -v path/to/output:/out \
+ -v path/to/work:/work \ # mount from host
+ nipreps/fmriprep:<latest-version> \
+ /data /out/out \
+ participant
+ -w /work # override default directory
+Best practices
+The ReproNim initiative +distributes materials and documentation of best practices +for containerized execution of neuroimaging workflows. +Most of these are organized within the +YODA (Yoda's Organigram on Data Analysis) principles.
+For example, mounting $PWD into $PWD and setting that path
+as current working directory can effectively resolve many issues.
+This strategy may be combined with the above suggestion about
+the application's work directory as follows:
$ docker run -ti --rm \
+ -v $PWD:$PWD \ # Mount the current directory with its own name
+ -w $PWD \ # DO NOT confuse with the application's work directory
+ nipreps/fmriprep:<latest-version> \
+ inputs/raw-data outputs/fmriprep \ # With YODA, the inputs are inside the working directory
+ participant
+ -w $PWD/work
+Mounting $PWD may be used with YODA so that all necessary parts
+in execution are reachable from under $PWD.
+This effectively
+(i) makes it easy to transfer configurations from
+outside the container to the inside execution runtime;
+(ii) the outside/inside filesystem trees are homologous, which
+makes post-processing and orchestration easier;
+(iii) execution in shared systems is easier as everything is
+sort of self-contained.
In addition to mounting $PWD, other advanced practices
+include mounting specific configuration files (for example, a
+Nipype configuration file)
+into the appropriate paths within the container.
BIDS Apps relying on TemplateFlow +for atlas and template management may require +the TemplateFlow Archive be mounted from the host. +Mounting the Archive from the host is an effective way +to prevent multiple downloads of templates that are not bundled in the image:
+$ docker run -ti --rm \
+ -v path/to/data:/data:ro \
+ -v path/to/output:/out \
+ -v path/to/work:/work \
+ -v path/to/tf-cache:/opt/templateflow \ # mount from host
+ -e TEMPLATEFLOW_HOME=/opt/templateflow \ # override TF home
+ nipreps/fmriprep:<latest-version> \
+ /data /out/out \
+ participant
+ -w /work
+Sharing the TemplateFlow cache can cause race conditions in parallel execution
+When sharing the TemplateFlow HOME folder across several parallel +executions against a single filesystem, these instance will likely +attempt to fetch unavailable templates without sufficient time between +actions for the data to be fully downloaded (in other words, +data downloads will be racing each other).
+To resolve this issue, you will need to make sure all necessary
+templates are already downloaded within the cache folder.
+If the TemplateFlow Client is properly installed in your system,
+this is possible with the following command line
+(example shows how to fully download MNI152NLin2009cAsym:
$ templateflow get MNI152NLin2009cAsym
+By default, Docker will run the container with the +user id (uid) 0, which is reserved for the default root +account in Linux. +In other words, by default Docker will use the superuser account +to execute the container and will write files with the corresponding +uid=0 unless configured otherwise. +Executing as superuser may result in permissions and security issues, +for example, with DataLad (discussed later). +One paramount example of permissions issues where beginners typically +run into is deleting files after a containerized execution. +If the uid is not overridden, the outputs of a containerized execution +will be owned by root and group root. +Therefore, normal users will not be able to modify the output and +superuser permissions will be required to delete data generated +by the containerized application. +Some shared systems only allow running containers as a normal user +because the user will not be able to operate on the outputs otherwise.
+Whether the container is available with default settings,
+or the execution has been customized to normal users,
+running as a normal user avoids these permissions issues.
+This can be achieved with
+Docker's -u/--user option:
--user=[ user | user:group | uid | uid:gid | user:gid | uid:group ]
+We can combine this option with Bash's id command to ensure the current user's uid and group id (gid) are being set:
$ docker run -ti --rm \
+ -v path/to/data:/data:ro \
+ -v path/to/output:/out \
+ -u $(id -u):$(id -g) \ # set execution uid:gid
+ -v path/to/tf-cache:/opt/templateflow \ # mount from host
+ -e TEMPLATEFLOW_HOME=/opt/templateflow \ # override TF home
+ nipreps/fmriprep:<latest-version> \
+ /data /out/out \
+ participant
+For example:
+$ docker run -ti --rm \
+ -v $HOME/ds005:/data:ro \
+ -v $HOME/ds005/derivatives:/out \
+ -v $HOME/tmp/ds005-workdir:/work \
+ -u $(id -u):$(id -g) \
+ -v $HOME/.cache/templateflow:/opt/templateflow \
+ -e TEMPLATEFLOW_HOME=/opt/templateflow \
+ nipreps/fmriprep:<latest-version> \
+ /data /out/fmriprep-<latest-version> \
+ participant \
+ -w /work
+Once the Docker Engine arguments are written, the remainder of the +command line follows the interface defined by the specific +BIDS App (for instance, +fMRIPrep +or MRIQC).
+The first section of a call consists of arguments specific to Docker, +which configure the execution of the container:
+$ docker run -ti --rm \
+ -v $HOME/ds005:/data:ro \
+ -v $HOME/ds005/derivatives:/out \
+ -v $HOME/tmp/ds005-workdir:/work \
+ -u $(id -u):$(id -g) \
+ -v $HOME/.cache/templateflow:/opt/templateflow \
+ -e TEMPLATEFLOW_HOME=/opt/templateflow \
+ nipreps/fmriprep:<latest-version> \
+ /data /out/fmriprep-<latest-version> \
+ participant \
+ -w /work
+Then, we specify the container image that we execute:
+$ docker run -ti --rm \
+ -v $HOME/ds005:/data:ro \
+ -v $HOME/ds005/derivatives:/out \
+ -v $HOME/tmp/ds005-workdir:/work \
+ -u $(id -u):$(id -g) \
+ -v $HOME/.cache/templateflow:/opt/templateflow \
+ -e TEMPLATEFLOW_HOME=/opt/templateflow \
+ nipreps/fmriprep:<latest-version> \
+ /data /out/fmriprep-<latest-version> \
+ participant \
+ -w /work
+Finally, the application-specific options can be added.
+We already described the work directory setting before, in the case
+of NiPreps such as MRIQC and fMRIPrep.
+Some options are BIDS Apps standard, such as
+the analysis level (participant or group)
+and specific participant identifier(s) (--participant-label):
$ docker run -ti --rm \
+ -v $HOME/ds005:/data:ro \
+ -v $HOME/ds005/derivatives:/out \
+ -v $HOME/tmp/ds005-workdir:/work \
+ -u $(id -u):$(id -g) \
+ -v $HOME/.cache/templateflow:/opt/templateflow \
+ -e TEMPLATEFLOW_HOME=/opt/templateflow \
+ nipreps/fmriprep:<latest-version> \
+ /data /out/fmriprep-<latest-version> \
+ participant \
+ --participant-label 001 002 \
+ -w /work
+Docker may be executed with limited resources. +Please read the documentation +to limit resources such as memory, memory policies, number of CPUs, etc.
+Memory will be a common culprit when working with large datasets +(+10GB). +However, the Docker Engine is limited to 2GB of RAM by default +for some installations of Docker for MacOSX and Windows. +The general resource settings can be also modified through the Docker Desktop +graphical user interface. +On a shell, the memory limit can be overridden with:
+$ service docker stop
+$ dockerd --storage-opt dm.basesize=30G
+The Brain Imaging Data Structure (BIDS) is a standard for organizing and describing +brain datasets, including MRI. +The common naming convention and folder structure allow researchers to easily reuse BIDS datasets, re-apply analysis protocols, and run standardized automatic data preprocessing pipelines (and particularly, BIDS Apps). +The BIDS starter-kit contains a wide collection of educational resources. +Validity of the structure can be assessed with the online BIDS-Validator. +The tree of a typical, valid (BIDS-compliant) dataset is shown below:
+ds000003/
+ ├─ CHANGES
+ ├─ dataset_description.json
+ ├─ participants.tsv
+ ├─ README
+ ├─ sub-01/
+ │ ├─ anat/
+ │ │ ├─ sub-01_inplaneT2.nii.gz
+ │ │ └─ sub-01_T1w.nii.gz
+ │ └─ func/
+ │ ├─ sub-01_task-rhymejudgment_bold.nii.gz
+ │ └─ sub-01_task-rhymejudgment_events.tsv
+ ├─ sub-02/
+ ├─ sub-03/
+(Taken from the BIDS Apps paper)
+A BIDS App is a container image capturing a neuroimaging pipeline that takes a BIDS-formatted dataset as input. +Since the input is a whole dataset, apps are able to combine multiple modalities, sessions, and/or subjects, but at the same time need to implement ways to query input datasets. +Each BIDS App has the same core set of command-line arguments, making them easy to run and integrate into automated platforms. +BIDS Apps are constructed in a way that does not depend on any software outside of the container image other than the container engine.
+BIDS Apps rely upon two technologies for container computing:
+BIDS Apps are deposited in the Docker Hub repository, making them openly accessible. Each app is versioned and all of the historical versions are available to download. By reporting the BIDS App name and version in a manuscript, authors can provide others with the ability to exactly replicate their analysis workflow.
+Docker is used for its excellent documentation, maturity, and the Docker Hub service for storage and distribution of the images. +Docker containers are easily run on personal computers and cloud services. +However, the Docker Runtime was originally designed to run different components of web services (HTTP servers, databases etc.) using cloud resources. +Docker thus requires root or root-like permissions, as well as modern versions of Linux kernel (to perform user mapping and management of network resources); though this is not a problem in context of renting cloud resources (which are not shared with other users), it makes it difficult or impossible to use in a multi-tenant environment such as an HPC system, which is often the most cost-effective computational resource available to researchers.
+Singularity, on the other hand, is a unique container technology designed from the ground up with the encapsulation of binary dependencies and HPC use in mind. +Its main advantage over Docker is that it does not require root access for container execution and thus is safe to use on multi-tenant systems. +In addition, it does not require recent Linux kernel functionalities (such as namespaces, cgroups and capabilities), making it easy to install on legacy systems.
+BIDS Apps decouple the individual level analysis (processing of independent subjects) from group-level analyses aggregating participants. +For the analysis of individual subjects, Apps need to understand the BIDS structure of the input dataset, so that the required inputs for the designated subject are found. +Apps are designed to easily process derivatives generated by the participant-level or other Apps. +The overall workflow has an entry-point and an end-point responsible of setting-up the map-reduce tasks and the tear-down including organizing the outputs for its archiving, respectively. +Each App may implement multiple map and reduce steps.
+
To improve user experience and ability to integrate BIDS Apps into various computational platforms, each App follows a set of core command-line arguments:
+$ <entrypoint> <bids_dataset> <output_path> <analysis_level>
+For instance, to run fMRIPrep on a dataset located in /data/bids_root and write the outputs to /data/bids_root/derivatives/:
$ fmriprep /data/bids_root /data/bids_root/derivatives/ participant
+In this case, we have selected to run the participant level (to process individual subjects).
+fMRIPrep does not have a group level, but other BIDS Apps may have.
+For instance, MRIQC generates group-level reports with the following command-line:
$ mriqc /data/bids_root /data/bids_root/derivatives/ group
+NiPreps generate derivatives of the original data, and they fulfill the BIDS specification for the results of Apps that are created for subsequent consumption by other BIDS-Apps. +These derivatives must follow the BIDS Derivatives specification (draft). +An example of BIDS Derivatives filesystem tree, generated with fMRIPrep 1.5: +
derivatives/
+├── fmriprep/
+│ ├── dataset_description.json
+│ ├── logs
+│ ├── sub-01.html
+│ ├── sub-01/
+│ │ ├── anat/
+│ │ │ ├── sub-01_desc-brain_mask.nii.gz
+│ │ │ ├── sub-01_dseg.nii.gz
+│ │ │ ├── sub-01_label-GM_probseg.nii.gz
+│ │ │ ├── sub-01_label-WM_probseg.nii.gz
+│ │ │ ├── sub-01_label-CSF_probseg.nii.gz
+│ │ │ ├── sub-01_desc-preproc_T1w.nii.gz
+│ │ │ ├── sub-01_space-MNI152_desc-brain_mask.nii.gz
+│ │ │ ├── sub-01_space-MNI152_dseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_label-GM_probseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_label-WM_probseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_label-CSF_probseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_desc-preproc_T1w.nii.gz
+│ │ │ ├── sub-01_from-MNI152_to-T1w_mode-image_xfm.h5
+│ │ │ ├── sub-01_from-T1w_to-MNI152_mode-image_xfm.h5
+│ │ │ └── sub-01_from-orig_to-T1w_mode-image_xfm.txt
+│ │ ├── figures/
+│ │ └── func/
+│ │ ├── sub-01_task-rhymejudgment_space-MNI152_boldref.nii.gz
+│ │ ├── sub-01_task-rhymejudgment_space-MNI152_desc-preproc_bold.nii.gz
+│ │ ├── sub-01_task-rhymejudgment_space-MNI152_desc-confounds_regressors.nii.gz
+│ │ └── sub-01_task-rhymejudgment_space-MNI152_desc-brain_mask.nii.gz
+│ ├── sub-02.html
+│ ├── sub-02/
+│ ├── sub-03.html
+│ └── sub-03/
+Summary
+Here, we describe how to run NiPreps with Singularity containers. +To illustrate the process, we will show the execution of fMRIPrep, but these guidelines extend to any other end-user NiPrep.
+Apptainer
+In 2021, Singularity was rebranded as Apptainer when the project was transferred to the Linux Foundation.
+As noted in the community announcement, all the commands below that contain singularity as the command line executable will execute Apptainer and can be replaced with the apptainer command with no change in function.
Singularity version >= 2.5: +If the version of Singularity installed on your HPC (High-Performance Computing) +system is modern enough you can create Singularity image +directly on the system. This is as simple as:
+$ singularity build /my_images/fmriprep-<version>.simg \
+ docker://nipreps/fmriprep:<version>
+where <version> should be replaced with the desired version of
+fMRIPrep that you want to download.
Singularity version < 2.5: +In this case, start with a machine (e.g., your personal computer) with +Docker installed. Use +docker2singularity +to create a singularity image. You will need an active internet +connection and some time:
+$ docker run --privileged -t --rm \
+ -v /var/run/docker.sock:/var/run/docker.sock \
+ -v D:\host\path\where\to\output\singularity\image:/output \
+ singularityware/docker2singularity \
+ nipreps/fmriprep:<version>
+Where <version> should be replaced with the desired version of
+fMRIPrep that you want to download.
Beware of the back slashes, expected for Windows systems. For *nix +users the command translates as follows: :
+$ docker run --privileged -t --rm \
+ -v /var/run/docker.sock:/var/run/docker.sock \
+ -v /absolute/path/to/output/folder:/output \
+ singularityware/docker2singularity \
+ nipreps/fmriprep:<version>
+Transfer the resulting Singularity image to the HPC, for example, using
+scp or rsync:
$ scp nipreps_fmriprep*.img user@hcpserver.edu:/my_images
+If the data to be preprocessed is also on the HPC, you are ready to run +the NiPrep:
+$ singularity run --cleanenv fmriprep.simg \
+ path/to/data/dir path/to/output/dir \
+ participant \
+ --participant-label label
+Singularity by default exposes all environment variables from the host inside the container.
+Because of this, your host libraries (e.g., NiPype or a Python environment) could be accidentally used instead of the ones inside the
+container. To avoid such a situation, we strongly recommend using the
+--cleanenv argument in all scenarios. For example:
$ singularity run --cleanenv fmriprep.simg \
+ /work/04168/asdf/lonestar/ $WORK/lonestar/output \
+ participant \
+ --participant-label 387 --nthreads 16 -w $WORK/lonestar/work \
+ --omp-nthreads 16
+Alternatively, conflicts might be preempted and some problems mitigated
+by unsetting potentially problematic settings, such as the PYTHONPATH
+variable, before running:
$ unset PYTHONPATH; singularity run fmriprep.simg \
+ /work/04168/asdf/lonestar/ $WORK/lonestar/output \
+ participant \
+ --participant-label 387 --nthreads 16 -w $WORK/lonestar/work \
+ --omp-nthreads 16
+It is possible to define environment variables scoped within the
+container by using the SINGULARITYENV_* magic, in combination with
+--cleanenv. For example, we can set the FreeSurfer license variable
+(see fMRIPrep's documentation on this) as follows: :
$ export SINGULARITYENV_FS_LICENSE=$HOME/.freesurfer.txt
+$ singularity exec --cleanenv fmriprep.simg env | grep FS_LICENSE
+FS_LICENSE=/home/users/oesteban/.freesurfer.txt
+As we can see, the export in the first line tells Singularity to set a
+corresponding environment variable of the same name after dropping the
+prefix SINGULARITYENV_.
Depending on how Singularity is configured on your cluster it might or
+might not automatically bind (mount or expose) host's folders to the
+container (e.g., /scratch, or $HOME). This is particularly relevant
+because, if you can't run Singularity in privileged mode (which is
+almost certainly true in all the scenarios), Singularity containers
+are read only. This is to say that you won't be able to write
+anything unless Singularity can access the host's filesystem in write
+mode.
By default, Singularity automatically binds (mounts) the user's home
+directory and a scratch directory. In addition, Singularity generally
+allows binding the necessary folders with the
+-B <host_folder>:<container_folder>[:<permissions>] Singularity
+argument. For example:
$ singularity run --cleanenv -B /work:/work fmriprep.simg \
+ /work/my_dataset/ /work/my_dataset/derivatives/fmriprep \
+ participant \
+ --participant-label 387 --nthreads 16 \
+ --omp-nthreads 16
+Warning
+If your Singularity installation doesn't allow you to bind non-existent
+bind points, you'll get an error saying
+WARNING: Skipping user bind, non existent bind point (directory) in container.
+In this scenario, you can either try to bind things onto some other bind
+point you know it exists in the image or rebuild your singularity image
+with docker2singularity as follows:
$ docker run --privileged -ti --rm -v /var/run/docker.sock:/var/run/docker.sock \
+ -v $PWD:/output singularityware/docker2singularity \
+ -m "/gpfs /scratch /work /share /lscratch /opt/templateflow"
+In the example above, the following bind points are created: /gpfs,
+/scratch, /work, /share, /opt/templateflow.
Important
+One great feature of containers is their confinement or isolation from
+the host system. Binding mount points breaks this principle, as the
+container has now access to create changes in the host. Therefore, it is
+generally recommended to use binding scarcely and granting very limited
+access to the minimum necessary resources. In other words, it is
+preferred to bind just one subdirectory of $HOME than the full $HOME
+directory of the host (see nipreps/fmriprep#1778 (comment)).
Relevant aspects of the $HOME directory within the container:
+By default, Singularity will bind the user's $HOME directory in the
+host into the /home/$USER (or equivalent) in the container. Most of
+the times, it will also redefine the $HOME environment variable and
+update it to point to the corresponding mount point in /home/$USER.
+However, these defaults can be overwritten in your system. It is
+recommended to check your settings with your system's administrators.
+If your Singularity installation allows it, you can workaround the
+$HOME specification combining the bind mounts argument (-B) with the
+home overwrite argument (--home) as follows:
$ singularity run -B $HOME:/home/fmriprep --home /home/fmriprep \
+ --cleanenv fmriprep.simg <fmriprep arguments>
+TemplateFlow is a helper tool that allows neuroimaging workflows to programmatically access a repository of standard neuroimaging templates. +In other words, TemplateFlow allows NiPreps to dynamically change +the templates that are used, e.g., in the atlas-based brain extraction +step or spatial normalization.
+Default settings in the Singularity image should get along with the
+Singularity installation of your system. However, deviations from the
+default configurations of your installation may break this
+compatibility. A particularly problematic case arises when the home
+directory is mounted in the container, but the $HOME environment
+variable is not correspondingly updated. Typically, you will experience
+errors like OSError: [Errno 30] Read-only file system or
+FileNotFoundError: [Errno 2] No such file or directory: '/home/fmriprep/.cache'.
If it is not explicitly forbidden in your installation, the first
+attempt to overcome this issue is manually setting the $HOME directory
+as follows:
$ singularity run --home $HOME --cleanenv fmriprep.simg <fmriprep arguments>
+If the user's home directory is not automatically bound, then the +second step would include manually binding it as in the section above: :
+$ singularity run -B $HOME:/home/fmriprep --home /home/fmriprep \
+ --cleanenv fmriprep.simg <fmriprep arguments>
+Finally, if the --home argument cannot be used, you'll need to
+provide the container with writable filesystems where TemplateFlow's
+files can be downloaded. In addition, you will need to indicate
+fMRIPrep to update the default paths with the new mount points setting
+the SINGULARITYENV_TEMPLATEFLOW_HOME variable. :
# Tell the NiPrep where TemplateFlow will place downloads
+$ export SINGULARITYENV_TEMPLATEFLOW_HOME=/opt/templateflow
+$ singularity run -B <writable-path-on-host>:/opt/templateflow \
+ --cleanenv fmriprep.simg <fmriprep arguments>
+We have identified several conditions in which running NiPreps might +fail because of spotty or impossible access to Internet.
+If your compute node cannot have access to Internet, then you'll need +to pull down from TemplateFlow all the resources that will be necessary +ahead of run-time.
+If that is not the case (i.e., you should be able to hit HTTP/s +endpoints), then you can try the following:
+VerifiedHTTPSConnection ... Failed to establish a new connection: [Errno 110] Connection timed out.
+ If you encounter an error like this, probably you'll need to set up an
+ http proxy exporting SINGULARITYENV_http_proxy (see nipreps/fmriprep#1778 (comment).
+ For example:
$ export SINGULARITYENV_https_proxy=http://<ip or proxy name>:<port>
+requests.exceptions.SSLError: HTTPSConnectionPool .... In this case,
+ your container seems to be able to reach the Internet, but unable to use
+ SSL encryption. There are two potential solutions to the issue. The
+ recommended one
+ is setting REQUESTS_CA_BUNDLE to the appropriate path, and/or binding
+ the appropriate filesystem:
$ export SINGULARITYENV_REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
+$ singularity run -B <path-to-certs-folder>:/etc/ssl/certs \
+ --cleanenv fmriprep.simg <fmriprep arguments>
+$ export TEMPLATEFLOW_HOME=/path/to/keep/templateflow
+$ python -m pip install -U templateflow # Install the client
+$ python
+>>> import templateflow.api
+>>> templateflow.api.TF_S3_ROOT = 'http://templateflow.s3.amazonaws.com'
+>>> api.get(‘MNI152NLin6Asym’)
+Finally, run the singularity image binding the appropriate folder:
+$ export SINGULARITYENV_TEMPLATEFLOW_HOME=/templateflow
+$ singularity run -B ${TEMPLATEFLOW_HOME:-$HOME/.cache/templateflow}:/templateflow \
+ --cleanenv fmriprep.simg <fmriprep arguments>
+When running multiple instances of a NiPreps on HPC, you may encounter
+errors like nipreps/mriqc#1170:
OSError: [Errno 98] Address already in use
+To solve this issue, you can try to isolate the container network from
+the host network by using the --network none setting of Singularity/Apptainer:
apptainer run --net --network none ...
+This solution prevents the container from accessing the Internet and from +downloading data, for example, templates. +Please follow the previous section's guidelines to pre-fetch templates +and then making them accessible.
+Setting up a functional execution framework with Singularity might be
+tricky in some HPC (high-performance computing) systems.
+Please make sure you have read the relevant
+documentation of Singularity, and checked all the
+defaults and configuration in your system. The next step is checking the
+environment and access to fMRIPrep resources, using
+singularity shell.
Check access to input data folder, and BIDS validity: +
$ singularity shell -B path/to/data:/data fmriprep.simg
+Singularity fmriprep.simg:~> ls /data
+CHANGES README dataset_description.json participants.tsv sub-01 sub-02 sub-03 sub-04 sub-05 sub-06 sub-07 sub-08 sub-09 sub-10 sub-11 sub-12 sub-13 sub-14 sub-15 sub-16 task-balloonanalogrisktask_bold.json
+Singularity fmriprep.simg:~> bids-validator /data
+ 1: [WARN] You should define 'SliceTiming' for this file. If you don't provide this information slice time correction will not be possible. (code: 13 - SLICE_TIMING_NOT_DEFINED)
+ ./sub-01/func/sub-01_task-balloonanalogrisktask_run-01_bold.nii.gz
+ ./sub-01/func/sub-01_task-balloonanalogrisktask_run-02_bold.nii.gz
+ ./sub-01/func/sub-01_task-balloonanalogrisktask_run-03_bold.nii.gz
+ ./sub-02/func/sub-02_task-balloonanalogrisktask_run-01_bold.nii.gz
+ ./sub-02/func/sub-02_task-balloonanalogrisktask_run-02_bold.nii.gz
+ ./sub-02/func/sub-02_task-balloonanalogrisktask_run-03_bold.nii.gz
+ ./sub-03/func/sub-03_task-balloonanalogrisktask_run-01_bold.nii.gz
+ ./sub-03/func/sub-03_task-balloonanalogrisktask_run-02_bold.nii.gz
+ ./sub-03/func/sub-03_task-balloonanalogrisktask_run-03_bold.nii.gz
+ ./sub-04/func/sub-04_task-balloonanalogrisktask_run-01_bold.nii.gz
+ ... and 38 more files having this issue (Use --verbose to see them all).
+ Please visit https://neurostars.org/search?q=SLICE_TIMING_NOT_DEFINED for existing conversations about this issue.
+Check access to output data folder, and whether you have write + permissions +
$ singularity shell -B path/to/data/derivatives/fmriprep-1.5.0:/out fmriprep.simg
+Singularity fmriprep.simg:~> ls /out
+Singularity fmriprep.simg:~> touch /out/test
+Singularity fmriprep.simg:~> rm /out/test
+Check access and permissions to $HOME:
+
$ singularity shell fmriprep.simg
+Singularity fmriprep.simg:~> mkdir -p $HOME/.cache/testfolder
+Singularity fmriprep.simg:~> rmdir $HOME/.cache/testfolder
+Check TemplateFlow operation: +
$ singularity shell -B path/to/templateflow:/templateflow fmriprep.simg
+Singularity fmriprep.simg:~> echo ${TEMPLATEFLOW_HOME:-$HOME/.cache/templateflow}
+/home/users/oesteban/.cache/templateflow
+Singularity fmriprep.simg:~> python -c "from templateflow.api import get; get(['MNI152NLin2009cAsym', 'MNI152NLin6Asym', 'OASIS30ANTs', 'MNIPediatricAsym', 'MNIInfant'])"
+ Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-01_atlas-HOCPA_desc-th0_dseg.nii.gz
+ 304B [00:00, 1.28kB/s]
+ Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-01_atlas-HOCPA_desc-th25_dseg.nii.gz
+ 261B [00:00, 1.04kB/s]
+ Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-01_atlas-HOCPA_desc-th50_dseg.nii.gz
+ 219B [00:00, 867B/s]
+ ...
+An example of sbatch script to run fMRIPrep on a SLURM system1 is
+given below. The submission script will
+generate one task per subject using a job array.
+
#!/bin/bash
+#
+#SBATCH -J fmriprep
+#SBATCH --time=48:00:00
+#SBATCH -n 1
+#SBATCH --cpus-per-task=16
+#SBATCH --mem-per-cpu=4G
+#SBATCH -p normal,mygroup # Queue names you can submit to
+# Outputs ----------------------------------
+#SBATCH -o log/%x-%A-%a.out
+#SBATCH -e log/%x-%A-%a.err
+#SBATCH --mail-user=%u@domain.tld
+#SBATCH --mail-type=ALL
+# ------------------------------------------
+
+BIDS_DIR="$STUDY/data"
+DERIVS_DIR="derivatives/fmriprep-20.2.2"
+LOCAL_FREESURFER_DIR="$STUDY/data/derivatives/freesurfer-6.0.1"
+
+# Prepare some writeable bind-mount points.
+TEMPLATEFLOW_HOST_HOME=$HOME/.cache/templateflow
+FMRIPREP_HOST_CACHE=$HOME/.cache/fmriprep
+mkdir -p ${TEMPLATEFLOW_HOST_HOME}
+mkdir -p ${FMRIPREP_HOST_CACHE}
+
+# Prepare derivatives folder
+mkdir -p ${BIDS_DIR}/${DERIVS_DIR}
+
+# Make sure FS_LICENSE is defined in the container.
+export SINGULARITYENV_FS_LICENSE=$HOME/.freesurfer.txt
+
+# Designate a templateflow bind-mount point
+export SINGULARITYENV_TEMPLATEFLOW_HOME="/templateflow"
+SINGULARITY_CMD="singularity run --cleanenv -B $BIDS_DIR:/data -B ${TEMPLATEFLOW_HOST_HOME}:${SINGULARITYENV_TEMPLATEFLOW_HOME} -B $L_SCRATCH:/work -B ${LOCAL_FREESURFER_DIR}:/fsdir $STUDY/images/fmriprep_20.2.2.simg"
+
+# Parse the participants.tsv file and extract one subject ID from the line corresponding to this SLURM task.
+subject=$( sed -n -E "$((${SLURM_ARRAY_TASK_ID} + 1))s/sub-(\S*)\>.*/\1/gp" ${BIDS_DIR}/participants.tsv )
+
+# Remove IsRunning files from FreeSurfer
+find ${LOCAL_FREESURFER_DIR}/sub-$subject/ -name "*IsRunning*" -type f -delete
+
+# Compose the command line
+cmd="${SINGULARITY_CMD} /data /data/${DERIVS_DIR} participant --participant-label $subject -w /work/ -vv --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --output-spaces MNI152NLin2009cAsym:res-2 anat fsnative fsaverage5 --use-aroma --fs-subjects-dir /fsdir"
+
+# Setup done, run the command
+echo Running task ${SLURM_ARRAY_TASK_ID}
+echo Commandline: $cmd
+eval $cmd
+exitcode=$?
+
+# Output results to a table
+echo "sub-$subject ${SLURM_ARRAY_TASK_ID} $exitcode" \
+ >> ${SLURM_JOB_NAME}.${SLURM_ARRAY_JOB_ID}.tsv
+echo Finished tasks ${SLURM_ARRAY_TASK_ID} with exit code $exitcode
+exit $exitcode
+$ export STUDY=/path/to/some/folder
+$ sbatch --array=1-$(( $( wc -l $STUDY/data/participants.tsv | cut -f1 -d' ' ) - 1 )) fmriprep.slurm
+assuming that job arrays and Singularity are available ↩
+layout: false +count: false
+.middle.center[
+
+
]
+layout: false +count: false
+.middle.center[
+
+
]
+???
+name: newsection +layout: true +class: section-separator
+.perma-sidebar[
+
+
+
+
+
+
+
+
+
]
+name: sidebar +layout: true
+.perma-sidebar[
+
+
+
+
+
+
+
+
+
]
+template: sidebar
+container technology, CI/CD
+a wealth of prior knowledge (esp. about humans)
+LOTS of data acquired everyday
+although many neuroimaging areas are still in search of methodological breakthroughs,
+challenges have moved on to the workflows:
+???
+
+
+
(Botvinik-Nezer et al., 2020)
+Around 50% of teams used fMRIPrep'ed inputs.
+???
+fMRIPrep takes in a task-based or resting-state +functional MRI dataset in BIDS-format +and returns preprocessed data ready for analysis.
+Preprocessed data can be used for a broad range of analysis, and they are +formatted following BIDS-Derivatives to maximize compatibility with: + * major software packages (AFNI, FSL, SPM*, etc.) + * further temporal filtering and denoising: fMRIDenoise + * any BIDS-Derivatives compliant tool (e.g., FitLins).
+--
+???
+fMRIPrep adopts the BIDS-App specifications. +That means the software is tested with every change to the codebase, +it also means that packaging, containerization, and deployment are also +automated and require tests to be passing. +BIDS-Apps are inter-operable (via BIDS-Derivatives), +and optimized for execution in HPC, Cloud, etc.
+--
+???
+fMRIPrep minimizes human intervention because the user does not +need to fiddle with any parameters - they are obtained from the BIDS structure. +However, fMRIPrep does allow some flexibility to ensure the preprocessing meets the requirements of the intended analyses.
+(we just wanted a robust tool to automatically preprocess incoming data of OpenNeuro.org)
+--
+--
+.pull-left[
+Preprocessing of fMRI was in need for division of labor.
+Obsession with transparency made early-adopters confident of the recipes they were applying.
+Responsiveness to feedback. +]
+.pull-right[
+
+
+
]
+???
+Preprocessing is a time-consuming effort, requires expertise converging imaging foundations & CS, typically addressed with legacy in-house pipelines.
+On the right-hand side, you'll find the chart of unique visitors +to fmriprep.org, which is the documentation website.
+
+
+
--
+(probably not preprocessing...)
+???
+With the development of fMRIPrep we understood that +researchers don't want to waste their time on preprocessing +(except for researchers developing new preprocessing techniques).
+--
+(research programs just can't cover the neuroscience and the engineering of the whole workflow - we need to divide the labor)
+???
+The current neuroimaging workflow requires extensive knowledge in +sometimes orthogonal fields such as neuroscience and computer science. +Dividing the labor in labs, communities or individuals with the necessary +expertise is the fundamental for the advance of the whole field.
+--
+(easy-to-use tools are risky because they might get a researcher very far with no idea whatsoever of what they've done)
+???
+There is an implicit risk in making things too easy to operate:
+For instance, imagine someone who runs fMRIPrep on diffusion data by +tricking the BIDS naming into an apparently functional MRI dataset. +If fMRIPrep reached the end at all, the garbage at the output could be fed into +further tools, in a sort of a snowballing problem.
+When researchers have access to the guts of the software and are given an opportunity to understand what's going on, the risk of misuse dips.
+--
+(and to some extent this is not necessarily bad, as long as they are kept well-tested and they embrace/help-develop some minimal standards)
+???
+AFNI, ANTs, FSL, FreeSurfer, SPM, etc. have comprehensive software validation tests, +methodological validation tests, stress tests, etc. - which pushed up their quality and made them fundamental for the field.
+Therefore, it is better to keep things that way (although some minimal efforts towards convergence in compatibility are of course welcome)
+After the success of fMRIPrep, some neuroimagers asked "when a diffusion MRI fMRIPrep?"
+(please note this down)
+--
+Same situation in the field of diffusion MRI:
+(https://www.ismrm.org/19/program_files/MIS15.htm)
+--
+Joseph, M.; Pisner, D.; Richie-Halford, A.; Lerma-Usabiaga, G.; Keshavan, A.; Kent, JD.; Cieslak, M.; Poldrack, RA.; Rokem, A.; Esteban, O.
+template: newsection +layout: false
+.middle.center[
+]
+???
+The enormous success of fMRIPrep led us to propose +its generalization to other MRI and non-MRI modalities, +as well as nonhuman species (for instance, rodents), +and particular populations currently unsupported by fMRIPrep +such as infants.
+
+
.pull-left[
+Analysis-grade data is an analogy to the concept of "sushi-grade (or sashimi-grade) fish" in that both are:
+.large[minimally preprocessed,]
+and
+.large[safe to consume directly.] +]
+.pull-right[
+
+]
???
+The goal, therefore, of NiPreps is to extend the scanner +so that, in a way, they produce data ready for analysis.
+We liken these analysis-grade data to sushi-grade fish, +because in both cases the product is minimally preprocessed +and at the same time safe to consume as is.
+template: newsection +layout: false
+.middle.center[
+
+]
???
+For the last two years we've been decomposing the architecture of fMRIPrep, spinning off its constituent parts that are valuable in other applications.
+This process of decoupling (to use a proper CS term) has been greatly facilitated by the modular nature of the code since its inception.
+
+???
+The processing elements extracted from fMRIPrep can be mapped to three +regimes of responsibility:
+As we can see, the boundaries of these three architectural layers are soft and tools such as TemplateFlow may stand in between.
+Only projects enclosed in the brain shape pertain to the NiPreps community. NiPype, NiBabel and BIDS are so deeply embedded as dependencies that NiPreps can't be understood without them.
+
BIDS provides a standard, guaranteeing I/O agreements:
+Allows workflows to self-adapt to the inputs
+Ensures the shareability of the results
+PyBIDS: a Python tool to query BIDS datasets (Yarkoni et al., 2019):
+>>> from bids import BIDSLayout
+
+# Point PyBIDS to the dataset's path
+>>> layout = BIDSLayout("/data/coolproject")
+
+# List the participant IDs of present subjects
+>>> layout.get_subjects()
+['01', '02', '03', '04', '05']
+
+# List session identifiers, if present
+>>> layout.get_sessions()
+['01', '02']
+
+# List functional MRI tasks
+>>> layout.get_tasks()
+['rest', 'nback']
+???
+BIDS is one of the keys to success for fMRIPrep and consequently, a strategic element of NiPreps.
+Because the tools so far are written in Python, PyBIDS is a powerful tool to index and query inputs and outputs.
+The code snippet illustrates the ease to find out the subject identifiers available in the dataset, sessions, and tasks.
+.cut-right[ +
derivatives/
+├── fmriprep/
+│ ├── dataset_description.json
+│ ├── logs
+│ ├── sub-01.html
+│ ├── sub-01/
+│ │ ├── anat/
+│ │ │ ├── sub-01_desc-brain_mask.nii.gz
+│ │ │ ├── sub-01_dseg.nii.gz
+│ │ │ ├── sub-01_label-GM_probseg.nii.gz
+│ │ │ ├── sub-01_label-WM_probseg.nii.gz
+│ │ │ ├── sub-01_label-CSF_probseg.nii.gz
+│ │ │ ├── sub-01_desc-preproc_T1w.nii.gz
+│ │ │ ├── sub-01_space-MNI152_desc-brain_mask.nii.gz
+│ │ │ ├── sub-01_space-MNI152_dseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_label-GM_probseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_label-WM_probseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_label-CSF_probseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_desc-preproc_T1w.nii.gz
+│ │ │ ├── sub-01_from-MNI152_to-T1w_mode-image_xfm.h5
+│ │ │ ├── sub-01_from-T1w_to-MNI152_mode-image_xfm.h5
+│ │ │ └── sub-01_from-orig_to-T1w_mode-image_xfm.txt
+│ │ ├── figures/
+│ │ └── func/
+│ │ ├── sub-01_task-rhymejudgment_space-MNI152_boldref.nii.gz
+│ │ ├── sub-01_task-rhymejudgment_space-MNI152_desc-preproc_bold.nii.gz
+│ │ ├── sub-01_task-rhymejudgment_space-MNI152_desc-confounds_regressors.nii.gz
+│ │ └── sub-01_task-rhymejudgment_space-MNI152_desc-brain_mask.nii.gz
+???
+All NiPreps must write out BIDS-Derivatives. +As illustrated in the example, the outputs of fMRIPrep are very similar to the BIDS standard for acquired data.
+
Use of containers & CI/CD
+Uniform interface: + .cut-right[ +
fmriprep /data /data/derivatives/fmriprep-20.1.1 participant [+OPTIONS]
+???
+All end-user applications in NiPreps must conform to the BIDS-Apps specifications.
+The BIDS-Apps paper identified a common pattern in neuroimaging studies, where individual participants (and runs) are processed first individually, and then based on the outcomes, further levels of data aggregation are executed.
+For this reason, BIDS-Apps define two major levels of execution: participant and group level.
+Finally, the paper also stresses the importance of containerizing applications to ensure long-term preservation of run-to-run repeatability and proposes a common command line interface as described at the bottom:
+.pull-left[
+
+
+
from nipype.interfaces.fsl import BET
+brain_extract = BET(
+ in_file="/data/coolproject/sub-01/ses-01/anat/sub-01_ses-01_T1w.nii",
+ out_file="/out/sub-01/ses-01/anat/sub-01_ses-01_desc-brain_T1w.nii"
+)
+brain_extract.run()
+Nipype is the gateway to mix-and-match from AFNI, ANTs, Dipy, FreeSurfer, FSL, MRTrix, SPM, etc. +]
+.pull-right[
+
+
+
]
+???
+Nipype is the glue stitching together all the underlying neuroimaging toolboxes and provides the execution framework.
+The snippet shows how the widely known BET tool from FSL can be executed using NiPype. This is a particular example instance of interfaces - which provide uniform access to the tooling with Python.
+Finally, combining these interfaces we generate processing workflows to fulfill higher level processing tasks.
+
+
+
???
+For instance, we may have a look into fMRIPrep's functional processing block.
+Nipype helps understand (and opens windows in the black box) generating these graph representation of the workflow.
+
"""Fix the affine of a rodent dataset, imposing 0.2x0.2x0.2 [mm]."""
+import numpy as np
+import nibabel as nb
+
+# Open the file
+img = nb.load("sub-25_MGE_MouseBrain_3D_MGE_150.nii.gz")
+
+# New (correct) affine
+aff = np.diag((-0.2, -0.2, 0.2, 1.0))
+
+# Use nibabel to reorient to canonical
+card = nb.as_closest_canonical(nb.Nifti1Image(
+ img.dataobj,
+ np.diag((-0.2, -0.2, 0.2, 1.0)),
+ None
+))
+
+# Save to disk
+card.to_filename("sub-25_T2star.nii.gz")
+???
+NiBabel allows Python to easily access neuroimaging data formats such as NIfTI, GIFTI and CIFTI2.
+Although this might be a trivial task, the proliferation of neuroimaging software has led to some sort of Wild West of formats, and sometimes interoperation is not ensured.
+.pull-left[
+
+
+
+
Transforms typically are the outcome of image registration methodologies
+The proliferation of software implementations of image registration methodologies has resulted in a spread of data structures and file formats used to preserve and communicate transforms.
+(Esteban et al., 2020) +]
+.pull-right[
+
+
+
]
+???
+NiTransforms is a super-interesting toy project where we are exercising our finest coding skills. +It completes NiBabel in the effort of making spatial transforms calculated by neuroimaging software tools interoperable.
+When it goes beyond the alpha state, it is expected to be merged into NiBabel.
+At the moment, NiTransforms is already integrated in fMRIPrep +20.1 +to concatenate LTA (linear affine transforms) transforms obtained with FreeSurfer, +ITK transforms obtained with ANTs, and motion parameters estimated with FSL.
+Compatibility across formats is hard due to the many arbitrary decisions in establishing the mathematical framework of the transform and the intrinsic confusion of applying a transform.
+While intuitively we understand applying a transform as "transforming the moving image so that I can represent it overlaid or fused with the reference image and both should look aligned", in reality, we only transform coordinates from the reference image into the moving image's space (step 1 on the right).
+Once we know where the center of every voxel of the reference image falls in the moving image coordinate system, we read in the information (in other words, a value) from the moving image. Because the location will probably be off-grid, we interpolate such a value from the neighboring voxels (step 2).
+Finally (step 3) we generate a new image object with the structure of the reference image and the data interpolated from the moving information. This new image object is the moving image "moved" on to the reference image space and thus, both look aligned.
+.pull-left[
+
+
+
>>> from templateflow import api as tflow
+>>> tflow.get(
+... 'MNI152NLin6Asym',
+... desc=None,
+... resolution=1,
+... suffix='T1w',
+... extension='nii.gz'
+... )
+PosixPath('/templateflow_home/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-01_T1w.nii.gz')
+.large[www.templateflow.org] +]
+.pull-right[
+
+
+
]
+???
+One of the most ancient feature requests received from fMRIPrep early adopters was improving the flexibility of spatial normalization to standard templates other than fMRIPrep's default.
+For instance, infant templates.
+TemplateFlow offers an Archive of templates where they are stored, maintained and re-distributed;
+and a Python client that helps accessing them.
+On the right hand side, an screenshot of the TemplateFlow browser shows some of the templates currently available in the repository. The browser can be reached at www.templateflow.org.
+The tool is based on PyBIDS, and the snippet will surely remind you of it. +In this case the example shows how to obtain the T1w template corresponding to FSL's MNI space, at the highest resolution.
+If the files requested are not in TemplateFlow's cache, they will be pulled down and kept for further utilization.
+
+
+
.small[(Ciric et al. 2020, in prep)]
+???
+The Archive allows a rich range of data and metadata to be stored with the template.
+Datatypes in the repository cover:
+Metadata can be stored with the usual BIDS options.
+Finally, templates allow having multiple cohorts, in a similar encoding to that of multi-session BIDS datasets.
+Multiple cohorts are useful, for instance, in infant templates with averages at several gestational ages.
+
+
NiWorkflows is a miscellaneous mixture of tooling used by downstream NiPreps:
+???
+NiWorkflows is, historically, the first component detached from fMRIPrep.
+For that reason, its scope and vision has very fuzzy boundaries as compared to the other tools.
+The most relevant utilities incorporated within NiWorkflows are:
+--
+???
+First, the individual report system which aggregates the visual elements or the reports (which we call "reportlets") and generates the final HTML document.
+Also, most of the engineering behind the generation of these reportlets and their integration within NiPype are part of NiWorkflows
+--
+???
+Beyond the extension of NiPype to generate a reportlet from any given interface, NiWorkflows is the test bed for many utilities that are then upstreamed to nipype.
+Also, special interfaces with a limited scope that should not be included in nipype are maintained here.
+--
+???
+Finally, NiWorkflows indeed offers workflows that can be used by end-user NiPreps. For instance atlas-based brain extraction of anatomical images, based on ANTs.
+
???
+Echo-planar imaging (EPI) are typically affected by distortions along the phase encoding axis, caused by the perturbation of the magnetic field at tissue interfaces.
+Looking at the reportlet, we can see how in the "before" panel, the image is warped.
+The distortion is most obvious in the coronal view (middle row) because this image has posterior-anterior phase encoding.
+Focusing on the changes between "before" and "after" correction in this coronal view, we can see how the blue contours delineating the corpus callosum fit better the dark shade in the data after correction.
+???
+So, what's coming up next?
+NiBabies is some sort of NiWorkflows equivalent for the preprocessing of infant imaging. At the moment, only atlas-based brain extraction using ANTs (and adapted from NiWorkflows) is in active developments.
+Next steps include brain tissue segmentation.
+Similarly, NiRodents is the NiWorkflows parallel for the prepocessing of rodent preclinical imaging. Again, only atlas-based brain extraction adapted from NiWorkflows is being developed.
+???
+To wrap-up, I've presented NiPreps, a framework for developing preprocessing +workflows inspired by fMRIPrep.
+The framework is heavily principle and tags along BIDS as a foundational component
+NiPreps should not reinvent any wheel, trying to reuse as much as possible of the widely used and tested existing software.
+Nipype serves as a glue components to orchestrate workflows.
+--
+???
+But why just preprocessing, with a very strict scope?
+We propose to think about preprocessing as part of the image acquisition and reconstruction process (in other words, scanning), rather than part of the analysis workflow.
+This decoupling from analysis comes with several upshots:
+First, there are less moving parts to play with for researchers in the attempt to fit their methods to the data (instead of fitting data with their methods).
+Second, such division of labor allows the researcher to use their time in the analysis.
+Finally, two preprocessed datasets from two different studies and scanning sites should be more homogeneous when processed with the same instruments, in comparison to processing them with idiosyncratic, lab-managed, preprocessing workflows.
+However, for NiPreps to work we need to make sure the tools are transparent.
+Not just with the individual reports and thorough documentation, also because of the community driven development. For instance, the peer-review process that goes around large incremental changes is fundamental to ensure the quality of the tool.
+In addition, best engineering practices suggested in the BIDS-Apps paper, along with those we have been including with fMRIPrep, are necessary to ensure the quality of the final product.
+--
+???
+As an open problem, validating the results of the tool remains extremely challenging for the lack in gold standard datasets that can tell us the best possible outcome.
+NMiND = NeverMIND, this Neuroimaging Method Is Not Duplicated
+M. Milham, D. Fair, T. Satterthwaite, S. Ghosh, R. Poldrack, etc.
+--
+nosology group, coding standards & patterns, sharing standards, testing standards, crediting contributors, funding strategy, benchmarking datasets.
+Please Join!
+template: newsection +layout: false
+.middle.center[
+]
+ + + + + + + + + + + + + + + + +layout: false +count: false
+.middle.center[
+
+
]
+layout: false +count: false
+.middle.center[
+
+
]
+???
+name: newsection +layout: true +class: section-separator
+.perma-sidebar[
+
+
+
+
+
+
+
+
+
]
+name: sidebar +layout: true
+.perma-sidebar[
+
+
+
+
+
+
+
+
+
]
+template: sidebar
+
+ 
+
(source: next slide)
+
+ 
+
(Strother, 2006; 10.1109/MEMB.2006.1607667)
+
+ 
+
Adapted (Strother, 2006)
+*a.B. = after BIDS (Brain Imaging Data Structure; Gorgolewski et al. (2016))
+container technology, CI/CD
+a wealth of prior knowledge (esp. about humans)
+LOTS of data acquired everyday
+A uniform and complete interface to data:
+Uniform: enables the workflow adapt to the data
+Complete: enables validation and minimizes human-intervention
+Extensible reproducibility:
+???
+
+
+
Around 50% of teams used fMRIPrep'ed inputs.
+???
+fMRIPrep takes in a task-based or resting-state +functional MRI dataset in BIDS-format +and returns preprocessed data ready for analysis.
+Preprocessed data can be used for a broad range of analysis, and they are +formatted following BIDS-Derivatives to maximize compatibility with: + * major software packages (AFNI, FSL, SPM*, etc.) + * further temporal filtering and denoising: fMRIDenoise + * any BIDS-Derivatives compliant tool (e.g., FitLins).
+--
+???
+fMRIPrep adopts the BIDS-App specifications. +That means the software is tested with every change to the codebase, +it also means that packaging, containerization, and deployment are also +automated and require tests to be passing. +BIDS-Apps are inter-operable (via BIDS-Derivatives), +and optimized for execution in HPC, Cloud, etc.
+--
+???
+fMRIPrep minimizes human intervention because the user does not +need to fiddle with any parameters - they are obtained from the BIDS structure. +However, fMRIPrep does allow some flexibility to ensure the preprocessing meets the requirements of the intended analyses.
+(we just wanted a robust tool to automatically preprocess incoming data of OpenNeuro.org)
+--
+--
+.pull-left[
+Preprocessing of fMRI was in need for division of labor.
+Obsession with transparency made early-adopters confident of the recipes they were applying.
+Responsiveness to feedback. +]
+.pull-right[
+
+
+
]
+???
+Preprocessing is a time-consuming effort, requires expertise converging imaging foundations & CS, typically addressed with legacy in-house pipelines.
+On the right-hand side, you'll find the chart of unique visitors +to fmriprep.org, which is the documentation website.
+
+
+
--
+(probably not preprocessing...)
+???
+With the development of fMRIPrep we understood that +researchers don't want to waste their time on preprocessing +(except for researchers developing new preprocessing techniques).
+--
+(research programs just can't cover the neuroscience and the engineering of the whole workflow - we need to divide the labor)
+???
+The current neuroimaging workflow requires extensive knowledge in +sometimes orthogonal fields such as neuroscience and computer science. +Dividing the labor in labs, communities or individuals with the necessary +expertise is the fundamental for the advance of the whole field.
+--
+(easy-to-use tools are risky because they might get a researcher very far with no idea whatsoever of what they've done)
+???
+There is an implicit risk in making things too easy to operate:
+For instance, imagine someone who runs fMRIPrep on diffusion data by +tricking the BIDS naming into an apparently functional MRI dataset. +If fMRIPrep reached the end at all, the garbage at the output could be fed into +further tools, in a sort of a snowballing problem.
+When researchers have access to the guts of the software and are given an opportunity to understand what's going on, the risk of misuse dips.
+--
+(and to some extent this is not necessarily bad, as long as they are kept well-tested and they embrace/help-develop some minimal standards)
+???
+AFNI, ANTs, FSL, FreeSurfer, SPM, etc. have comprehensive software validation tests, +methodological validation tests, stress tests, etc. - which pushed up their quality and made them fundamental for the field.
+Therefore, it is better to keep things that way (although some minimal efforts towards convergence in compatibility are of course welcome)
+
+ 
+
+(Esteban et al., 2019) +
+After the success of fMRIPrep, Dr. A. Keshavan asked "when a dMRIPrep?"
+(please note this down)
+After the success of fMRIPrep, Dr. A. Keshavan asked "when a dMRIPrep?"
+
+
+
Image Processing: Possible Guidelines for the Standardization & Clinical Applications
+(Veraart, 2019)
+
+Joseph, M.; Pisner, D.; Richie-Halford, A.; Lerma-Usabiaga, G.; Keshavan, A.; Kent, JD.; Veraart, J.; Cieslak, M.; Poldrack, RA.; Rokem, A.; Esteban, O.
+template: newsection +layout: false
+.middle.center[
+]
+???
+Let's walk through one example of report. +Reports have several sections, starting with a summary +indicating the particularities of this dataset +and workflow choices made based on the input data.
+The anatomical section follows with several visualizations +to assess the anatomical processing steps mentioned before, +spatial normalization to template spaces (the flickering +panel helps assess alignment) and finally surface reconstruction.
+Then, all functional runs are concatenated, and all show the same +structure. +After an initial summary of this particular run, +the alignment to the same subject's anatomical image is presented, +with contours of the white and pial surfaces as cues. +Next panel shows the brain mask and ROIs utilized by the CompCor +denoising. +For each run we then find some visualizations to assess the +generated confounding signals.
+After all functional runs are presented, the About section keeps +information to aid reproducibility of results, such as the software's +version, or the exact command line run.
+The boilerplate is found next, with a text version shown by default +and tabs to convert to Markdown and LaTeX.
+Reports conclude with a list of encountered errors (if any).
+.pull-left[
+
+
+
]
+.pull-right[
+.distribute[ +fMRIPrep generates one participant-wide report after execution.
+Reports describe the data as found, and the steps applied +(providing .blue[visual support to look inside the box]):
+show researchers their data;
+show how fMRIPrep interpreted the data (describing the actual preprocessing steps);
+quality control of results, facilitating early error detection. +] +]
+???
+Therefore, reports have become a fundamental feature of fMRIPrep +because they not only allow assessing the quality of the processing, +but also provide an insight about the logic supporting such processing.
+In other words, reports help respond to the what was done and the why was it done +in addition to the how well it did.
+template: newsection +layout: false
+.middle.center[
+]
+???
+The enormous success of fMRIPrep led us to propose +its generalization to other MRI and non-MRI modalities, +as well as nonhuman species (for instance, rodents), +and particular populations currently unsupported by fMRIPrep +such as infants.
+
+
.pull-left[
+Analysis-grade data is an analogy to the concept of "sushi-grade (or sashimi-grade) fish" in that both are:
+.large[minimally preprocessed,]
+and
+.large[safe to consume directly.] +]
+.pull-right[
+
+]
???
+The goal, therefore, of NiPreps is to extend the scanner +so that, in a way, they produce data ready for analysis.
+We liken these analysis-grade data to sushi-grade fish, +because in both cases the product is minimally preprocessed +and at the same time safe to consume as is.
+template: newsection +layout: false
+.middle.center[
+
+]
???
+For the last two years we've been decomposing the architecture of fMRIPrep, spinning off its constituent parts that are valuable in other applications.
+This process of decoupling (to use a proper CS term) has been greatly facilitated by the modular nature of the code since its inception.
+
+???
+The processing elements extracted from fMRIPrep can be mapped to three +regimes of responsibility:
+As we can see, the boundaries of these three architectural layers are soft and tools such as TemplateFlow may stand in between.
+Only projects enclosed in the brain shape pertain to the NiPreps community. NiPype, NiBabel and BIDS are so deeply embedded as dependencies that NiPreps can't be understood without them.
+
BIDS provides a standard, guaranteeing I/O agreements:
+Allows workflows to self-adapt to the inputs
+Ensures the shareability of the results
+PyBIDS: a Python tool to query BIDS datasets (Yarkoni et al., 2019):
+>>> from bids import BIDSLayout
+
+# Point PyBIDS to the dataset's path
+>>> layout = BIDSLayout("/data/coolproject")
+
+# List the participant IDs of present subjects
+>>> layout.get_subjects()
+['01', '02', '03', '04', '05']
+
+# List session identifiers, if present
+>>> layout.get_sessions()
+['01', '02']
+
+# List functional MRI tasks
+>>> layout.get_tasks()
+['rest', 'nback']
+???
+BIDS is one of the keys to success for fMRIPrep and consequently, a strategic element of NiPreps.
+Because the tools so far are written in Python, PyBIDS is a powerful tool to index and query inputs and outputs.
+The code snippet illustrates the ease to find out the subject identifiers available in the dataset, sessions, and tasks.
+.cut-right[ +
derivatives/
+├── fmriprep/
+│ ├── dataset_description.json
+│ ├── logs
+│ ├── sub-01.html
+│ ├── sub-01/
+│ │ ├── anat/
+│ │ │ ├── sub-01_desc-brain_mask.nii.gz
+│ │ │ ├── sub-01_dseg.nii.gz
+│ │ │ ├── sub-01_label-GM_probseg.nii.gz
+│ │ │ ├── sub-01_label-WM_probseg.nii.gz
+│ │ │ ├── sub-01_label-CSF_probseg.nii.gz
+│ │ │ ├── sub-01_desc-preproc_T1w.nii.gz
+│ │ │ ├── sub-01_space-MNI152_desc-brain_mask.nii.gz
+│ │ │ ├── sub-01_space-MNI152_dseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_label-GM_probseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_label-WM_probseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_label-CSF_probseg.nii.gz
+│ │ │ ├── sub-01_space-MNI152_desc-preproc_T1w.nii.gz
+│ │ │ ├── sub-01_from-MNI152_to-T1w_mode-image_xfm.h5
+│ │ │ ├── sub-01_from-T1w_to-MNI152_mode-image_xfm.h5
+│ │ │ └── sub-01_from-orig_to-T1w_mode-image_xfm.txt
+│ │ ├── figures/
+│ │ └── func/
+│ │ ├── sub-01_task-rhymejudgment_space-MNI152_boldref.nii.gz
+│ │ ├── sub-01_task-rhymejudgment_space-MNI152_desc-preproc_bold.nii.gz
+│ │ ├── sub-01_task-rhymejudgment_space-MNI152_desc-confounds_regressors.nii.gz
+│ │ └── sub-01_task-rhymejudgment_space-MNI152_desc-brain_mask.nii.gz
+???
+All NiPreps must write out BIDS-Derivatives. +As illustrated in the example, the outputs of fMRIPrep are very similar to the BIDS standard for acquired data.
+
Use of containers & CI/CD
+Uniform interface: + .cut-right[ +
fmriprep /data /data/derivatives/fmriprep-20.1.1 participant [+OPTIONS]
+???
+All end-user applications in NiPreps must conform to the BIDS-Apps specifications.
+The BIDS-Apps paper identified a common pattern in neuroimaging studies, where individual participants (and runs) are processed first individually, and then based on the outcomes, further levels of data aggregation are executed.
+For this reason, BIDS-Apps define two major levels of execution: participant and group level.
+Finally, the paper also stresses the importance of containerizing applications to ensure long-term preservation of run-to-run repeatability and proposes a common command line interface as described at the bottom:
+.pull-left[
+
+
+
from nipype.interfaces.fsl import BET
+brain_extract = BET(
+ in_file="/data/coolproject/sub-01/ses-01/anat/sub-01_ses-01_T1w.nii",
+ out_file="/out/sub-01/ses-01/anat/sub-01_ses-01_desc-brain_T1w.nii"
+)
+brain_extract.run()
+Nipype is the gateway to mix-and-match from AFNI, ANTs, Dipy, FreeSurfer, FSL, MRTrix, SPM, etc. +]
+.pull-right[
+
+
+
]
+???
+Nipype is the glue stitching together all the underlying neuroimaging toolboxes and provides the execution framework.
+The snippet shows how the widely known BET tool from FSL can be executed using NiPype. This is a particular example instance of interfaces - which provide uniform access to the tooling with Python.
+Finally, combining these interfaces we generate processing workflows to fulfill higher level processing tasks.
+
+
+
???
+For instance, we may have a look into fMRIPrep's functional processing block.
+Nipype helps understand (and opens windows in the black box) generating these graph representation of the workflow.
+
"""Fix the affine of a rodent dataset, imposing 0.2x0.2x0.2 [mm]."""
+import numpy as np
+import nibabel as nb
+
+# Open the file
+img = nb.load("sub-25_MGE_MouseBrain_3D_MGE_150.nii.gz")
+
+# New (correct) affine
+aff = np.diag((-0.2, -0.2, 0.2, 1.0))
+
+# Use nibabel to reorient to canonical
+card = nb.as_closest_canonical(nb.Nifti1Image(
+ img.dataobj,
+ np.diag((-0.2, -0.2, 0.2, 1.0)),
+ None
+))
+
+# Save to disk
+card.to_filename("sub-25_T2star.nii.gz")
+???
+NiBabel allows Python to easily access neuroimaging data formats such as NIfTI, GIFTI and CIFTI2.
+Although this might be a trivial task, the proliferation of neuroimaging software has led to some sort of Wild West of formats, and sometimes interoperation is not ensured.
+.pull-left[
+
+
+
+
Transforms typically are the outcome of image registration methodologies
+The proliferation of software implementations of image registration methodologies has resulted in a spread of data structures and file formats used to preserve and communicate transforms.
+(Esteban et al., 2020) +]
+.pull-right[
+
+
+
]
+???
+NiTransforms is a super-interesting toy project where we are exercising our finest coding skills. +It completes NiBabel in the effort of making spatial transforms calculated by neuroimaging software tools interoperable.
+When it goes beyond the alpha state, it is expected to be merged into NiBabel.
+At the moment, NiTransforms is already integrated in fMRIPrep +20.1 +to concatenate LTA (linear affine transforms) transforms obtained with FreeSurfer, +ITK transforms obtained with ANTs, and motion parameters estimated with FSL.
+Compatibility across formats is hard due to the many arbitrary decisions in establishing the mathematical framework of the transform and the intrinsic confusion of applying a transform.
+While intuitively we understand applying a transform as "transforming the moving image so that I can represent it overlaid or fused with the reference image and both should look aligned", in reality, we only transform coordinates from the reference image into the moving image's space (step 1 on the right).
+Once we know where the center of every voxel of the reference image falls in the moving image coordinate system, we read in the information (in other words, a value) from the moving image. Because the location will probably be off-grid, we interpolate such a value from the neighboring voxels (step 2).
+Finally (step 3) we generate a new image object with the structure of the reference image and the data interpolated from the moving information. This new image object is the moving image "moved" on to the reference image space and thus, both look aligned.
+.pull-left[
+
+
+
>>> from templateflow import api as tflow
+>>> tflow.get(
+... 'MNI152NLin6Asym',
+... desc=None,
+... resolution=1,
+... suffix='T1w',
+... extension='nii.gz'
+... )
+PosixPath('/templateflow_home/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-01_T1w.nii.gz')
+.large[www.templateflow.org] +]
+.pull-right[
+
+
+
]
+???
+One of the most ancient feature requests received from fMRIPrep early adopters was improving the flexibility of spatial normalization to standard templates other than fMRIPrep's default.
+For instance, infant templates.
+TemplateFlow offers an Archive of templates where they are stored, maintained and re-distributed;
+and a Python client that helps accessing them.
+On the right hand side, an screenshot of the TemplateFlow browser shows some of the templates currently available in the repository. The browser can be reached at www.templateflow.org.
+The tool is based on PyBIDS, and the snippet will surely remind you of it. +In this case the example shows how to obtain the T1w template corresponding to FSL's MNI space, at the highest resolution.
+If the files requested are not in TemplateFlow's cache, they will be pulled down and kept for further utilization.
+
+
+
.small[(Ciric et al. 2020, in prep)]
+???
+The Archive allows a rich range of data and metadata to be stored with the template.
+Datatypes in the repository cover:
+Metadata can be stored with the usual BIDS options.
+Finally, templates allow having multiple cohorts, in a similar encoding to that of multi-session BIDS datasets.
+Multiple cohorts are useful, for instance, in infant templates with averages at several gestational ages.
+
+
NiWorkflows is a miscellaneous mixture of tooling used by downstream NiPreps:
+???
+NiWorkflows is, historically, the first component detached from fMRIPrep.
+For that reason, its scope and vision has very fuzzy boundaries as compared to the other tools.
+The most relevant utilities incorporated within NiWorkflows are:
+--
+???
+First, the individual report system which aggregates the visual elements or the reports (which we call "reportlets") and generates the final HTML document.
+Also, most of the engineering behind the generation of these reportlets and their integration within NiPype are part of NiWorkflows
+--
+???
+Beyond the extension of NiPype to generate a reportlet from any given interface, NiWorkflows is the test bed for many utilities that are then upstreamed to nipype.
+Also, special interfaces with a limited scope that should not be included in nipype are maintained here.
+--
+???
+Finally, NiWorkflows indeed offers workflows that can be used by end-user NiPreps. For instance atlas-based brain extraction of anatomical images, based on ANTs.
+
???
+Echo-planar imaging (EPI) are typically affected by distortions along the phase encoding axis, caused by the perturbation of the magnetic field at tissue interfaces.
+Looking at the reportlet, we can see how in the "before" panel, the image is warped.
+The distortion is most obvious in the coronal view (middle row) because this image has posterior-anterior phase encoding.
+Focusing on the changes between "before" and "after" correction in this coronal view, we can see how the blue contours delineating the corpus callosum fit better the dark shade in the data after correction.
+???
+So, what's coming up next?
+NiBabies is some sort of NiWorkflows equivalent for the preprocessing of infant imaging. At the moment, only atlas-based brain extraction using ANTs (and adapted from NiWorkflows) is in active developments.
+Next steps include brain tissue segmentation.
+Similarly, NiRodents is the NiWorkflows parallel for the prepocessing of rodent preclinical imaging. Again, only atlas-based brain extraction adapted from NiWorkflows is being developed.
+???
+To wrap-up, I've presented NiPreps, a framework for developing preprocessing +workflows inspired by fMRIPrep.
+The framework is heavily principle and tags along BIDS as a foundational component
+NiPreps should not reinvent any wheel, trying to reuse as much as possible of the widely used and tested existing software.
+Nipype serves as a glue components to orchestrate workflows.
+--
+???
+But why just preprocessing, with a very strict scope?
+We propose to think about preprocessing as part of the image acquisition and reconstruction process (in other words, scanning), rather than part of the analysis workflow.
+This decoupling from analysis comes with several upshots:
+First, there are less moving parts to play with for researchers in the attempt to fit their methods to the data (instead of fitting data with their methods).
+Second, such division of labor allows the researcher to use their time in the analysis.
+Finally, two preprocessed datasets from two different studies and scanning sites should be more homogeneous when processed with the same instruments, in comparison to processing them with idiosyncratic, lab-managed, preprocessing workflows.
+However, for NiPreps to work we need to make sure the tools are transparent.
+Not just with the individual reports and thorough documentation, also because of the community driven development. For instance, the peer-review process that goes around large incremental changes is fundamental to ensure the quality of the tool.
+In addition, best engineering practices suggested in the BIDS-Apps paper, along with those we have been including with fMRIPrep, are necessary to ensure the quality of the final product.
+--
+???
+As an open problem, validating the results of the tool remains extremely challenging for the lack in gold standard datasets that can tell us the best possible outcome.
+template: newsection +layout: false
+.middle.center[
+]
+template: newsection +layout: false
+.middle.center[
+]
+ + + + + + + + + + + + + + + + +