"openFiles": ["workshops/scipy2024/index.ipynb"]

[](https://mybinder.org/v2/gh/xarray-contrib/xarray-tutorial/HEAD?labpath=workshops/scipy2024/index.ipynb)

Multi-dimensional (a.k.a. N-dimensional, ND) arrays (sometimes called “tensors”)

are an essential part of computational science. They are encountered in a wide

range of fields, including physics, astronomy, geoscience, bioinformatics,

engineering, finance, and deep learning. In Python, [NumPy](https://numpy.org/)

provides the fundamental data structure and API for working with raw ND arrays.

However, real-world datasets are usually more than just raw numbers; they have

labels which encode information about how the array values map to locations in

space, time, etc.

The N-dimensional nature of Xarray’s data structures makes it suitable for

dealing with multi-dimensional scientific data, and its use of dimension names

instead of axis labels (`dim='time'` instead of `axis=0`) makes such arrays much

more manageable than the raw NumPy ndarray: with Xarray, you don’t need to keep

track of the order of an array’s dimensions or insert dummy dimensions of size 1

to align arrays (e.g., using np.newaxis).

The immediate payoff of using Xarray is that you’ll write less code. The

long-term payoff is that you’ll understand what you were thinking when you come

back to look at it weeks or months later.

Here is an example of how we might structure a dataset for a weather forecast:

<img src="https://docs.xarray.dev/en/stable/_images/dataset-diagram.png" align="center" width="80%">

You'll notice multiple data variables (temperature, precipitation), coordinate

variables (latitude, longitude), and dimensions (x, y, t). We'll cover how these

fit into Xarray's data structures below.

Xarray doesn’t just keep track of labels on arrays – it uses them to provide a

powerful and concise interface. For example:

- Apply operations over dimensions by name: `x.sum('time')`.

- Select values by label (or logical location) instead of integer location:

`x.loc['2014-01-01']` or `x.sel(time='2014-01-01')`.

- Mathematical operations (e.g., `x - y`) vectorize across multiple dimensions

(array broadcasting) based on dimension names, not shape.

- Easily use the split-apply-combine paradigm with groupby:

`x.groupby('time.dayofyear').mean()`.

- Database-like alignment based on coordinate labels that smoothly handles

missing values: `x, y = xr.align(x, y, join='outer')`.

- Keep track of arbitrary metadata in the form of a Python dictionary:

`x.attrs`.

Although the Xarray library was originally developed with Earth Science datasets in mind, the datastructures work well across many other domains! For example, below is a side-by-side view of a data schematic on the left and Xarray Dataset representation on the right taken from a mosquito genetics analysis:


The data can be stored as a 3-dimensional array, where one dimension of the array corresponds to positions (**variants**) within a reference genome, another dimension corresponds to the individual mosquitoes that were sequenced (**samples**), and a third dimension corresponds to the number of genomes within each individual (**ploidy**)."

You can explore this dataset in detail via the [training course in data analysis for genomic surveillance of African malaria vectors](https://anopheles-genomic-surveillance.github.io/workshop-5/module-1-xarray.html)!

The following collection of notebooks provide interactive code examples for working with example datasets and constructing Xarray data structures manually.

"In this lesson, we cover the basics of Xarray data structures. By the end of the lesson, we will be able to:\n",

":::{admonition} Learning Goals\n",

"- Understand the basic Xarray data structures `DataArray` and `Dataset` \n",

"- Customize the display of Xarray data structures\n",

"- The connection between Pandas and Xarray data structures\n",

":::"

"multi-dimensional arrays while `Dataset` combines multiple DataArrays.\n",

"To learn how to create a DataArray or Dataset manually, see the [Creating Data Structures](01.1_creating_data_structures.ipynb) tutorial."

"import xarray as xr\n",

"import pandas as pd\n",

"\n",

"# When working in a Jupyter Notebook you might want to customize Xarray display settings to your liking\n",

"# The following settings reduce the amount of data displayed out by default\n",

"xr.set_options(display_expand_attrs=False, display_expand_data=False)\n",

"np.set_printoptions(threshold=10, edgeitems=2)"

"`Dataset` objects are dictionary-like containers of DataArrays, mapping a variable name to each DataArray.\n",

"\n",

"Xarray has a few small real-world tutorial datasets hosted in this GitHub repository https://github.com/pydata/xarray-data.\n",

"We'll use the [xarray.tutorial.load_dataset](https://docs.xarray.dev/en/stable/generated/xarray.tutorial.open_dataset.html#xarray.tutorial.open_dataset) convenience function to download and open the `air_temperature` (National Centers for Environmental Prediction) Dataset by name."

"#### HTML vs text representations\n",

"The `\"html\"` representation is interactive, allowing you to collapse sections (▶) and\n",

"view attributes and values for each value (📄 and ≡)."

"☝️ From top to bottom the output consists of:\n",

"- **Dimensions**: summary of all *dimensions* of the `Dataset` `(lat: 25, time: 2920, lon: 53)`: this tells us that the first dimension is named `lat` and has a size of `25`, the second dimension is named `time` and has a size of `2920`, and the third dimension is named `lon` and has a size of `53`. Because we will access the dimensions by name, the order doesn't matter.\n",

"- **Coordinates**: an unordered list of *coordinates* or dimensions with coordinates with one item per line. Each item has a name, one or more dimensions in parentheses, a dtype and a preview of the values. Also, if it is a dimension coordinate, it will be printed in **bold** font. *dimensions without coordinates* appear in plain font (there are none in this example, but you might imagine a 'mask' coordinate that has a value assigned at every point).\n",

"- **Data variables**: names of each nD *measurement* in the dataset, followed by its dimensions `(time, lat, lon)`, dtype, and a preview of values.\n",

"- **Indexes**: Each dimension with coordinates is backed by an \"Index\". In this example, each dimension is backed by a `PandasIndex`\n",

"- **Attributes**: an unordered list of metadata (for example, a paragraph describing the dataset)"

"### to_series\n",

"This will always convert `DataArray` objects to `pandas.Series`, using a `MultiIndex` for higher dimensions\n"

"### to_dataframe\n",

"\n",

"This will always convert `DataArray` or `Dataset` objects to a `pandas.DataFrame`. Note that `DataArray` objects have to be named for this. Since columns in a `DataFrame` need to have the same index, they are\n",

"broadcasted."

:::{note}

You can access a recording of this tutorial [here](https://www.youtube.com/watch?v=L4FXcIOMlsY)

:::

"July 8, 13:30–17:30 (US/Pacific), Tacoma Convention Center Ballroom B/C\n",

"- Understand how Xarray integrates with other libraries in the scientific Python ecosystem\n",

"| The Xarray Data Model | 1:40 (40 min) | [Data structures](../../fundamentals/01_data_structures.md) <br> [Basic Indexing](../../fundamentals/02.1_indexing_Basic.ipynb) | \n",

"| Indexing & Computational Patterns | 2:30 (50 min) | [Advanced Indexing](../../intermediate/indexing/indexing.md) <br> [Computational Patterns](../../intermediate/01-high-level-computation-patterns.ipynb) <br> | \n",

"| Xarray Integrations and Extensions | 3:30 (50 min) | [The Xarray Ecosystem](../../intermediate/xarray_ecosystem.ipynb) | \n",

"| Backends & Remote data| 4:30 (50 min) | [Remote Data](../../intermediate/remote_data/remote-data.ipynb) |\n",

":::{warning}\n",

"Consider Codespaces as ephemeral environments. You may lose your connection and any edits you make.\n",

":::\n",