Working with pandas#
One of the most important features of xarray is the ability to convert to and
from pandas objects to interact with the rest of the PyData
ecosystem. For example, for plotting labeled data, we highly recommend
using the visualization built in to pandas itself or provided by the pandas
aware libraries such as Seaborn.
Hierarchical and tidy data#
Tabular data is easiest to work with when it meets the criteria for tidy data:
Each column holds a different variable.
Each rows holds a different observation.
In this “tidy data” format, we can represent any Dataset and
DataArray in terms of DataFrame and
Series, respectively (and vice-versa). The representation
works by flattening non-coordinates to 1D, and turning the tensor product of
coordinate indexes into a pandas.MultiIndex.
Dataset and DataFrame#
To convert any dataset to a DataFrame in tidy form, use the
Dataset.to_dataframe() method:
In [1]: ds = xr.Dataset(
...: {"foo": (("x", "y"), np.random.randn(2, 3))},
...: coords={
...: "x": [10, 20],
...: "y": ["a", "b", "c"],
...: "along_x": ("x", np.random.randn(2)),
...: "scalar": 123,
...: },
...: )
...:
In [2]: ds
Out[2]:
<xarray.Dataset>
Dimensions: (x: 2, y: 3)
Coordinates:
* x (x) int64 10 20
* y (y) <U1 'a' 'b' 'c'
along_x (x) float64 0.1192 -1.044
scalar int64 123
Data variables:
foo (x, y) float64 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732
In [3]: df = ds.to_dataframe()
In [4]: df
Out[4]:
foo along_x scalar
x y
10 a 0.469112 0.119209 123
b -0.282863 0.119209 123
c -1.509059 0.119209 123
20 a -1.135632 -1.044236 123
b 1.212112 -1.044236 123
c -0.173215 -1.044236 123
We see that each variable and coordinate in the Dataset is now a column in the
DataFrame, with the exception of indexes which are in the index.
To convert the DataFrame to any other convenient representation,
use DataFrame methods like reset_index(),
stack() and unstack().
For datasets containing dask arrays where the data should be lazily loaded, see the
Dataset.to_dask_dataframe() method.
To create a Dataset from a DataFrame, use the
Dataset.from_dataframe() class method or the equivalent
pandas.DataFrame.to_xarray() method:
In [5]: xr.Dataset.from_dataframe(df)
Out[5]:
<xarray.Dataset>
Dimensions: (x: 2, y: 3)
Coordinates:
* x (x) int64 10 20
* y (y) object 'a' 'b' 'c'
Data variables:
foo (x, y) float64 0.4691 -0.2829 -1.509 -1.136 1.212 -0.1732
along_x (x, y) float64 0.1192 0.1192 0.1192 -1.044 -1.044 -1.044
scalar (x, y) int64 123 123 123 123 123 123
Notice that that dimensions of variables in the Dataset have now
expanded after the round-trip conversion to a DataFrame. This is because
every object in a DataFrame must have the same indices, so we need to
broadcast the data of each array to the full size of the new MultiIndex.
Likewise, all the coordinates (other than indexes) ended up as variables, because pandas does not distinguish non-index coordinates.
DataArray and Series#
DataArray objects have a complementary representation in terms of a
Series. Using a Series preserves the Dataset to
DataArray relationship, because DataFrames are dict-like containers
of Series. The methods are very similar to those for working with
DataFrames:
In [6]: s = ds["foo"].to_series()
In [7]: s
Out[7]:
x y
10 a 0.469112
b -0.282863
c -1.509059
20 a -1.135632
b 1.212112
c -0.173215
Name: foo, dtype: float64
# or equivalently, with Series.to_xarray()
In [8]: xr.DataArray.from_series(s)
Out[8]:
<xarray.DataArray 'foo' (x: 2, y: 3)>
array([[ 0.469, -0.283, -1.509],
[-1.136, 1.212, -0.173]])
Coordinates:
* x (x) int64 10 20
* y (y) object 'a' 'b' 'c'
Both the from_series and from_dataframe methods use reindexing, so they
work even if not the hierarchical index is not a full tensor product:
In [9]: s[::2]
Out[9]:
x y
10 a 0.469112
c -1.509059
20 b 1.212112
Name: foo, dtype: float64
In [10]: s[::2].to_xarray()
Out[10]:
<xarray.DataArray 'foo' (x: 2, y: 3)>
array([[ 0.469, nan, -1.509],
[ nan, 1.212, nan]])
Coordinates:
* x (x) int64 10 20
* y (y) object 'a' 'b' 'c'
Multi-dimensional data#
Tidy data is great, but it sometimes you want to preserve dimensions instead of
automatically stacking them into a MultiIndex.
DataArray.to_pandas() is a shortcut that lets you convert a
DataArray directly into a pandas object with the same dimensionality, if
available in pandas (i.e., a 1D array is converted to a
Series and 2D to DataFrame):
In [11]: arr = xr.DataArray(
....: np.random.randn(2, 3), coords=[("x", [10, 20]), ("y", ["a", "b", "c"])]
....: )
....:
In [12]: df = arr.to_pandas()
In [13]: df
Out[13]:
y a b c
x
10 -0.861849 -2.104569 -0.494929
20 1.071804 0.721555 -0.706771
To perform the inverse operation of converting any pandas objects into a data
array with the same shape, simply use the DataArray
constructor:
In [14]: xr.DataArray(df)
Out[14]:
<xarray.DataArray (x: 2, y: 3)>
array([[-0.862, -2.105, -0.495],
[ 1.072, 0.722, -0.707]])
Coordinates:
* x (x) int64 10 20
* y (y) object 'a' 'b' 'c'
Both the DataArray and Dataset constructors directly convert pandas
objects into xarray objects with the same shape. This means that they
preserve all use of multi-indexes:
In [15]: index = pd.MultiIndex.from_arrays(
....: [["a", "a", "b"], [0, 1, 2]], names=["one", "two"]
....: )
....:
In [16]: df = pd.DataFrame({"x": 1, "y": 2}, index=index)
In [17]: ds = xr.Dataset(df)
In [18]: ds
Out[18]:
<xarray.Dataset>
Dimensions: (dim_0: 3)
Coordinates:
* dim_0 (dim_0) object MultiIndex
* one (dim_0) object 'a' 'a' 'b'
* two (dim_0) int64 0 1 2
Data variables:
x (dim_0) int64 1 1 1
y (dim_0) int64 2 2 2
However, you will need to set dimension names explicitly, either with the
dims argument on in the DataArray constructor or by calling
rename on the new object.
Transitioning from pandas.Panel to xarray#
Panel, pandas’ data structure for 3D arrays, was always a second class
data structure compared to the Series and DataFrame. To allow pandas
developers to focus more on its core functionality built around the
DataFrame, pandas removed Panel in favor of directing users who use
multi-dimensional arrays to xarray.
Xarray has most of Panel’s features, a more explicit API (particularly around
indexing), and the ability to scale to >3 dimensions with the same interface.
As discussed in the data structures section of the docs, there are two primary data structures in
xarray: DataArray and Dataset. You can imagine a DataArray as a
n-dimensional pandas Series (i.e. a single typed array), and a Dataset
as the DataFrame equivalent (i.e. a dict of aligned DataArray objects).
So you can represent a Panel, in two ways:
As a 3-dimensional
DataArray,Or as a
Datasetcontaining a number of 2-dimensional DataArray objects.
Let’s take a look:
In [19]: data = np.random.RandomState(0).rand(2, 3, 4)
In [20]: items = list("ab")
In [21]: major_axis = list("mno")
In [22]: minor_axis = pd.date_range(start="2000", periods=4, name="date")
With old versions of pandas (prior to 0.25), this could stored in a Panel:
In [23]: pd.Panel(data, items, major_axis, minor_axis)
Out[23]:
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 3 (major_axis) x 4 (minor_axis)
Items axis: a to b
Major_axis axis: m to o
Minor_axis axis: 2000-01-01 00:00:00 to 2000-01-04 00:00:00
To put this data in a DataArray, write:
In [24]: array = xr.DataArray(data, [items, major_axis, minor_axis])
In [25]: array
Out[25]:
<xarray.DataArray (dim_0: 2, dim_1: 3, date: 4)>
array([[[0.549, 0.715, 0.603, 0.545],
[0.424, 0.646, 0.438, 0.892],
[0.964, 0.383, 0.792, 0.529]],
[[0.568, 0.926, 0.071, 0.087],
[0.02 , 0.833, 0.778, 0.87 ],
[0.979, 0.799, 0.461, 0.781]]])
Coordinates:
* dim_0 (dim_0) <U1 'a' 'b'
* dim_1 (dim_1) <U1 'm' 'n' 'o'
* date (date) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04
As you can see, there are three dimensions (each is also a coordinate). Two of
the axes of were unnamed, so have been assigned dim_0 and dim_1
respectively, while the third retains its name date.
You can also easily convert this data into Dataset:
In [26]: array.to_dataset(dim="dim_0")
Out[26]:
<xarray.Dataset>
Dimensions: (dim_1: 3, date: 4)
Coordinates:
* dim_1 (dim_1) <U1 'm' 'n' 'o'
* date (date) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04
Data variables:
a (dim_1, date) float64 0.5488 0.7152 0.6028 ... 0.3834 0.7917 0.5289
b (dim_1, date) float64 0.568 0.9256 0.07104 ... 0.7992 0.4615 0.7805
Here, there are two data variables, each representing a DataFrame on panel’s
items axis, and labeled as such. Each variable is a 2D array of the
respective values along the items dimension.
While the xarray docs are relatively complete, a few items stand out for Panel users:
A DataArray’s data is stored as a numpy array, and so can only contain a single type. As a result, a Panel that contains
DataFrameobjects with multiple types will be converted todtype=object. ADatasetof multipleDataArrayobjects each with its own dtype will allow original types to be preserved.Indexing is similar to pandas, but more explicit and leverages xarray’s naming of dimensions.
Because of those features, making much higher dimensional data is very practical.
Variables in
Datasetobjects can use a subset of its dimensions. For example, you can have one dataset with Person x Score x Time, and another with Person x Score.You can use coordinates are used for both dimensions and for variables which _label_ the data variables, so you could have a coordinate Age, that labelled the Person dimension of a Dataset of Person x Score x Time.
While xarray may take some getting used to, it’s worth it! If anything is unclear, please post an issue on GitHub or StackOverflow, and we’ll endeavor to respond to the specific case or improve the general docs.