Testing your code

Hypothesis testing

Note

Testing with hypothesis is a fairly advanced topic. Before reading this section it is recommended that you take a look at our guide to xarray’s Data Structures, are familiar with conventional unit testing in pytest, and have seen the hypothesis library documentation.

The hypothesis library is a powerful tool for property-based testing. Instead of writing tests for one example at a time, it allows you to write tests parameterized by a source of many dynamically generated examples. For example you might have written a test which you wish to be parameterized by the set of all possible integers via hypothesis.strategies.integers().

Property-based testing is extremely powerful, because (unlike more conventional example-based testing) it can find bugs that you did not even think to look for!

Strategies

Each source of examples is called a “strategy”, and xarray provides a range of custom strategies which produce xarray data structures containing arbitrary data. You can use these to efficiently test downstream code, quickly ensuring that your code can handle xarray objects of all possible structures and contents.

These strategies are accessible in the xarray.testing.strategies module, which provides

These build upon the numpy and array API strategies offered in hypothesis.extra.numpy and hypothesis.extra.array_api:

In [1]: import hypothesis.extra.numpy as npst

Generating Examples

To see an example of what each of these strategies might produce, you can call one followed by the .example() method, which is a general hypothesis method valid for all strategies.

In [2]: import xarray.testing.strategies as xrst

In [3]: xrst.variables().example()
Out[3]: 
<xarray.Variable (3qŻ: 4)> Size: 32B
array([-2.160e+263,  2.000e+000,  1.572e+308, -6.104e-005])

In [4]: xrst.variables().example()
Out[4]: 
<xarray.Variable (Ĉęþ: 5)> Size: 40B
array([52103, 44314, 39805, 23329, 39805], dtype=uint64)
Attributes: (12/21)
    Ł:        True
    őàçı:     ũſ³
    ħžĉ1D:    None
    õjřĠ:    ĎŻž
    żSä:      ŲŻŀſŲ
    Ď:        None
    ...       ...
    y:        None
    óĉVí:     True
    żŎ:       [ True  True]
    ŗśńÃü:    Ł
    ŹīŦ:      None
    Ĩëëřſ:    None

In [5]: xrst.variables().example()
Out[5]: 
<xarray.Variable (çŽ: 1)> Size: 8B
array([-1810044441])
Attributes:
    127:      True
    :         False
    Ġęďž:     [62914 'NaT']
    Ż:        False
    żżŻŁź:    True

You can see that calling .example() multiple times will generate different examples, giving you an idea of the wide range of data that the xarray strategies can generate.

In your tests however you should not use .example() - instead you should parameterize your tests with the hypothesis.given() decorator:

In [6]: from hypothesis import given
In [7]: @given(xrst.variables())
   ...: def test_function_that_acts_on_variables(var):
   ...:     assert func(var) == ...
   ...: 

Chaining Strategies

Xarray’s strategies can accept other strategies as arguments, allowing you to customise the contents of the generated examples.

# generate a Variable containing an array with a complex number dtype, but all other details still arbitrary
In [8]: from hypothesis.extra.numpy import complex_number_dtypes

In [9]: xrst.variables(dtype=complex_number_dtypes()).example()
Out[9]: 
<xarray.Variable (Ė: 1)> Size: 16B
array([2.22e-16+nanj], dtype='>c16')
Attributes:
    ůĺż2ê:    {'œķſăt': False, 'ţſsĖP': 'ċ', '9': None, '': None, 'äàªĴ': True}
    Ĺ:        {}

This also works with custom strategies, or strategies defined in other packages. For example you could imagine creating a chunks strategy to specify particular chunking patterns for a dask-backed array.

Fixing Arguments

If you want to fix one aspect of the data structure, whilst allowing variation in the generated examples over all other aspects, then use hypothesis.strategies.just().

In [10]: import hypothesis.strategies as st

# Generates only variable objects with dimensions ["x", "y"]
In [11]: xrst.variables(dims=st.just(["x", "y"])).example()
Out[11]: 
<xarray.Variable (x: 2, y: 1)> Size: 2B
array([[0],
       [0]], dtype=int8)

(This is technically another example of chaining strategies - hypothesis.strategies.just() is simply a special strategy that just contains a single example.)

To fix the length of dimensions you can instead pass dims as a mapping of dimension names to lengths (i.e. following xarray objects’ .sizes() property), e.g.

# Generates only variables with dimensions ["x", "y"], of lengths 2 & 3 respectively
In [12]: xrst.variables(dims=st.just({"x": 2, "y": 3})).example()
Out[12]: 
<xarray.Variable (x: 2, y: 3)> Size: 12B
array([[ 13839,  -3238,   5247],
       [-31491,  17725,  23562]], dtype=int16)
Attributes:
    ű:        ĠĄĈķõ

You can also use this to specify that you want examples which are missing some part of the data structure, for instance

# Generates a Variable with no attributes
In [13]: xrst.variables(attrs=st.just({})).example()
Out[13]: 
<xarray.Variable (żżHſÈ: 5)> Size: 40B
array([36225,   -47,  -211,   205, 38668])

Through a combination of chaining strategies and fixing arguments, you can specify quite complicated requirements on the objects your chained strategy will generate.

In [14]: fixed_x_variable_y_maybe_z = st.fixed_dictionaries(
   ....:     {"x": st.just(2), "y": st.integers(3, 4)}, optional={"z": st.just(2)}
   ....: )
   ....: 

In [15]: fixed_x_variable_y_maybe_z.example()
Out[15]: {'x': 2, 'y': 3, 'z': 2}

In [16]: special_variables = xrst.variables(dims=fixed_x_variable_y_maybe_z)

In [17]: special_variables.example()
Out[17]: 
<xarray.Variable (x: 2, y: 3, z: 2)> Size: 24B
array([[[19717,  7507],
        [28917, 49356],
        [65180, 48163]],

       [[ 8366, 19108],
        [40235, 62670],
        [48759,  4316]]], dtype=uint16)
Attributes: (12/15)
    :         True
    ĂjĥBŻ:    AččªÎ
    ŰŽ:       None
    ſCaÛŋ:    ö
    ſçĚ2x:    True
    ķw:       
    ...       ...
    øċŽŭž:    řĻĹŃň
    ĠŰvŻ:     ũ
    ŷŵ:       RŜ0ż
    Ő:        False
    VfŽåş:    None
    ŽDijŻÒ:    False

In [18]: special_variables.example()
Out[18]: 
<xarray.Variable (x: 2, y: 4, z: 2)> Size: 128B
array([[[-65537, -65537],
        [-65537, -65537],
        [-65537, -65537],
        [-65537, -65537]],

       [[-65537, -65537],
        [     3, -65537],
        [-65537, -65537],
        [-65537, -65537]]])
Attributes:
    ĎúÇŽ:     None
    :         None
    ūĒP¹þ:    [['NaT' 'NaT']\n ['NaT'   120]]

Here we have used one of hypothesis’ built-in strategies hypothesis.strategies.fixed_dictionaries() to create a strategy which generates mappings of dimension names to lengths (i.e. the size of the xarray object we want). This particular strategy will always generate an x dimension of length 2, and a y dimension of length either 3 or 4, and will sometimes also generate a z dimension of length 2. By feeding this strategy for dictionaries into the dims argument of xarray’s variables() strategy, we can generate arbitrary Variable objects whose dimensions will always match these specifications.

Generating Duck-type Arrays

Xarray objects don’t have to wrap numpy arrays, in fact they can wrap any array type which presents the same API as a numpy array (so-called “duck array wrapping”, see wrapping numpy-like arrays).

Imagine we want to write a strategy which generates arbitrary Variable objects, each of which wraps a sparse.COO array instead of a numpy.ndarray. How could we do that? There are two ways:

1. Create a xarray object with numpy data and use the hypothesis’ .map() method to convert the underlying array to a different type:

In [19]: import sparse
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[19], line 1
----> 1 import sparse

ModuleNotFoundError: No module named 'sparse'
In [20]: def convert_to_sparse(var):
   ....:     return var.copy(data=sparse.COO.from_numpy(var.to_numpy()))
   ....: 
In [21]: sparse_variables = xrst.variables(dims=xrst.dimension_names(min_dims=1)).map(
   ....:     convert_to_sparse
   ....: )
   ....: 

In [22]: sparse_variables.example()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[22], line 1
----> 1 sparse_variables.example()

File /usr/lib/python3/dist-packages/hypothesis/strategies/_internal/strategies.py:348, in SearchStrategy.example(self)
    336 @given(self)
    337 @settings(
    338     database=None,
   (...)
    344 )
    345 def example_generating_inner_function(ex):
    346     self.__examples.append(ex)
--> 348 example_generating_inner_function()
    349 shuffle(self.__examples)
    350 return self.__examples.pop()

File /usr/lib/python3/dist-packages/hypothesis/strategies/_internal/strategies.py:337, in SearchStrategy.example.<locals>.example_generating_inner_function()
    332 from hypothesis.core import given
    334 # Note: this function has a weird name because it might appear in
    335 # tracebacks, and we want users to know that they can ignore it.
    336 @given(self)
--> 337 @settings(
    338     database=None,
    339     max_examples=100,
    340     deadline=None,
    341     verbosity=Verbosity.quiet,
    342     phases=(Phase.generate,),
    343     suppress_health_check=list(HealthCheck),
    344 )
    345 def example_generating_inner_function(ex):
    346     self.__examples.append(ex)
    348 example_generating_inner_function()

    [... skipping hidden 1 frame]

Cell In[20], line 2, in convert_to_sparse(var)
      1 def convert_to_sparse(var):
----> 2     return var.copy(data=sparse.COO.from_numpy(var.to_numpy()))

NameError: name 'sparse' is not defined
while generating 'ex' from variables(dims=lists(text(alphabet=characters(max_codepoint=383, categories=['L', 'N']), min_size=1, max_size=5), min_size=1, max_size=3, unique=True)).map(convert_to_sparse)

In [23]: sparse_variables.example()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[23], line 1
----> 1 sparse_variables.example()

File /usr/lib/python3/dist-packages/hypothesis/strategies/_internal/strategies.py:348, in SearchStrategy.example(self)
    336 @given(self)
    337 @settings(
    338     database=None,
   (...)
    344 )
    345 def example_generating_inner_function(ex):
    346     self.__examples.append(ex)
--> 348 example_generating_inner_function()
    349 shuffle(self.__examples)
    350 return self.__examples.pop()

File /usr/lib/python3/dist-packages/hypothesis/strategies/_internal/strategies.py:337, in SearchStrategy.example.<locals>.example_generating_inner_function()
    332 from hypothesis.core import given
    334 # Note: this function has a weird name because it might appear in
    335 # tracebacks, and we want users to know that they can ignore it.
    336 @given(self)
--> 337 @settings(
    338     database=None,
    339     max_examples=100,
    340     deadline=None,
    341     verbosity=Verbosity.quiet,
    342     phases=(Phase.generate,),
    343     suppress_health_check=list(HealthCheck),
    344 )
    345 def example_generating_inner_function(ex):
    346     self.__examples.append(ex)
    348 example_generating_inner_function()

    [... skipping hidden 1 frame]

Cell In[20], line 2, in convert_to_sparse(var)
      1 def convert_to_sparse(var):
----> 2     return var.copy(data=sparse.COO.from_numpy(var.to_numpy()))

NameError: name 'sparse' is not defined
while generating 'ex' from variables(dims=lists(text(alphabet=characters(max_codepoint=383, categories=['L', 'N']), min_size=1, max_size=5), min_size=1, max_size=3, unique=True)).map(convert_to_sparse)
  1. Pass a function which returns a strategy which generates the duck-typed arrays directly to the array_strategy_fn argument of the xarray strategies:

In [24]: def sparse_random_arrays(shape: tuple[int]) -> sparse._coo.core.COO:
   ....:     """Strategy which generates random sparse.COO arrays"""
   ....:     if shape is None:
   ....:         shape = npst.array_shapes()
   ....:     else:
   ....:         shape = st.just(shape)
   ....:     density = st.integers(min_value=0, max_value=1)
   ....:     return st.builds(sparse.random, shape=shape, density=density)
   ....: 
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[24], line 1
----> 1 def sparse_random_arrays(shape: tuple[int]) -> sparse._coo.core.COO:
      2     """Strategy which generates random sparse.COO arrays"""
      3     if shape is None:

NameError: name 'sparse' is not defined

In [25]: def sparse_random_arrays_fn(
   ....:     *, shape: tuple[int, ...], dtype: np.dtype
   ....: ) -> st.SearchStrategy[sparse._coo.core.COO]:
   ....:     return sparse_random_arrays(shape=shape)
   ....: 
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[25], line 3
      1 def sparse_random_arrays_fn(
      2     *, shape: tuple[int, ...], dtype: np.dtype
----> 3 ) -> st.SearchStrategy[sparse._coo.core.COO]:
      4     return sparse_random_arrays(shape=shape)

NameError: name 'sparse' is not defined
In [26]: sparse_random_variables = xrst.variables(
   ....:     array_strategy_fn=sparse_random_arrays_fn, dtype=st.just(np.dtype("float64"))
   ....: )
   ....: 
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[26], line 2
      1 sparse_random_variables = xrst.variables(
----> 2     array_strategy_fn=sparse_random_arrays_fn, dtype=st.just(np.dtype("float64"))
      3 )

NameError: name 'sparse_random_arrays_fn' is not defined

In [27]: sparse_random_variables.example()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[27], line 1
----> 1 sparse_random_variables.example()

NameError: name 'sparse_random_variables' is not defined

Either approach is fine, but one may be more convenient than the other depending on the type of the duck array which you want to wrap.

Compatibility with the Python Array API Standard

Xarray aims to be compatible with any duck-array type that conforms to the Python Array API Standard (see our docs on Array API Standard support).

Warning

The strategies defined in testing.strategies are not guaranteed to use array API standard-compliant dtypes by default. For example arrays with the dtype np.dtype('float16') may be generated by testing.strategies.variables() (assuming the dtype kwarg was not explicitly passed), despite np.dtype('float16') not being in the array API standard.

If the array type you want to generate has an array API-compliant top-level namespace (e.g. that which is conventionally imported as xp or similar), you can use this neat trick:

In [28]: from numpy import array_api as xp  # available in numpy 1.26.0

In [29]: from hypothesis.extra.array_api import make_strategies_namespace

In [30]: xps = make_strategies_namespace(xp)

In [31]: xp_variables = xrst.variables(
   ....:     array_strategy_fn=xps.arrays,
   ....:     dtype=xps.scalar_dtypes(),
   ....: )
   ....: 

In [32]: xp_variables.example()
Out[32]: 
<xarray.Variable (ŻŒđē: 6, Í: 5, w: 6)> Size: 180B
Array([[[130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130]],

       [[130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130]],

       [[130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130]],

       [[130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130]],

       [[130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130]],

       [[130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130],
        [130, 130, 130, 130, 130, 130]]], dtype=uint8)
Attributes:
    ĂũºêM:    [['1969-12-31T20:05:19' '1970-04-13T09:40:20']]
    żÑĽ:      [6193]
    ʼnʼnę4ſ:    [[       '1969-12-31T05:47:43']\n ['-2283412285-02-07T11:07:44']]

Another array API-compliant duck array library would replace the import, e.g. import cupy as cp instead.

Testing over Subsets of Dimensions

A common task when testing xarray user code is checking that your function works for all valid input dimensions. We can chain strategies to achieve this, for which the helper strategy unique_subset_of() is useful.

It works for lists of dimension names

In [33]: dims = ["x", "y", "z"]

In [34]: xrst.unique_subset_of(dims).example()
Out[34]: ['y', 'z']

In [35]: xrst.unique_subset_of(dims).example()
Out[35]: ['y']

as well as for mappings of dimension names to sizes

In [36]: dim_sizes = {"x": 2, "y": 3, "z": 4}

In [37]: xrst.unique_subset_of(dim_sizes).example()
Out[37]: {'y': 3, 'x': 2, 'z': 4}

In [38]: xrst.unique_subset_of(dim_sizes).example()
Out[38]: {'z': 4, 'y': 3}

This is useful because operations like reductions can be performed over any subset of the xarray object’s dimensions. For example we can write a pytest test that tests that a reduction gives the expected result when applying that reduction along any possible valid subset of the Variable’s dimensions.

import numpy.testing as npt


@given(st.data(), xrst.variables(dims=xrst.dimension_names(min_dims=1)))
def test_mean(data, var):
    """Test that the mean of an xarray Variable is always equal to the mean of the underlying array."""

    # specify arbitrary reduction along at least one dimension
    reduction_dims = data.draw(xrst.unique_subset_of(var.dims, min_size=1))

    # create expected result (using nanmean because arrays with Nans will be generated)
    reduction_axes = tuple(var.get_axis_num(dim) for dim in reduction_dims)
    expected = np.nanmean(var.data, axis=reduction_axes)

    # assert property is always satisfied
    result = var.mean(dim=reduction_dims).data
    npt.assert_equal(expected, result)