Code Tests

A number of code (i.e. integration/functional) tests are available in the examples/ directory. They can be used to verify that core functionality, accuracy, and/or correctness of the code has not changed. A small helper script is available to run one or more tests on the local code base, verify the results, and optionally visualize the output.

Although such full tests are the ideal case, simpler compilation tests, which only make sure that the code compiles for a given configuration, are also present to provide some rudimentary coverage even in the absence of proper tests. Such tests can be added in the examples/sanity_checks/compilation_examples/ directory by dropping a Config.sh file in a sub-directory (see also examples/sanity_checks/compilation_examples/README.txt).

Required libraries

The test scripts are written in Python and generally require the libraries numpy, scipy, and h5py to read and verify simulation results, and matplotlib to create plots. The compilation tests examples/sanity_checks/compilation_basic additionally use the PyYAML library.

These libraries can be installed using the pip Python package manager:

pip install numpy
pip install scipy
pip install h5py
pip install matplotlib
pip install pyyaml

Alternatively, the Anaconda Python distribution also contains all of these libraries.

Running the test suite

Execution of the test.sh script in the root directory will run a series of tests as specified (or commented out) therein:

$ ./test.sh

Alternatively, execution of the more advanced test.py script:

$ ./test.py
usage: test.py [-h] [--withvis] [--number-of-tasks NUMBER_OF_TASKS]
               [--number-of-compilers NUMBER_OF_COMPILERS] [--no-cleanup]
               [--print-timestep PRINT_TIMESTEP] [--no-print-output]
               [--print-all-output]
               test_name [test_name ...]

For example, to execute all tests sequentially:

./test.py all

Or, to run only the Kelvin–Helmholtz instability test in 2D, with generation of visualization frames (and optionally a movie if ffmpeg is present):

./test.py --withvis khi_hydro_2d

If visualization is requested, a vis/ subdirectory within the test directory will be made and left for inspection.

Adding a new test

Each test resides in a subdirectory with a _1d, _2d, or _3d suffix, as appropriate. To add a new test, create a subdirectory with the following required files:

Config.sh
param.txt
create.py
check.py

If the test does not generate its initial conditions on the fly, and instead requires pre-made ICs, then the following file should also be present:

ics.hdf5

The configuration and parameter files should set the code execution behavior. For simplicity, disabled configuration options can be excluded (i.e. don’t keep the whole Template-Config.sh file). The create.py file should be a simple Python script, with a mandatory function: create_ics(). This should be invoked if the script is executed from the command line, following the existing examples.

If possible, create_ics() should generate the initial conditions of the test, such that binary files do not have to be added to the repo. In this case, a file with the name ics.hdf5 should be created, and the function should return True, to indicate this. For this purpose, a simple helper function write_ic_file() is available in the file examples/utils.py which will create a HDF5 format ICs file given a dictionary of particle properties (see example below). Otherwise, if using pre-made ICs, the function can simply return immediately.

The Arepo executable is called from the respective run directory, which contains a copy of the example directory with all its content and the output created by the create.py script. Therefore, the relative path to the output directory (containing the initial condition file) during the Arepo run is ./.

In check.py, two functions should be present: verify_result() and visualize_result(). The first should be invoked if the script is executed from the command line, and the script should finish with an exit code 0 if successful, and nonzero otherwise.

The verify_result() function should assess the validity of the output of the test run. It can do so in any way desired, most likely by comparison to an analytical/exact result, or by comparison to a previously saved snapshot of the final state of the test, run on the code base when the test was introduced. It should return a 2-tuple, the first entry being False to indicate test failure, or True to indicate success. The second entry is a list, optionally empty, of string messages giving further information.

The optional visualize_result() function should do any visualization on the output of the run, if possible, putting any results into a vis/ subdirectory of the test, which is always kept for inspection. Two simple helper functions are available, visualize_result_2d() and visualize_result_1d(), which can be called if the test is of a standard type. visualize_result() does not need to return any value.

More precisely, here are the function signatures which the functions required by the test framework must be compatible with:

.. code-block:: python

# create.py: # return value is currently not checked def create_ics(path, filename=’ics.hdf5’) -> Optional[bool]:

…

# check.py: def verify_result(path) -> Tuple[bool, list]:

…

def visualize_result(path, Lx, Ly) -> Any:

…

where path is the path to the run directory.

Tests can also optionally define the following attributes in create.py to influence how the test is run:

.. code-block:: python

# create.py: # enforce a fixed number of MPI tasks to run the test with # (e.g. 1 for serial 1D tests) numTasksMPI = … # plot options which will be used in the parameter file and # passed to visualize_result() Lx = … Ly = ..

For a full working example of a 1D test, which verifies its result based on an exact solution, see examples/shocktube/shocktube_sod_1d/. For an example of a 2D test, which verifies its result based on a previously saved output, see examples/khi_hydro_2d. The easiest way to add a new test is to start from an existing example.

Test considerations

  • If possible, generate the initial conditions for the test on the fly. This avoids binary files in the repo, enables transparency and reproducibility, and makes it possible to modify the test setup or use it in a different context.

  • Tests should be short, as the entire test suite should be run on each commit. Ideally, stick to configurations which can complete on a single core in a few minutes or less, ideally in a few seconds. Memory usage should be minimized.

  • Let us add the other common tests from the various technical Arepo papers as time permits, i.e. 1D acoustic wave, 1D interacting waves/compression, 2D and 3D Sedov-Taylor explosion, 2D Gresho vortex, 2D Yee vortex, 2D and 3D Noh, 2D RT, 2D implosion, 1D and multi-D Evrard collapse, 1D and multi-D Zeldovich pancake, 2D Orszag-Tang, 1D MHD shocktube, magnetic rotor and other MHD tests, 3D turbulence driven boxes. We can also add 3D low resolution versions of canonical simulations: isolated disks, spherical cooling gas clouds, cosmological boxes, etc.

  • Any updates to the framework itself are welcome.