fieldcompare: A Python package for regression testing simulation results

Abstract

In various research areas such as engineering, physics, and mathematics, numerical simulations play an important role. A number of research software simulation frameworks have been established, for instance, Dune (Bastian et al., 2008, 2021), Dumux (Flemisch et al., 2011; Koch et al., 2021), Deal.II (Arndt et al., 2022), FEniCS (A. Logg, 2012; FEniCS, 2023), and VirtualFluids (Kutscher et al., 2022). Numerical software typically has a high inherent complexity as it aims at solving complex physical model equations by using advanced mathematical methods for solving partial differential equations. Beyond this, the model equations often involve parameters that are described by means of empirical constitutive relationships. Thus, a numerical simulation usually brings together various software components: for the domain discretization, the discretization method for the equations, the physics, and a non-linear and/or linear solver to obtain a solution for the discretized equations. While each of these components can be unit tested, it is important to have system tests that verify that a particular type of simulation can be carried out successfully. By successful we mean here that the simulation produces the correct results. As sufficiently complex problems often lack analytical solutions, determining correctness of numerical simulations poses a significant challenge. In the absence of an analytical solution, a common strategy is to use a trusted reference for comparison (e.g., data measured in experiments or results from previous publications). From the perspective of software quality assurance, it suffices to define a reference result as the correct one and continuously verify that the code still reproduces it. In numerical software, such regression tests play a vital role at the level of system tests (Kempf & Koch, 2017). They make sure that developers notice when a certain change to the code affects the results produced by the simulations. Whether the new results are better or worse has to be decided by the developers, and in the case of the former, the reference results may be updated. In order to carry out regression tests, one must be able to detect significant deviations between newly-computed and reference results. What a significant deviation is has to be decided by the developers as well, and adequate tolerances have to be chosen that are big enough to avoid false negatives from machine precision issues, but small enough to ensure that physically relevant deviations in the results are detected. Some numerical software packages as, for instance, DUNE and DuMux (Flemisch et al., 2011), provide mechanisms to detect such deviations. However, the functionality is not provided independent of the frameworks themselves and is therefore only available to their users. Besides this, only those mesh file formats that are used by the frameworks are supported. Very recently, DuMux incorporated fieldcompare into its test suite in place of its in-house solutions.