MPI-BugBench

MPI-BugBench [MBB24] is an benchmark suite to evaluate the classification quality of MPI correctness tools. It covers various error classes in MPI while incorporating a broad range of real-world MPI usage scenarios.

Preparation

For simple reproduction, all workflows are executed within Docker containers.

Build the docker image and run it

docker build -f Dockerfile.<parcoach|itac|must> -t mbb:latest .
docker run -v `pwd`:/MBB -it mbb:latest bash

Generating the codes

MBB uses a generation infrastructure written in Python to generate all test codes and supports different test generation levels (for details, see our paper). To generate the test cases for a given generation level (1-5) run the following commands:

python3 MBB.py -c generate --level 2

or

cd /MBB
tar -xf real_world_data.csv.tar.gz
cd /MBB/scripts
python3 generate.py --generator_dir errors --level 2 --real_world_data ../output.csv

Run a tool on the codes

cd /MBB
python3 MBB.py -c run -x <parcoach|itac|must> -t <timeout in seconds> -l <logs output dir> -n <number of workers>

Example for PAROACH

python3 MBB.py -c run -x parcoach -t 180 -l mylogdir -n 4

Analyzing results

CSV Results

python3 MBB.py -x <parcoach|itac|must> -l <logs output dir> -c csv

Plots

python3 MBB.py -x <parcoach|itac|must> -l <logs output dir> -c plots

References

[MBB24] Tim Jammer, Emmanuelle Saillard, Simon Schwitanski, Joachim Jenke, Radjasouria Vinayagame, Alexander Hück, Christian Bischof. 2024. MPI-BugBench: A Framework for Assessing MPI Correctness Tools]. In EuroMPI'24, September 25-27, 2024, Perth, Australia. Springer.