Overview

ReFrame is a powerful framework for writing system regression tests and benchmarks, specifically targeted to HPC systems. The goal of the framework is to abstract away the complexity of the interactions with the system, separating the logic of a test from the low-level details, which pertain to the system configuration and setup. This allows users to write portable tests in a declarative way that describes only the test's functionality.

Tests in ReFrame are simple Python classes that specify the basic variables and parameters of the test. ReFrame offers an intuitive and very powerful syntax that allows users to create test libraries, test factories, as well as complete test workflows using other tests as fixtures. ReFrame will load the tests and send them down a well-defined pipeline that will execute them in parallel. The stages of this pipeline take care of all the system interaction details, such as programming environment switching, compilation, job submission, job status query, sanity checking and performance assessment.

ReFrame also supports storing the test results in a database allowing for later inspection, basic analytics and performance comparisons.

Please visit the project's documentation page for all the details!

Installation

ReFrame is fairly easy to install as PyPI package.

# Fetch uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install ReFrame
uv tool install reframe-hpc

# Check the installation
reframe -V

This will place the reframe executable under $HOME/.local/bin and the package under $HOME/.local/share/uv/tools/reframe-hpc. To make available the manpage and the shell completions add the following lines in your shell's profile script:

Bash/Zsh

Add the following lines to your $HOME/.profile:

export MANPATH=${HOME}/.local/share/uv/tools/reframe-hpc/share/man:${MANPATH}:
source ${HOME}/.local/share/uv/tools/reframe-hpc/share/bash-completion/completions/reframe

Fish

Add the following lines to your $HOME/.config/fish/config.fish:

set -apgx MANPATH ${HOME}/.local/share/uv/tools/reframe-hpc/share/man ""
source ${HOME}/.local/share/uv/tools/reframe-hpc/share/fish/vendor_completions.d/reframe.fish

NOTE: Using uv is not required to install ReFrame. You can use any modern Python build system that recognizes the pyproject.toml file.

Multi-architecture installations on shared filesystem

If you plan to install ReFrame for multiple platforms in a shared installation, you should make sure each installation resides in a different prefix. You can achieve this with uv as follows:

export UV_TOOL_BIN_DIR=$HOME/.local/$(uname -m)/bin
export UV_TOOL_DIR=$HOME/.local/$(uname -m)/share/uv/tools
uv tool install reframe-hpc
export PATH=$UV_TOOL_BIN_DIR:$PATH
reframe -V

Alternatively, you can use ephemeral venvs and let uv handle them with uvx:

uvx reframe --version

This will pull ReFrame's dependencies and run it. It also caches them, so that the next time you invoke it, it will not download and install them again. The only "downside" of this method is that you have to always invoke ReFrame through uvx, as opposed to the uv tool install method, where the reframe execcutable is installed in a standard path.

Running from source

If you want to run the latest ReFrame directly from the repo, you can simply clone the repo and uv run ReFrame:

git clone https://github.com/reframe-hpc/reframe.git
cd reframe
uv run reframe --version

Running the unit tests

To run the the framework's unit tests use the following command:

uv sync --group dev
uv run ./test_reframe.py -v

NOTE: Unit tests require a POSIX-compliant C compiler (available through the cc command), as well as the make utility.

Building the documentation locally

You may also build the documentation locally:

uv sync --group docs
uv run make -C docs

For viewing it, you may do the following:

cd docs/html
python3 -m http.server

The documentation is now up on localhost:8000, where you can navigate with your browser.

Public test repositories

The ReFrame HPC community Github page provides mirror forks of interesting ReFrame test repositories maintained by various sites or projects. You can use those tests as an additional guidance to implement your own tests.

If you maintain a public test repository and you would like it to be listed in the community page, feel free to open an issue or contact us through Slack.

Contact

You can get in contact with the ReFrame community in the following ways:

Slack

Please join the community's Slack channel for keeping up with the latest news about ReFrame, posting questions and, generally getting in touch with other users and the developers.

NOTE: Slack invite links expire for security reasons. Although we update them regularly, if you encounter problems joining the community Slack channel, please open an issue.

Contributing back

ReFrame is an open-source project and we welcome and encourage contributions! Check out our Contribution Guide here.

Citing ReFrame

You can cite ReFrame in publications as follows:

Vasileios Karakasis et al. "Enabling Continuous Testing of HPC Systems Using ReFrame". In: Tools and Techniques for High Performance Computing. HUST - Annual Workshop on HPC User Support Tools (Denver, Colorado, USA, Nov. 17–18, 2019). Ed. by Guido Juckeland and Sunita Chandrasekaran. Vol. 1190. Communications in Computer and Information Science. Cham, Switzerland: Springer International Publishing, Mar. 2020, pp. 49–68. isbn: 978-3-030-44728-1. doi: 10.1007/978-3-030-44728-1_3.

The corresponding BibTeX entry is the following:

@InProceedings{karakasis20a,
  author     = {Karakasis, Vasileios and Manitaras, Theofilos and Rusu, Victor Holanda and
                Sarmiento-P{\'e}rez, Rafael and Bignamini, Christopher and Kraushaar, Matthias and
                Jocksch, Andreas and Omlin, Samuel and Peretti-Pezzi, Guilherme and
                Augusto, Jo{\~a}o P. S. C. and Friesen, Brian and He, Yun and Gerhardt, Lisa and
                Cook, Brandon and You, Zhi-Qiang and Khuvis, Samuel and Tomko, Karen},
  title      = {Enabling Continuous Testing of {HPC} Systems Using {ReFrame}},
  booktitle  = {Tools and Techniques for High Performance Computing},
  editor     = {Juckeland, Guido and Chandrasekaran, Sunita},
  year       = {2020},
  month      = mar,
  series     = {Communications in Computer and Information Science},
  volume     = {1190},
  pages      = {49--68},
  address    = {Cham, Switzerland},
  publisher  = {Springer International Publishing},
  doi        = {10.1007/978-3-030-44728-1_3},
  venue      = {Denver, Colorado, USA},
  eventdate  = {2019-11-17/2019-11-18},
  eventtitle = {{HUST} - Annual Workshop on {HPC} User Support Tools},
  isbn       = {978-3-030-44728-1},
  issn       = {1865-0937},
}

For a complete list of publications and talks about ReFrame, please refer to the latest documentation page here.