Pixi dependency management

[sbrandstaeter]

Summary of big changes:

• switch to pixi for dependency management (keep backward compatibility to pip and conda)
• switch to Python 3.12 and bump versions of dependencies
• add official support for macos

Minor changes

• clean up GitHub actions
• switch to pip-licenses to check licenses of all production dependencies (also transitive dependencies)
• fix several pylint errors that popped up with the newer version
• fix some warning in the documentation
• added markers for the tutorial tests: tutorials_tests, tutorial_tests_fourc, tutorial_tests_remote
• added a marker for convergence tests
• fix all tests failing due to upgraded dependencies
• clean up dependencies
• enable dependabot automatic updating of github actions
• enable regular automatic upgrading of dependencies
• enable regular automatic upgrading of pixi version
• cleanup of pre-commit hooks: ensure version control in ci and locally, ensure unique configs

Description and Context:
What and Why?

This PR moves the project to pixi as the main dependency and environment manager.

pixi gives us a modern workflow for mixed Conda/PyPI environments:

• Conda and PyPI dependencies can be declared together
• Conda takes precedence where applicable
• PyPI dependencies are resolved via uv under the hood
• environment creation and locking are fast and reproducible

At the same time, this setup does not force users to adopt pixi. QUEENS can still be used through standard Python packaging workflows such as pip install -e ., and pure-PyPI workflows using uv remain possible. Even so, I strongly recommend trying pixi: it is easy to use and very fast.

Reminder:

One major reason for this change is that we should bump the dependencies' versions
This is also done with this PR. Starting with switching to Python 3.12.
The new versions will also require several changes and adaptations to the tests.
One adaptation that was already done is the switch to pip-licenses for checking the licenses of QUEENS' dependencies including transitive dependencies (dependencies of direct dependencies).

What Changed Compared To main

This branch introduces:

• pixi workspace-based dependency and environment management
• a pixi.lock-based reproducible environment workflow
• update to Python 3.12 and bump of many dependency versions
• CI updates for GitHub and GitLab to build and test against pixi-managed environments
• remote environment bootstrapping aligned with pixi
• consistency checks between PEP-style dependency declarations and pixi dependency declarations
• pre-commit and CI checks to prevent those duplicate dependency definitions from drifting apart
• lockfile integrity checks in CI so dependency changes and stale lockfiles are surfaced early

Why We Now Have Duplicate Dependency Definitions

A central part of this setup is that dependencies are now defined in two forms:

1. PEP-compliant package metadata:

• project.dependencies
• dependency-groups
• project.optional-dependencies

2. pixi dependency definitions:

• tool.pixi.dependencies
• tool.pixi.pypi-dependencies
• tool.pixi.feature.*

This duplication is intentional.

The PEP-style declarations are needed for standard Python packaging metadata and pure-PyPI workflows.
The pixi declarations are needed to drive mixed Conda/PyPI environment resolution efficiently and reproducibly.

To make sure these two views do not diverge, this PR adds automated integrity checks in:

• pre-commit
• GitHub Actions

CI / Workflow Improvements

The pipeline now:

• uses pixi-managed environments
• checks dependency declaration integrity
• checks whether dependency-related pyproject.toml changes would require a lockfile update
• fails when dependency declarations changed and pixi.lock is stale

This gives us much better guardrails around environment reproducibility.

What Is Still Missing

One thing not fully supported yet, but straightforward to add, is a pure Conda environment-management workflow for users who want to stay entirely in Conda-style tooling.

The natural path for that is:

• exporting environments from pixi
• using those exported environment files for Conda-only workflows

This should be added next.

Possible Next Steps

Short Usage Tutorials

1. Using pixi

Install pixi, then from the repository root:

Do the following once (per environment):

pixi install --environment queens-dev
pixi run -e queens-dev pip install -e ./ --no-deps

Useful environments currently include:

• default
• all
• dev

Then you to for example run the test suite

pixi run -e dev pytest

Open a shell in an environment:

pixi shell -e dev

Refresh the lockfile after dependency changes:

pixi lock

Use the lockfile strictly:

pixi install --locked --environment dev
pixi run -e dev pip install -e ./ --no-deps

2. Pure PyPI Workflows

These remain supported.

2.1 pip install

python -m venv .venv
source .venv/bin/activate
pip install -e .

This is the classic editable-install workflow.

2.2 uv

uv venv
source .venv/bin/activate
uv pip install -e .

This is a good option for a fast pure-PyPI workflow.

3. Conda via Pixi Export

This is the intended future direction for Conda-only usage.

The idea is:

• define dependencies centrally via pixi
• export a Conda-compatible environment
• use that exported file in Conda-only workflows

That support is not the main path yet, but it can be added next.

Open points

• add support for macos
• fix test in macos
• fix all tests with the new setup
• remove unnecessary explicit dependencies (as analysed by @gilrrei)
• introduce and test pure PyPI environment
• introduce and test "pure" conda environment
• fix warnings in the pipeline
• write documentation on how to install QUEENS (up to three cases?)
• write documentation on how to change (add, remove, upgrade, ...) dependencies
• extract workspace setup in a dedicated GitHub action (reduce code duplication)
• extract GitHub action to run QUEENS tests for specific markers (reduce code duplication)
• fix xml to md report generation (for multiple junit reports)
• enable testing of tutorials using 4C Tutorials 3 and 4 are currently not included in the tutorial test suite. #302
• does this PR justify bumping the Version of QUEENS?
• add changes to the CHANGELOG.md

Follow-ups

• handling of remote environments
• GitLab test pipeline
• add coverage data to GitHub code quality feature
• publish coverage report
• fix coverage report if tests are split (core tests vs 4C tests vs tutorials)
• track progress of mapping tensorflow-probability (tfp-nightly) between conda and PyPi (ubuntu) -> readd default pypi dependency to development queens = {path = "./", editable=True} (alternatively: pixi adds --no-deps feature)
• publish QUEENS on pypi @sbrandstaeter
• publish QUEENS on conda-forge @sbrandstaeter
• automatic updating of dependencies (see e.g. updating lockfile)
• automatic updating of pixi version (see notes on pin your version
• automatically bump version of GitHub actions
• add 4C tests on mac once Add cmake preset to compile 4C on Darwin 4C-multiphysics/4C#1795 is merged
• (optional) support pure Conda environment management via pixi environment export
• allow tutorials to run in colab

Related Issues and Pull Requests

• Closes Conda driven dependency management #297 Tutorials 3 and 4 are currently not included in the tutorial test suite. #302 Find an alternative to liccheck #290
• Related to

Interested Parties

Note: More information on the merge request procedure in QUEENS can be found in the Submit a pull request section in the CONTRIBUTING.md file.

[danielwolff1]

One comment from my side: I have not yet found the time to check all the changes, but I used the new pixi-based workflow today to setup QUEENS on the charon cluster of IMCS and perform some optimisations with it and everything seems to be working seamlessly for me! Thanks a lot again for your effort with this feature @sbrandstaeter 🙏

[Copilot]

Pull request overview

Copilot reviewed 98 out of 101 changed files in this pull request and generated 5 comments.

[leahaeusel]

I finally managed to finish the review. Just a few minor things are left to do, and then we should be ready to finally merge this beast 💪

[sbrandstaeter]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fdd436d0cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[leahaeusel]

🎉