Tutorials 3 and 4 are currently not included in the tutorial test suite.

[sbrandstaeter]

Description of the Issue:

Summary

Tutorials 3 and 4 are currently not included in the tutorial test suite.

Running:

pytest -m "tutorial_tests"

does not execute tests for:

tutorials/3_orchestrating_4c_simulations/3_orchestrating_4c_simulations.ipynb
tutorials/4_quantifying_uncertainty_due_to_heterogeneous_material_fields/4_quantifying_uncertainty_due_to_heterogeneous_material_fields.ipynb

I think these tutorials should be covered by the test suite as well, but they require 4C, so they should probably carry an additional marker and only run when explicitly requested.

Current behavior

The tutorial test collection currently only looks for notebooks directly under tutorials/:

tests/tutorial_tests/test_tutorials.py#L25-L35

for patch in sorted(Path("tutorials").glob("*.ipynb"))

Because tutorials 3 and 4 live in subdirectories, they are not discovered
At the same time, the current marker assignment only gives tutorial tests the tutorial_tests marker:

tests/conftest.py#L128-L133

And the GitHub workflow runs tutorials via:

.github/workflows/local_testsuite.yml#L78-L82

pytest -v -m "tutorial_tests"

Expected behavior

Tutorials 3 and 4 should also be part of the tutorial test suite, but only when 4C support is explicitly requested.

Proposed Solution:

Proposed solution

Change tutorial notebook discovery to recurse into subdirectories, e.g. replace:

Path("tutorials").glob("*.ipynb")

with something like:

Path("tutorials").rglob("*.ipynb")

The following needs refining but might point towards a solution:

Mark tutorials 3 and 4 with an additional marker, for example 4C_integration_tests, in addition to tutorial_tests.

Keep the default tutorial test run lightweight by not running these 4C tutorials with:

pytest -m "tutorial_tests and not 4C_integration_tests"

Run them only when explicitly requested, e.g.:

pytest -m "tutorial_tests and 4C_integration_tests"

Action Items:

No response

Related Issues:

No response

Interested Parties:

@reginabuehler @BishrMaradni

[leahaeusel]

As brought up by @BishrMaradni, Tutorial 5 introduced by #265 also needs its own pytest marker since it can only be executed when a remote computing resource is available, which is only the case for our GitLab runner.