Skip to content

docs: add experiment run guide, citation, and badges#63

Merged
Colinho22 merged 1 commit into
mainfrom
feat-repo-documentation-improvement
Jun 13, 2026
Merged

docs: add experiment run guide, citation, and badges#63
Colinho22 merged 1 commit into
mainfrom
feat-repo-documentation-improvement

Conversation

@Colinho22

@Colinho22 Colinho22 commented Jun 13, 2026

Copy link
Copy Markdown
Owner

Here's the PR description for the doc branch, as clean copy-paste markdown:


Summary

Documentation pass toward v1.0.0, addressing the reviewer's replicability recommendation (Empfehlung #1). Adds a high-level "Running the experiment" guide to the README, honest status badges, and a CITATION.cff for the thesis artifact. Closes #26.

Phrased for v1.0.0, which ships without results — the guide describes how to produce results, not reproduce them. The replication framing (re-running against published numbers) follows in v1.0.1 once the experiment data and polished viz land.

What changed

README

  • New "Running the experiment" section — the main entry point, in priority order: prerequisites → install → configure keys → validate with a small run → run the full matrix → analyse → explore in the dashboard. Each step gives both the local and Docker command.
  • Reproducibility audit trail note — points at the run_environments table (per-run OS, arch, Python, library versions, git commit, image digest), which is what makes future replication diagnosable.
  • Provider names link to each API's docs for key setup (Anthropic, OpenAI, Mistral, Gemini, DeepSeek).
  • Platform note: the local path is macOS-tested; Docker runs Linux in-container and is the platform-independent route (relevant to the pending Windows work, fix: make check_mermaid_valid work on Windows (replace /dev/stdin + /dev/null) #44).
  • Demoted tests to a "Local development" section at the end (contributor concern, not the artifact's purpose).

Badges

  • Added honest, verifiable badges alongside the existing CI + CodeRabbit: License (MIT), Python 3.11, Ruff, pre-commit.
  • Deliberately omitted PyPI/conda (not published) and coverage (no Codecov wired) — no badges that 404 or show "unknown".

CITATION.cff

  • GitHub-native citation file (renders the "Cite this repository" button). Valid now; doi and date-released are left commented out.

Scope

  • In scope: one high-level README section, badges, citation file.
  • Out of scope (deferred to the July polish / a later docs issue): troubleshooting, screenshots, full CLI reference, and a potential GitHub Pages site (v1.0.2, once viz docs + metric definitions justify it).

Verification

  • All documented commands and entry points exist (maestro.run, maestro.analysis, viz/app.py; --strategy/--tier/--repeats are real flags).
  • All README internal links resolve (LICENSE, CITATION.cff, .env.template, .pre-commit-config.yaml).
  • CITATION.cff validates as well-formed CFF with all required keys.

Summary by CodeRabbit

  • Documentation
    • Added citation metadata file for proper software attribution.
    • Expanded README with comprehensive workflow guide covering installation, configuration, experiment execution with resumability, result analysis, dashboard access, and reproducibility tracking.

@Colinho22 Colinho22 added this to the 🧪 Experimental Artifact milestone Jun 13, 2026
@Colinho22 Colinho22 self-assigned this Jun 13, 2026
@Colinho22 Colinho22 added the documentation Improvements or additions to documentation label Jun 13, 2026
@coderabbitai

coderabbitai Bot commented Jun 13, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b2f2b07b-b7ad-4d26-8349-8fca407af8c7

📥 Commits

Reviewing files that changed from the base of the PR and between 0771ffe and b924d4f.

📒 Files selected for processing (2)
  • CITATION.cff
  • README.md
✅ Files skipped from review due to trivial changes (2)
  • CITATION.cff
  • README.md

📝 Walkthrough

Walkthrough

This PR adds citation metadata and comprehensively rewrites the README to guide users through experiment replication. A new CITATION.cff file provides structured citation information. The README now documents prerequisites, installation, environment configuration, a validation run, full experiment matrix execution, analysis, and dashboard access, plus reproducibility tracking via environment snapshots. Local development instructions are expanded with test and format commands.

Changes

Documentation and Citation Setup

Layer / File(s) Summary
Citation metadata and project overview
CITATION.cff, README.md
CITATION.cff specifies software metadata (title, abstract, author, version, license, repository). README header and project tagline are consolidated.
Experiment execution and reproducibility guide
README.md
"Running the experiment" section documents prerequisites (Python 3.11, API keys, optional mmdc and Docker), installation, .env setup, validation and full matrix runs with resumability and SQLite output, analysis invocation, and dashboard access. "Reproducibility audit trail" section describes per-invocation environment snapshotting into run_environments table including git commit and Docker image digest.
Local development and citation reference
README.md
"Local development" expanded with dev extras, pytest, ruff check/format, and pre-commit setup. New "Citing" section points to CITATION.cff.

🎯 1 (Trivial) | ⏱️ ~3 minutes

🐰 A citation file and a guide so clear,
For replication and reproducibility here!
From running tests to dashing through the graphs,
The README now tells the complete tale with tags. 🏷️

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main changes: adding an experiment run guide, citation metadata, and documentation badges to the README and new CITATION.cff file.
Linked Issues check ✅ Passed The PR meets all coding requirements from issue #26: README section with prerequisites, install, .env config, validation run command, full matrix, analysis, dashboard, run_environments link, and badges (license, Python 3.11, Ruff, pre-commit).
Out of Scope Changes check ✅ Passed All changes are in scope: README experiment guide, CITATION.cff file, and badges as specified in #26. Troubleshooting, screenshots, and full CLI reference are deferred as planned.
Docstring Coverage (Src Only) ✅ Passed For changed src files (experiment_config.py, metrics.py, mermaid_render.py), AST scan found 20/20 public modules/classes/functions have docstrings (100%).

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat-repo-documentation-improvement

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Dockerfile`:
- Line 10: Create and switch to a non-root user in the Dockerfile before running
mmdc/Chromium: add a dedicated unprivileged user (e.g., with useradd/groupadd or
adduser), set HOME, chown/chmod any directories or files that the app or
Chromium needs (cache, /tmp, .config, mmdc output dirs), and add a USER
directive so mmdc/Chromium runs as that user instead of root; ensure any install
steps that require root remain before the USER switch and that the runtime
invocation of mmdc/Chromium uses the non-root user to remove --no-sandbox
exposure.
- Line 10: Replace the mutable base image tag in the FROM line (currently "FROM
python:3.11-slim") with a digest-pinned image (e.g.,
"python:3.11-slim@sha256:<digest>") or otherwise resolve and lock the exact
image digest to ensure immutability; in the apt-get install block pin each
system package to a specific version or switch the APT source to a dated
snapshot (or use apt-get install package=version for each package) and add
apt-get update && apt-get install with explicit package versions and apt-get
clean steps so package versions are reproducible; keep the existing mermaid-cli
pin (mermaid-cli@11.4.2) as-is.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 679208cd-128e-40eb-a023-3b62700ae200

📥 Commits

Reviewing files that changed from the base of the PR and between 50c6d8b and 0771ffe.

📒 Files selected for processing (8)
  • CITATION.cff
  • Dockerfile
  • README.md
  • docker-compose.yml
  • out/.gitkeep
  • src/maestro/analysis/metrics.py
  • src/maestro/experiment_config.py
  • src/maestro/viz/mermaid_render.py

Comment thread Dockerfile
@Colinho22 Colinho22 force-pushed the feat-repo-documentation-improvement branch from 0771ffe to f40830f Compare June 13, 2026 06:55
Lead with 'Running the experiment' (prereqs → install → validate → run →
analyse → dashboard, local and Docker commands) and a reproducibility
audit-trail note pointing at run_environments. Phrase it as producing results
(v1.0.0 ships without data; replication framing follows in v1.0.1). Demote tests
to a 'Local development' section (macOS-tested) and note Docker as the
platform-independent route. Link each provider's API docs for key setup. Add
honest status badges (License, Python, Ruff, pre-commit) and a CITATION.cff for
the thesis artifact (DOI and release date left commented out.)
@Colinho22 Colinho22 force-pushed the feat-repo-documentation-improvement branch from f40830f to b924d4f Compare June 13, 2026 07:01
@Colinho22 Colinho22 merged commit 85d511f into main Jun 13, 2026
2 checks passed
@Colinho22 Colinho22 deleted the feat-repo-documentation-improvement branch June 13, 2026 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: high-level replication section in README

1 participant