MAESTRO

Multi-Agent Evaluation for Structured Relational Output

Comparing agentic orchestration frameworks for automated relational diagram generation.

---

Running the experiment

The benchmark runs a matrix of inputs × strategies × models × repeats, scores each generated Mermaid diagram against its ground truth, and records every result — plus the runtime environment — in a SQLite database. The steps below run the experiment from a clean checkout.

This is a high-level walkthrough. A detailed guide (troubleshooting, full CLI reference) will follow as the code stabilises.

Prerequisites

• Python 3.11
• API keys for the providers you intend to run — Anthropic, OpenAI, Mistral, Gemini, DeepSeek (see each provider's docs for obtaining a key)
• mmdc (mermaid-cli) for the structural-validity metric — optional locally (the metric is skipped if it is absent), bundled in the Docker image
• Docker (optional) — only if you prefer the container path over a local install

The local install path is tested on macOS. The Docker path runs Linux inside the container, so it is platform-independent and is the recommended route on Windows.

1. Clone and install

git clone https://github.com/Colinho22/maestro.git
cd maestro
pip install -e .            # or: pip install -e ".[dev]" for the test/lint tools

Or build the container, which bundles Python, mermaid-cli, and Chromium:

docker compose build

2. Configure API keys

Copy the template and fill in the keys for the providers you will use:

cp .env.template .env
# edit .env — keys are read from the environment at run time

3. Validate the setup with a small run

A single tier-1 cell confirms the install, keys, and scoring pipeline work before committing to the full matrix:

python -m maestro.run --strategy single_agent --tier 1 --repeats 1
# Docker: docker compose run --rm maestro python -m maestro.run --strategy single_agent --tier 1 --repeats 1

4. Run the full matrix

python -m maestro.run
# Docker: docker compose run --rm maestro python -m maestro.run

Runs are resumable by default: already-completed cells are skipped, so an interrupted run can be restarted with the same command. Results are written to maestro.db (or ./out/maestro.db under Docker).

5. Analyse the results

python -m maestro.analysis

6. Explore the results in the dashboard

docker compose up          # → http://localhost:8501
# Local (without Docker): streamlit run src/maestro/viz/app.py

Reproducibility audit trail

Every invocation snapshots its runtime environment — OS, architecture, Python version, library versions, git commit, and (under Docker) the image digest — into the run_environments table, linked to each run. This lets a later replication attempt diagnose diverging numbers against the exact stack that produced the original data.

---

Local development

Setup is tested on macOS. Install the dev extras and run the test suite and linters from the project root:

pip install -e ".[dev]"
pytest
ruff check .
ruff format --check .

pre-commit hooks (ruff lint + format) are configured in .pre-commit-config.yaml; enable them with pre-commit install.

---

Citing

If you use MAESTRO in your work, please cite it via the CITATION.cff file (GitHub's "Cite this repository" button), or see that file for the reference details.