Skip to content

docs: document single-file form for eval questions #11145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
keydunov merged 1 commit into from
Jun 23, 2026
+8 −2
Merged

docs: document single-file form for eval questions #11145

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions docs-mintlify/admin/ai/evals.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,19 @@ sub-tabs: **Evaluations** (runs) and **Questions** (the benchmark set).
## Authoring benchmark questions

Questions live in your [data model repository](/admin/ai#agent-configuration),
versioned and branched like the rest of it — under `agents/eval_questions/*.yml`.
versioned and branched like the rest of it. You can keep them in a single
top-level `agents/eval_questions.yml` file — the simplest place to start — or
split them across any number of `agents/eval_questions/*.yml` files as your set
grows. The parser picks up both and merges every file's `eval_questions` list
into one set, so you can move from one file to many at any time without changing
anything else.

Each file has a top-level `eval_questions` list. A question needs a unique
`name`, a `question`, and exactly one ground truth: a `certifiedQuery`
reference **or** inline `sql`.

```yaml
# agents/eval_questions/revenue.yml
# agents/eval_questions.yml
eval_questions:
- name: revenue_by_quarter
question: What was our revenue by quarter over the last two years?
Expand Down
Loading