docs: document single-file form for eval questions

[keydunov]

Summary

The Evals docs page documented eval-question authoring as only the directory form (agents/eval_questions/*.yml), implying a single file is not supported. That's inaccurate.

The cube-runtime agents-config parser picks up both a single top-level agents/eval_questions.yml file and any number of files under agents/eval_questions/, flattening every file's eval_questions: list into one set (verified on a live deployment).

Changes

• Lead the "Authoring benchmark questions" section with the single-file agents/eval_questions.yml form (simplest entry point), then present the directory form as the way to split as the set grows.
• State that the parser merges both, so you can move from one file to many without changing anything else.
• Update the example snippet's path comment to # agents/eval_questions.yml.

Everything else (the eval_questions: list shape; name + question + exactly one of sql/certifiedQuery; optional space:) is unchanged.

🤖 Generated with Claude Code

[claude]

Claude finished @keydunov's task in 41s —— View job

---

Review: docs: document single-file form for eval questions

Verdict: LGTM — accurate, well-scoped clarification. No blockers.

Details

• Read PR diff and surrounding section in docs-mintlify/admin/ai/evals.mdx
• Validate accuracy against PR description (parser merges both forms)
• Check Mintlify style conventions (docs-mintlify/CLAUDE.md)
• Look for code-quality / security / perf issues (docs-only — N/A)

What's good

• Leads with the simplest path (agents/eval_questions.yml) and presents the directory form as a scale-up, matching the PR's stated intent.
• Explicitly states the parser merges both forms — removes the ambiguity the previous wording created.
• Example comment now matches the recommended starter path (# agents/eval_questions.yml).
• Style is consistent with docs-mintlify/CLAUDE.md: H2 section, short paragraphs, fenced YAML with language tag, second-person voice.
• The rest of the section (eval_questions: shape, name + question + exactly one of sql/certifiedQuery, optional space:) is preserved unchanged, as the PR body claims.

Nits (non-blocking)

• evals.mdx:42 — "any number of agents/eval_questions/*.yml files" is slightly redundant with the later "move from one file to many." Could tighten to "split them across agents/eval_questions/*.yml files," but the current phrasing is fine.
• No mention of file-ordering or duplicate-name behavior across files. If duplicates across files are an error (or last-wins), one sentence would help authors who actually split. Optional — can be deferred until someone hits it.

Out of scope for this review

• Code/security/performance/testing sections from the review template don't apply — this is a docs-only change to a single .mdx file.
  · branch docs/evals-single-file-questions

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
cubed3	🔴 Failed	–	Jun 23, 2026, 4:59 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.