feat: inject compile_command guidance into agent context files#18
Open
frederikprijck wants to merge 3 commits into
Open
feat: inject compile_command guidance into agent context files#18frederikprijck wants to merge 3 commits into
frederikprijck wants to merge 3 commits into
Conversation
Add an optional `compile_command` PROMPT.md frontmatter field. When set, a verify-compiles instruction is appended to the agent's native context file (CLAUDE.md / GEMINI.md / AGENTS.md / copilot-instructions.md) alongside the existing "no docs files" guidance, so the agent verifies the project compiles and the command appears in the tool trace. Wires the field into all 10 quickstart evals.
Uncomment the ranCommand/ranCommandOneOf build graders in the six frontend quickstarts (react, vue, spa-js, angular, nuxt, nextjs) now that compile_command instructs the agent to run the build. Install graders stay disabled — a valid solution may edit package.json then run a bare `npm install`, which the install grader would not match.
The injected compile-verification guidance used permissive wording
("you can use this command"), so capable models produced correct code
but skipped the build — failing the mandatory build-verification grader.
Rephrase as a "you MUST run" instruction and assert the mandatory
wording in tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
compile_commandPROMPT.md frontmatter field. When set, a verify-compiles instruction is injected into the agent's native context file (CLAUDE.md/GEMINI.md/AGENTS.md/.github/copilot-instructions.md) alongside the existing "no docs files" guidance — so the agent verifies the project compiles and the command shows up in the tool trace.Implementation
eval-core/types/eval.ts—compileCommand?onEvalDefinitioneval-core/loader.ts— parsecompile_commandfrontmattereval-core/workspace/workspace.ts— newcompileGuidance()helper;writeAgentGuidance()takes an optionalcompileCommandeval/cli/run.ts— passevalDef.compileCommandtowriteAgentGuidanceAGENTS.mddoc-sync table + checklist;docs/ADDING_EVALS.mdfrontmatter table + examplesTest plan
npm run buildpassesnpm run lintpassesnpm testpasses (684 tests)compile_commandparsing;writeAgentGuidancewith/without compile command;compileGuidancehelper