Reframe Guide Whitelisted models around harness-picks-model#302
Open
PunchTheDev wants to merge 1 commit into
Open
Reframe Guide Whitelisted models around harness-picks-model#302PunchTheDev wants to merge 1 commit into
PunchTheDev wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Guide
Whitelisted models(L604-637) was a list of 18 chips fronted by a generic "LLM agents receive a harness-injected LLMClient" lead and footnoted with the most important fact (the harness picks which model runs, agents cannot override). Rebuilt around that fact, deduped a 3rdllm.chat()code block already covered in Step 3 and Patterns, and tooltipped the two canonical sources first-timers would otherwise need to grep the repo for.Motivation
A first-timer reading the prior version sees 18 chips and reasonably concludes "let me pick claude-opus." That's wrong — agents call
llm.chat()once and the harness setsFORGE_MODELper eval run (defaultanthropic/claude-haiku-4-5, overridable bysecrets.FORGE_MODELin CI). The footnote saying "model is fixed by harness" buried the only fact that prevents the misconception.The section also re-documented the SDK call shape that Steps 3 and Patterns already cover post-PRs #297 and #300 — a 3rd duplicate. Per
feedback_link_to_canonical_explainer.md, route to the canonical site, don't duplicate.Changes
QuickstartGuide.tsxL604-637 — lead rewritten withharness picks which model runs itforegrounded;FORGE_MODELenv var tooltip (326 chars) cites the exact CI workflow lines (eval.ymlL157,score.ymlL104/151) and SDK enforcement (forge/sdk/llm.pyL23, L32-35 — whitelist check raises beforechat()ever runs)config/model-whitelist.txtlink gets a 200-char tooltip citing the file's own "CI reads it directly. Update this file to add or remove models" header — names it as the canonical authority, not just a referencellm.chat([...])code block; replaced with a 1-sentence routed<a href="#write">Step 3 - Write your agent ↑</a>pointer carrying the canonicalmax_tokens=4096SDK signature fromforge/sdk/llm.pyL41#anti-gaming ↓(carries the step 281/372/373/375 cross-link pattern — surfaces the enforcement chain instead of restating it inline)WHITELISTED_MODELS.lengthso the lead and grid never drift if the list is editedBACKLOG.mdL111 flipped ● ● ● with rationaleVerification
Puppeteer 1440x900 on
/guide#models:hasHarnessPicks=truehasForgeModelInline=truehasNoApiKey=trueoldLlmChatBlockGone=trueoldFootnoteGone=true(noThe model is fixed by the harness viaphrase)sdkRefersStep3=truemaxTokens4096=truelinkToWrite=truelinkToAntiGaming=truelinkToWhitelistFile=truechipCount=18(matches canonicalconfig/model-whitelist.txt18 entries)FORGE_MODELenv var (326 chars),config/model-whitelist.txt(200 chars)