Add ThinkBooster launch article (Medium/LinkedIn draft)#258
Open
smirnovlad wants to merge 19 commits into
Open
Add ThinkBooster launch article (Medium/LinkedIn draft)#258smirnovlad wants to merge 19 commits into
smirnovlad wants to merge 19 commits into
Conversation
Medium/LinkedIn post introducing ThinkBooster for a practitioner audience: the test-time compute problem, the strategy/scorer split, the one-URL proxy hook, three benchmark findings (confidence beating a PRM on HumanEval+, beam-vs-best-of-N compute, GPT-OSS CUDA kernels), a ReProbe + K2-Think-V2 generality result, the OptiLLM comparison, usage, limitations, and CTA. Section plan and figure placement in STRUCTURE.md. Also ignore the local experiment_results/ directory.
- Scope the beam-search 'top math accuracy' claim (best-of-N+PRM ties/beats it on MATH-500 and both AIME sets) - Soften OptiLLM compute-accounting wording to match our own comparison table - Hedge the CUDA compilation causal claim - Trim announced-honesty phrasing; subtitle no longer over-promises 'any LLM' - Record the K2 baseline provenance (SLURM job 2651567)
Collaborator
Author
Prose review (independent pass) — verdict: publishable after fixes, no reworkRan a skeptical-editor pass over Anti-slop: clean. One em-dash total (inside a figure placeholder), zero banned vocab (delve/leverage/seamless/robust/powerful/crucial/…), zero sentence-initial "Additionally/Moreover," no negative parallelisms, no rule-of-three puffery. Factual fixes applied
Verified clean, kept as-is: HumanEval+ 79.3→88.8 (MUR + entropy) and the under-a-quarter-compute claim (22.9% of beam+PRM); GPT-OSS-120B CUDA 26→30 correctness, −5pp syntax, 65→64 compile; nine strategies / four scorers; the 9-vs-7-strict / 20+-loose OptiLLM framing; the self-assessment caveat on the comparison table. K2 baseline provenance (the one item the reviewer flagged as unverifiable): confirmed — 66.1 / 80.7 is the logged single-shot run, SLURM job Still open (need a call before publish)
|
Option 1: run the service locally (link to the PRM service guide) and call the OpenAI-compatible endpoint. Option 2: import the modules as a library (self-consistency over an OpenRouter model). Both snippets verified against the current API (BlackboxModelWithStreaming, StepCandidateGeneratorThroughAPI, StrategySelfConsistency, generate_trajectories_batch).
Option 1 now links to docs/service/running_locally.md (general build-and-serve guide) on main instead of the PRM-specific draft.
Lead the CTA with the in-browser demo (http://demo-thinkbooster.nlpresearch.group, verified up). Note the http-only/ephemeral-pod caveat in the pre-publish checklist; mark PR #259 (service guide) merged.
From the ACL poster: surface the registry/plug-in design (one subclass works across gateway, benchmark, debugger; any scorer x any strategy) and vLLM single-GPU-to-multi-node scaling in the overview; enrich the visual-debugger description with per-step scores, alternative candidates, and side-by-side strategy comparison under identical compute; add the screencast (YouTube) link to the CTA.
Fix an invalid endpoint route (best_of_n/confidence -> offline_bon/entropy; the service only registers self_consistency/offline_bon/online_bon/beam_search). Reframe the 'visual debugger' claim (ReasonGraph also ships one) to 'what no other framework offers all at once'. Drop the 'first tool' superlative and the unverified OptiLLM '20-plus' count. Scope the math claim (PRM/self-consistency win, not entropy). Soften the decision box, the 'no other framework ships' line, and 'almost no paper'. Move the internal pre-publish checklist into blog/PUBLISH_CHECKLIST.md so the article is clean to publish. Kept: the running_locally.md link (valid on main, codex saw a stale local checkout) and the ReProbe name.
Collaborator
Author
Codex review (OpenAI Codex CLI, high effort) — applied in
|
| # | Finding | Fix |
|---|---|---|
| P1 | /v1/best_of_n/confidence is an invalid route |
→ /v1/offline_bon/entropy. The service only registers self_consistency / offline_bon / online_bon / beam_search with scorers entropy / perplexity / sequence_prob / prm. Real bug, and it contradicted our own running_locally.md. |
| P1 | "a visual debugger" listed under "what the others do not ship" | Reframed to "what no other framework offers all at once" — ReasonGraph also has a debugger (✓ in our table), so the claim is about the combination. |
| P1 | "ThinkBooster is the first tool…" | Dropped the superlative (competitor-attackable). |
| P1 | "both cover 20-plus methods" | Softened — keep ThinkBooster's "20+" (sourced in main.tex), don't assert OptiLLM's count. |
| P1 | "On math, a trained reward model still wins" | Scoped: PRM-scored methods lead, self-consistency wins the hardest sets (AIME) — not a blanket "reward model wins". |
| P1 | Internal PRE-PUBLISH checklist in an HTML comment could leak if pasted into Medium | Moved to blog/PUBLISH_CHECKLIST.md; article file is now clean. |
| P2 | "almost no paper reports the cost" | → "most papers don't report the cost". |
| P2 | "no other framework ships [ReProbe]" | → "no other framework in our comparison offers it as a first-class scorer". |
| P2 | decision box "beam search for max accuracy" | Added "best-of-N with the same model is often as good for far less". |
| P2 | "the rest each specialize in one strategy" | Softened (OpenR/LLM Reasoners have several). |
Not applied (verified)
| Finding | Why |
|---|---|
P1: running_locally.md link "broken" |
False positive — codex reviewed a stale local main. The doc is on main (this PR's sibling #259 merged), link resolves. Kept. |
P1: scorer is called UHead/step_reasoning, not ReProbe |
Intentional — ReProbe is the deliberate public name (README already uses it). The single-seed caveat is already in the text. Kept. |
AI-slop check: codex found no blockers — prose avoids the banned vocabulary, sentence-initial "Additionally/Moreover", and "not X but Y" patterns.
Net: gate started FAIL (8 P1), now resolved (6 fixed, 1 false positive, 1 intentional keep) plus all 4 P2. Article is publishable pending the items in blog/PUBLISH_CHECKLIST.md (figures, single-seed K2, keep the demo pod up).
Reader-facing article shouldn't carry seed/run-count caveats; the before/after numbers speak for themselves. The single-seed note stays in PUBLISH_CHECKLIST.md for internal tracking.
H1 -> 'ThinkBooster: unlock your LLM's Pro reasoning mode by changing one URL'. Trimmed the subtitle's redundant opening clause so it no longer echoes the title.
…e code section Export endpoint.pdf -> blog/endpoint_hero.png (1392x889, 149KB) as the Medium-ready hero. Remove the now-redundant endpoint.pdf placeholder from 'The hook, in one line of code' (the code snippet is that section's visual and the hero already shows the gateway).
Convert the §4 accuracy-vs-compute plots (qwen3_humaneval, qwen25_aggregate) to PNG and place one after each result paragraph instead of bundling them. Rebuild the GPT-OSS and framework-comparison tables as styled HTML rendered to PNG (sources + recipe in fig_*.html and FIGURES.md). Point the article at the local PNGs.
Rebuild tab:tts_strategies as a styled PNG (fig_strategies.png + source) and point §2 at it. Updates FIGURES.md.
Pull the debugger out of 'Try it in five minutes' into its own section after the comparison: main interface (demo-config), reasoning timeline + step inspector (demo-result), and trajectory tree (demo-treeview), plus the side-by-side strategy comparison. The usage section now just points to it.
Replace the three separate demo screenshots in the visual-debugger section with the paper's polished composites: fig_debugger_a (interface + strategy comparison + reasoning timeline + step inspector) and fig_debugger_b (trajectory tree), exported from main-1-new.pdf / main-2-new.pdf and trimmed to content. Merge the interface/timeline prose accordingly.
fig_debugger_a: drop the empty tree-header section at the bottom and the title block, easing the top trim (123->116pt) so the Custom Input Runner card keeps a margin. fig_debugger_b: the tree's own trim (120/360/120/40) drops the card border and empty space. Recipe recorded in FIGURES.md.
The hosted demo is the visual debugger, so the section now points readers straight to it.
Announcement built from the Medium article: ACL 2026 acceptance + the one-URL hook up top, the problem, the four-point what-it-is, two headline results, and links to the paper, Medium, demo, and repo.
Replace the inline base/plus numbers in the ReProbe section with two clean tables (MBPP+ base, MBPP+ plus) comparing single-shot CoT vs ReProbe best-of-N. ReProbe naming for the article; dropped the duplicated tokens/steps columns (identical across base/plus) so each figure is a tight accuracy table.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A draft launch article for ThinkBooster, aimed at a Medium/LinkedIn audience (builders + researchers). It introduces the framework through one argument: thinking models made test-time compute mainstream, but nobody can tell you which way of spending it is cheapest for a given accuracy — ThinkBooster lets you measure that and ship it by changing a URL.
Two files under
blog/:ARTICLE.md— the full Medium draft (~1,900 words, nine short sections).STRUCTURE.md— the section plan, figure→section map, voice guardrails, and the list of claims verified / to-fix.Section flow
Problem → what it is → the one-URL hook → three benchmark results → ReProbe on K2-Think-V2 → which strategy to use → vs OptiLLM & others → usage → limitations → CTA.
ReProbe + K2-Think-V2
Added a section featuring ReProbe (the group's internal-state scorer: Efficient Test-Time Scaling of Multi-Step Reasoning by Probing Internal States of Large Language Models) on K2-Think-V2 (MBZUAI + G42). ReProbe-guided best-of-N lifts MBPP+ from 66.1→66.9 (plus) / 80.7→81.7 (base) over single-shot decoding. Framed as a model-agnostic generality point rather than an accuracy headline — it is single-seed and being scaled up.
Review status
The structure went through an independent fact-check that caught and fixed two compute-framing claims a competitor could unpick (the beam-vs-best-of-N cost claim, and "confidence beats the PRM at a fraction of the compute"). All load-bearing numbers were cross-checked against the paper tables. A second review pass on the prose draft is running; findings will be posted as a comment here. (The codex connector isn't available in this environment, so this is an independent agent review in its place.)
Open questions before publishing
demo-thinkbooster.nlpresearch.groupup? Reviewers flagged it down; the CTA currently routes to GitHub + arXiv.Figures and tables are marked with placeholders in
ARTICLE.mdfor insertion.