fix(pipeline): graph routing infinite loop + search allow_tools#82
Merged
Conversation
Two bugs found during the first full pipeline run after the search milestone: 1. generate_queries self-loop: _needs_generate_queries checked state["raw_queries"] but the node writes state["queries"], so the router always saw an empty list and looped. Replaced the conditional self-edge with a direct edge to search_jobs. Also fixed the cache-hit path which read from state["raw_queries"] (always []) instead of the queries file. 2. anthropic_web search returned nothing: allow_tools: false in config was applied globally, including to the search LLM. The Claude CLI needs --dangerously-skip-permissions to invoke web-search tools. Factory now overrides allow_tools=True when task="search". Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Drop the redundant "→" link column; the run ID cell now carries the href so the table is one column narrower and every run is still one click away. Fixed the test that checked for the old trailing link cell pattern. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_node_row_html computed total_tokens = in + out, dropping cache_read and cache_creation. This caused the pipeline execution table to show lower numbers than the grand total, making the two sections appear inconsistent. Now matches _usage_row_html which already counted all four buckets. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
build_llm(cfg["llm"]) without a task arg defaults to task="default" which resolves allow_tools=False. Company searches with url: hints invoke the Claude CLI and need --dangerously-skip-permissions to browse the web — same fix as search_jobs already had. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Grand total line now shows '≈Xk effective compute' in green when cache tokens are present (formula: new_in + out + 0.1×cache_read), making it clear that high cache-read is efficient, not wasteful. Per-node pipeline table replaces the single token total with 'Xin / Yout' and adds '/ Zcached' (green) when cache-read tokens are present for that node. Same change applied to the live-page JS so the live view is consistent. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ctive Three connected fixes to improve search volume and quality: 1. url_validator: drop known aggregator/listing-page URL patterns before hitting Tavily (builtin.com, hnhiring, arc.dev listing pages, etc.). These pass URL validation because Tavily can fetch them, but they're search-result category pages, not individual postings — scoring rejects them at near-100% rate, wasting Tavily extract quota. 2. search_jobs: raise _DIRECTIVE_TARGET 30→50 and _DIRECTIVE_LLM_MAX 50→80 to target 30-50 validated individual postings per run. 3. web_search SEARCH_DIRECTIVE: instruct the LLM to search each major job board (WTTJ, LinkedIn /jobs/view, Lever, Greenhouse, Ashby, Workday) with dedicated queries rather than relying on broad web results. Explicitly forbid listing/search pages in the FORBIDDEN block so the LLM understands what counts as an individual posting. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Prompts extracted from web_search.py now live in: query/SEARCH_DIRECTIVE_PROMPT.md — directive (jobs search, URL candidates) query/SEARCH_COMPANY_PROMPT.md — company single-query search Edit these files to tune search behaviour without touching Python code. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bayrem
approved these changes
May 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Infinite loop in
generate_queries:_needs_generate_queriesrouted based onstate["raw_queries"], but the node writesstate["queries"]— so the router always saw an empty list and looped back. Replaced the conditional self-edge with a directadd_edge("generate_queries", "search_jobs"). Also fixed the cache-hit path which was reading fromstate["raw_queries"](always[]on first run) instead of the queries file directly.anthropic_websearch returned nothing:allow_tools: falseinconfig.yamlwas applied globally via the factory, including to the search LLM. The Claude CLI needs--dangerously-skip-permissionsto invoke its web-search tool.build_llm()now overridesallow_tools=Truewhentask="search", regardless of config.Test plan
infisical run --env=dev -- python run.py—generate_queriesadvances tosearch_jobswithout looping;anthropic_websearch returns URL candidates🤖 Generated with Claude Code