feat: 14 vendor-derived agent optimizations#39
Merged
Conversation
…n settings schema - Add DynamicPromptContext struct with stagnation_alert, planning_guidance, budget_warning, loop_nudge fields (all Option<String>) - Add build_dynamic_section() that renders non-None fields into section 9 - Extend build_system_prompt() to accept Option<&DynamicPromptContext>; None preserves byte-identical output to current behavior - Add Arc<Mutex<Option<Vec<String>>>> prompt_override slot to ConversationRuntime; prepare_iteration() checks and applies the override before each API call - Add Arc<Mutex<Option<String>>> last_assistant_text slot to ConversationRuntime; stream_assistant_message() writes the latest response text for agent to read - CrawlerAgent holds both slots and wires them to ConversationRuntime at construction - Add OptimizationSettings struct with 18 fields (all Option<T>, all default false/0) - Add optimization: Option<OptimizationSettings> to Settings struct with serde(default) - Add 18 settings_get_* getter functions all defaulting to safe off-by-default values - All existing tests pass; new tests verify backward compat and dynamic section behavior
… planning interval - HtmlDiffTracker: sections-based diff on repeated URL visits; returns only changed sections with [unchanged: N sections] markers; 50%+ token savings on revisits - LoopDetector: rolling window action hash with escalating nudges at 5/8/12 repeats; stagnation detection after 5 consecutive identical page fingerprints; no hard blocks - PageFingerprint: lightweight FNV-1a hash of url+element_count+first-1000-chars text; deterministic, no timestamps, used by LoopDetector and ActionCache (Task 7) - Planning interval: every N steps inject planning-checkpoint or execution-mode guidance into DynamicPromptContext via prompt_override_slot; disabled by default (interval=0) - All features gated by settings.optimization.* flags (all default OFF) - CrawlState gains html_diff_tracker, loop_detector, page_fingerprints fields (ephemeral)
…on caching, confidence tracking - FailureClassifier: 16-category keyword-based error taxonomy (zero LLM cost); classify() maps error messages to SelectorNotFound/CaptchaDetected/RateLimited etc; retry_strategy() returns RetryWithHealing/RetryWithDelay/NoRetry per category - SelfHealingSelectors: on SelectorNotFound/SelectorAmbiguous, fetches fresh page_map and text-match heals to correct element ref; logs [healed: @Eold -> @Enew]; max retries from settings (default 2); zero LLM calls - ActionCache: FNV-1a keyed in-memory cache for read-only tools (page_map, read_content, list_resources, execute_js); invalidated on page fingerprint change; TTL 30s default; interaction tools never cached; in-memory only (no cross-session persistence) - ConfidenceTracking: parses [confidence: HIGH/MEDIUM/LOW] from assistant text; 2+ consecutive LOWs triggers stagnation_alert via DynamicPromptContext; advisory only - All features gated by settings.optimization.* flags (all default OFF) - CrawlState gains action_cache field; CrawlerAgent gains confidence_tracker field
…udget enforcement, cost attribution - CompoundComponentEnrichment: bridge_script.rs getEnrichment() adds optional enrichment JSON to form controls (date/range/number/color/file/select/textarea); max 200 bytes/elem, 20 select options + overflow; guarded by compound_enrichment flag - ContentAwareCleaningProfiles: CleaningProfile enum (Default/Minimal/Aggressive/ReadingMode) with per-profile thresholds and tag-weight multipliers; select_profile() auto-selects by task keyword and content size; prune_html() unchanged, new prune_html_with_profile() - BudgetEnforcement: BudgetEnforcer (Warn/Block/RouteDown modes), SharedCostCounter Arc<AtomicU64> millicents slot shared runtime-to-agent; Warn injects budget_warning into DynamicPromptContext; Block terminates loop via ToolError; RouteDown acts as Block - PerAgentCostAttribution: AgentCostReport + build_cost_breakdown() walks flat child_sessions and reconstructs per-child cost; /cost command shows per-agent breakdown when per_agent_cost_tracking flag is ON - Fix: mcp-server execute_async call updated to pass CrawlState parameter
- README: add optimization settings table to settings.json section - README: add Performance Optimizations section with feature table + example config - AGENTS.md: add Optimization layer section describing shared infrastructure, per-module table, and execution order in CrawlerAgent::execute() - AGENTS.md: update test count from ~770 to ~1,100
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…child attribution Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…om action cache Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…ntext additive Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…ion in tool handlers Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…gning-clones lint Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…diff_mode Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements 14 optimizations derived from analysis of 10 competitor products (browser-use, Stagehand, crawl4ai, Skyvern, Spider, nanobrowser, ZeroClaw, Gottem). All features are disabled by default and opt-in via settings.optimization.*.
Optimizations
Token Savings
Loop Prevention
Reliability
Performance
Budget
Foundation
Verification
Commits