Update screenshot collector to use existing screenshots, handle events automagically & update evals to all use collector

[tkattkat]

why

• Currently, when using screenshot collector with hybrid mode, it takes a new screenshot on every step
• we have outdated evals which are flaky, and less reliably than screenshot collector based ones
• screenshot collector currently requires manually handling the events in the file it is defined within

what changed

• now, we just pass the screenshot taken by agent, directly into the collector when hybrid mode is used.
• for dom mode, we still take on every step as a non vision model is much more likely to take its own screenshots
• removed dead code from screenshot collector type
• moved event handling for screenshot collector into the collector.
• removed timeout from all evals, so they utilize only event images
• updated all non screenshot collector evals to utilize them, and fixed two evals which would throw false negatives ( apple_tv & nba trades)

test plan

---

Summary by cubic

Use agent-provided screenshots in hybrid mode to stop duplicate captures and reduce eval overhead. The screenshot collector now auto-subscribes to agent events, and all evals were updated to use it.

• Refactors

  • Hybrid: forward screenshot tool output directly to the collector (no new page screenshot).
  • DOM (evals only): still capture after each tool execution.
  • Collector: automatically handles bus events; optional interval capture remains.
  • Evals: switched to event-driven collector, removed manual listeners, and pass screenshots to the evaluator.
  • Removed unused ScreenshotCapablePage type.

• Bug Fixes

  • Corrected event name to agent_screenshot_taken_event.

^{Written for commit d9d5e26. Summary will update automatically on new commits.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: d9d5e26

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[greptile-apps]

Greptile Summary

Optimizes screenshot collection in hybrid mode by reusing screenshots already taken by the vision model agent, rather than taking redundant screenshots on every step.

Key Changes:

• Intercepts screenshot tool output in hybrid mode and forwards it to the event bus for the screenshot collector
• Restricts automatic screenshot capture to DOM mode only (non-vision models)
• Removes unused ScreenshotCapablePage type definition from evals types

Issues Found:

• Missing null safety check for screenshotOutput before accessing .base64 property (line 173) - if the screenshot tool fails, this will throw an error

Confidence Score: 3/5

• This PR is mostly safe to merge but has a logic issue that could cause runtime errors
• The optimization logic is sound and improves efficiency, but the missing null check on screenshotOutput.base64 could cause runtime errors if the screenshot tool fails or returns an unexpected structure
• Pay close attention to packages/core/lib/v3/handlers/v3AgentHandler.ts - add null safety check before accessing screenshot output

Important Files Changed

Filename	Overview
packages/core/lib/v3/handlers/v3AgentHandler.ts	Optimizes screenshot collection for hybrid mode by reusing agent screenshots; adds screenshot interception logic but lacks null safety check
packages/evals/types/screenshotCollector.ts	Removes unused ScreenshotCapablePage type definition

Sequence Diagram

sequenceDiagram
    participant Agent as Agent (Hybrid Mode)
    participant Handler as V3AgentHandler
    participant Tool as Screenshot Tool
    participant Bus as Event Bus
    participant Collector as Screenshot Collector

    Note over Agent,Collector: Hybrid Mode (Vision Model)
    Agent->>Tool: Call screenshot tool
    Tool->>Tool: Take screenshot
    Tool-->>Agent: Return {base64, timestamp, pageUrl}
    Agent->>Handler: Tool execution complete
    Handler->>Handler: Intercept screenshot output
    Handler->>Bus: Emit agent_screensot_taken_event
    Bus->>Collector: Forward screenshot buffer

    Note over Agent,Collector: DOM Mode (Non-Vision Model)
    Agent->>Handler: Tool execution complete
    Handler->>Handler: Check mode === "dom"
    Handler->>Handler: captureAndEmitScreenshot()
    Handler->>Tool: Take new screenshot
    Tool-->>Handler: Return screenshot buffer
    Handler->>Bus: Emit agent_screensot_taken_event
    Bus->>Collector: Forward screenshot buffer

Loading

[greptile-apps]

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[cubic-dev-ai]

1 issue found across 2 files

Prompt for AI agents (all 1 issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="packages/core/lib/v3/handlers/v3AgentHandler.ts">

<violation number="1" location="packages/core/lib/v3/handlers/v3AgentHandler.ts:173">
P1: Potential TypeError: `screenshotOutput` could be `undefined` due to the optional chaining, but `.base64` is accessed without a null check. This will crash if `toolResults[i]` or its `output` is missing.</violation>
</file>

_{Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR}

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (all 1 issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="packages/core/lib/v3/handlers/v3AgentHandler.ts">

<violation number="1" location="packages/core/lib/v3/handlers/v3AgentHandler.ts:509">
P1: This typo fix is incomplete - the listeners in `webvoyager.ts` and `onlineMind2Web.ts` still subscribe to the old typo `agent_screensot_taken_event`, and `v3CuaAgentHandler.ts` still emits the old typo. This change will silently break screenshot collection for evals using this handler.</violation>
</file>

_{Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR}