feat(physical-ios): add WebDriverAgent parity support#489
Draft
pr-zkp2p wants to merge 26 commits into
Draft
Conversation
Drive a real iPhone (iOS 27+) app-free via Apple's CoreDevice remote-control services through pymobiledevice3 — no on-device app, no WebDriverAgent. Gated behind the `physical-ios-devices` flag. - Discovery: classify physical UDIDs (8hex-16hex) as ios/device; list connected iPhones via `xcrun devicectl`, flag-gated. - CoreDevice backend (blueprints/core-device.ts): resolves the tunneld RSD endpoint, ensures the DDI is mounted, shells to pymobiledevice3 for screenshot, tap (dwell-drag — a zero-dwell tap is ignored by iOS), swipe, and buttons. - Branded privilege escalation: the tunnel needs root once to create its network interface. Instead of manual sudo, Argent auto-starts it via a signed macOS host helper (argent-device-auth) that shows the standard auth modal branded as "Argent" with the Argent icon (Authorization Services + dlsym of AuthorizationExecuteWithPrivileges). Falls back to the osascript admin prompt, then manual-sudo instructions (headless). HOME is pinned so tunneld finds its pairing records under the privileged-exec environment. - Wire screenshot/gesture-tap/gesture-swipe/button to the backend; launch-app via devicectl; boot-device starts the tunnel (branded prompt). Unsupported tools on physical iOS are cleanly rejected (simulator-server / ax-service / native-devtools guards; open-url/reinstall/restart handlers). - Helper resolvers + Argent icon in @argent/native-devtools-ios; download-native-binaries.sh fetches the signed helper (optional until published); bundle-tools copies it for distribution. Helper source + signing live in argent-private. - Tests for classification, devicectl parsing, coord mapping, and privileged command building. Overrides: ARGENT_PYMOBILEDEVICE3, ARGENT_PMD3_TUNNELD_PORT, ARGENT_DEVICE_AUTH_HELPER, ARGENT_DEVICE_ICON.
…y target list The preview / Argent Lens window streams frames over simulator-server, which refuses physical iOS devices (kind === "device", driven over CoreDevice). With the physical-ios-devices flag on, a connected iPhone leaked into GET /preview/simulators because the iOS branch matched any platform === "ios" without checking kind — surfacing a target the UI can't drive (and emitting an entry with no runtime under a type that asserted runtime: string). - Exclude physical iOS (kind === "device") from the preview list, mirroring the existing Chromium exclusion. - Consume list-devices' own exported ListDevicesResult type instead of a hand-rolled copy that had already drifted from it (the soundness hole tsc couldn't catch because invokeTool<T> takes a caller-supplied generic). - Extract the mapping into a pure, exported devicesToPreviewEntries() and add regression tests asserting physical iOS + Chromium are excluded while the simulator and Android emulator are kept.
…app behavior list-devices description and the simulator-server guard message listed screenshot/gesture-tap/gesture-swipe/button but omitted launch-app, which is implemented for physical iOS via devicectl. The README limitations list omitted restart-app, which (like open-url/reinstall-app) returns a clear "not supported" error on physical iOS.
…ice-support # Conflicts: # packages/tool-server/src/preview.ts # packages/tool-server/src/tools/open-url/platforms/ios.ts
…lity matrix Review follow-ups on the physical-iOS CoreDevice feature: - discovery: parsePhysicalIosDevices surfaced every host iOS simulator as a phantom physical device (devicectl lists simulators with transportType "sameMachine"); gate on the physical ECID UDID shape, which also aligns discovery with classifyDevice's routing. Verified against real devicectl JSON and a connected iPhone (returns only the device, not the host's simulators). - run-sequence: was eagerly resolving simulator-server for a physical iPhone, failing the whole call before step 1; hold no eager service so per-step CoreDevice routing works. - describe: swallowed the simulator-only guards and returned an empty tree with a misleading "reboot the simulator" hint; reject with a clear error instead. - swipe: command timeout was a fixed 15s, killing long swipes mid-drag; scale the timeout with the requested duration and clamp degenerate durations (extracted swipeDragParams, unit-tested). - native-profiler: guard the session blueprint so a physical iPhone isn't driven via simulator-only simctl (which mislabeled it an unresponsive "simulator"). - capability matrix: tools whose iOS backend is simulator-only (keyboard, paste, rotate, gesture-pinch/rotate/custom, screenshot-diff, native-devtools-*, native-profiler-start) now declare apple.device:false, so physical iOS is rejected with a clean 400 at the capability gate rather than a 500 from a deep service guard. open-url/reinstall-app/restart-app throw UnsupportedOperationError. - launch-app: wrap the devicectl launch so "app not installed/signed" surfaces a clean message instead of a raw subprocess blob. - core-device: check the feature flag before the pymobiledevice3 setup probe; echo a custom tunneld port in the manual-fallback hint. - docs: flags description and README now match the supported tool set and the auto-start prompt reality. Verified end-to-end on a physical iPhone (iOS 27): discovery, tunnel auto-escalation, screenshot (1179x2556), and swipe.
…e entry The previous commit guarded the shared native-profiler-session blueprint against physical iOS, but that blueprint also backs the device-agnostic analysis tools (profiler-stack-query / profiler-combined-report / -analyze), which profiler-query-android-capability.test.ts intentionally keeps supported on Apple device. The guard made those analysis tools 500 on a physical-iOS device_id. Scope the fix to where the actual bug is: only native-profiler-start enumerates running processes via simulator-only `simctl` (mislabeling a real iPhone as an unresponsive "simulator"), and it already declares apple.device:false, so the capability gate rejects it cleanly (400) before execute() runs. Drop the redundant session-blueprint guard; the analysis tools keep their cross-device capability. Also formats physical-ios-followups.test.ts.
launch-app drives a physical iPhone via `xcrun devicectl` directly rather than the CoreDevice service, so its handler never hit the flag check that gates every other physical-iOS operation — with `physical-ios-devices` off, launch-app on a physical UDID still launched the app. Extract the gate into a shared assertPhysicalIosEnabled() (reused by the CoreDevice factory and tunnel start) and call it from the launch-app physical-iOS branch before shelling devicectl. Add coverage: the launch-app flag gate, assertPhysicalIosEnabled, and the gesture-swipe physical-iOS routing (coords + duration forwarding).
… as unsupported The README limitations section listed only the interaction/lifecycle tools that reject on physical iOS; align it with the actual behavior by also noting that describe (accessibility), the native-* inspection tools, native-profiler-*, and screenshot-diff are simulator-only.
…; cover pmd3 argv On iOS 18-26 Apple gates host-driven input (the CoreDevice "remote control" services behind tap/swipe/button) to iOS 27+, so the device is discovered and advertised as tap-capable, then fails mid-session with a raw `CoreDeviceError 9021`. Detect that specific failure in `conciseError` (matched against pymobiledevice3's stderr/stdout only, never the argv-echoing execFile message, so a HID coordinate that contains "9021" can't false-positive) and surface an actionable message that points at iOS 27 and notes screenshots still work. `screen-capture` is not gated, so the screenshot path is untouched. Also note the physical-iOS/CoreDevice routing in the gesture-tap and gesture-swipe descriptions, and add a focused test that drives the CoreDevice factory through a mocked execFile boundary to pin the exact pmd3 argv for tap, button, screenshot, and swipe (previously untested — every other physical-iOS test mocks the CoreDeviceApi wholesale) plus the 9021 translation.
…ice-support # Conflicts: # packages/tool-server/src/preview.ts
…quivalent button's services() resolved coreDeviceRef() unconditionally for any physical-iOS button press, before execute() validates whether the button is even supported. Since the registry resolves services() before execute(), pressing appSwitch/actionButton on a physical iPhone triggered CoreDevice's tunnel-setup side effects (possibly a macOS admin prompt) just to reject the button afterward. Skip resolving the service when the button has no CoreDevice HID mapping.
…essage accuracy core-device.ts threw plain Error at all 12 sites, and ax-service.ts / simulator-server.ts's own new physical-iOS rejection branches did too - unlike every sibling blueprint, which uses FailureError with a dedicated error_code. Every physical-iOS failure (auth declined, no tunnel, missing pymobiledevice3, iOS<27 gate, flag disabled) was landing in telemetry as generic REGISTRY_TOOL_FAILURE_UNCLASSIFIED, indistinguishable from any other crash. Add CORE_DEVICE_* / AX_PHYSICAL_DEVICE_UNSUPPORTED / SIMULATOR_SERVER_PHYSICAL_DEVICE_UNSUPPORTED failure codes and convert every throw site. Also, three smaller correctness fixes found in the same pass: - resolveTunnel's `tunnel-port == null` check let a port of 0 through as "present", inconsistent with the sibling tunnel-address check which treats any falsy value as missing. - The iOS<27 gate message claimed "tap/swipe/button" all require iOS 27+, but this is hardware-verified false: screenshot and hardware buttons work below iOS 27, only touch (tap/swipe) is actually gated. - conciseError's line-matching heuristic could pick a misleading rich-boxed Python source-context line (e.g. "if rsd is not None:") instead of the real exception (e.g. "OSError: No route to host") from a stale-tunnel traceback, and searched execFile's synthesized "Command failed: <argv>" message with equal priority to the real stderr/stdout output.
… too native-profiler-start already rejects physical iOS (apple.device:false) since live capture uses simctl process enumeration that mislabels a real iPhone as a simulator. stop/analyze were left at device:true, so - since a session can never exist for physical iOS - they always fell through to a generic "no active session" error instead of the same clean, actionable rejection. Also fixes the README's "native-profiler-*" claim, which the stop/analyze gap made inaccurate, and two other accuracy issues found in the same pass: the iOS<27 limitation overstated to include hardware buttons, and a note that launch-app (devicectl) doesn't need the CoreDevice tunnel other tools require.
# Conflicts: # packages/argent/scripts/bundle-tools.cjs # packages/native-devtools-ios/src/index.ts # packages/tool-server/src/blueprints/native-devtools.ts # packages/tool-server/src/blueprints/simulator-server.ts # packages/tool-server/src/tools/describe/platforms/ios/index.ts # packages/tool-server/src/tools/devices/boot-device.ts # packages/tool-server/src/tools/devices/list-devices.ts # packages/tool-server/src/tools/gesture-custom/index.ts # packages/tool-server/src/tools/gesture-pinch/index.ts # packages/tool-server/src/tools/gesture-rotate/index.ts # packages/tool-server/src/tools/keyboard/index.ts # packages/tool-server/src/tools/launch-app/index.ts # packages/tool-server/src/tools/launch-app/platforms/ios.ts # packages/tool-server/src/tools/native-devtools/native-describe-screen.ts # packages/tool-server/src/tools/native-devtools/native-devtools-status.ts # packages/tool-server/src/tools/native-devtools/native-find-views.ts # packages/tool-server/src/tools/native-devtools/native-full-hierarchy.ts # packages/tool-server/src/tools/native-devtools/native-network-logs.ts # packages/tool-server/src/tools/native-devtools/native-user-interactable-view-at-point.ts # packages/tool-server/src/tools/native-devtools/native-view-at-point.ts # packages/tool-server/src/tools/paste/index.ts # packages/tool-server/src/tools/restart-app/index.ts # packages/tool-server/src/tools/restart-app/platforms/ios.ts # packages/tool-server/src/tools/rotate/index.ts # packages/tool-server/src/tools/run-sequence/index.ts # packages/tool-server/src/tools/screenshot/index.ts # packages/tool-server/src/utils/device-info.ts # packages/tool-server/src/utils/ios-devices.ts # scripts/download-native-binaries.sh
# Conflicts: # packages/tool-server/src/preview.ts
The app-free CoreDevice accessibility path was investigated on a physical
iPhone (iOS 27). The on-device accessibility tree is served by CoreDevice's
axAuditDaemon, but Apple gates it: the DTX service
`com.apple.accessibility.axAuditDaemon.remoteserver.shim.remote` requires the
`com.apple.mobile.lockdown.remote.trusted` entitlement — over the developer
(untrusted) CoreDevice tunnel pymobiledevice3 forms, the daemon accepts the
socket but terminates it on the first request (every audit selector, and even
the standard DTX capability handshake). The RemoteXPC replacement
`…axAuditDaemon.remoteAXService` requires `AppleInternal`, unreachable by any
third party. DTX transport itself works over the same tunnel
(`developer dvt proclist` succeeds), so this is Apple's auth wall, not a
transport gap. Screenshot (`canViewDeviceDisplay`) remains the only app-free
screen capture.
Replace the imprecise describe rejection ("no on-device accessibility/describe
path") and the ax-service backstop's "not supported yet" with accurate,
evidence-based messages, and split describe out of the README "not yet" list.
Drive a physical iPhone through one long-lived pymobiledevice3 process per device instead of spawning the CLI per interaction. Each `pymobiledevice3` invocation cost ~0.8s (~0.5s just `import pymobiledevice3`); the sidecar connects the RSD tunnel and opens the HID/screenshot services once, so a warm tap/button/swipe is now ~150-320ms (5x) — verified live on an iPhone 15 (iOS 27) through the tool-server. The agent is a small Python program, base64-embedded in coredevice-agent.ts (survives esbuild, no build-pipeline wiring), materialized to a temp file and run with pmd3's own venv interpreter; it speaks newline-delimited JSON over stdio. core-device.ts now maps its responses (incl. the iOS-27 9021 gate) to FailureErrors and tears the process down on dispose. Also make `describe` work on a physical iPhone: it returns the SpringBoard home-screen layout (app icons + dock) via CoreDevice's springboardservices — the only app-free structured screen data on a real device. Icon frames are derived from the home-screen grid (row-major first-fit so multi-cell widgets don't shift the icons after them) and are approximate; the hint says to confirm with screenshot and that in-app content isn't reachable. Verified the describe→tap loop live (tapped Settings via describe's frame; it opened). In-app accessibility stays Apple-gated: the axAuditDaemon DTX service drops the connection on the first message over the developer tunnel, and its RemoteXPC replacement needs the AppleInternal entitlement. Replaces the per-call argv test with sidecar/adapter/agent-protocol coverage.
Replaces the SpringBoard home-screen-only describe with the device's live on-screen accessibility tree, read app-free over CoreDevice via the iOS-26+ axAudit service. Works in ANY app and on the home screen (the same VoiceOver walk): labels, values, traits (mapped to roles) and reading order are exact. The unlock is the RSDCheckin handshake iOS 26 added to RSD DTX services. pmd3's DtxServiceProvider opens RSD services with a raw connect and skips RSDCheckin, so iOS 26/27 dropped the accessibility daemon on the first byte (which had read as an "Apple wall"); the sidecar monkeypatches the open to go through start_lockdown_service, which performs the checkin. Verified live on iPhone/iOS 27: describe Settings -> read the "Other…" button's frame -> gesture-tap it -> the Join-Wi-Fi form opened. Home screen lists every app + widget. Frames: Apple exposes no per-element geometry on hardware, so exact rects come only from the accessibility audit (a subset of elements, correlated to the tree by element id); the rest are interpolated from reading-order neighbours - enough to tap a list row, with a hint to confirm precise taps via screenshot. Pixel-exact in-app frames would need an on-device XCUITest runner (code-signing). - sidecar: RSDCheckin fix + an `axtree` op (iter_elements walk + run_audit rects + screen size), replacing `homescreen`. - new ios-coredevice-ax-adapter (caption->role/label parse, rect normalize, frame interpolation); removes the springboard adapter. Source: coredevice-ax.
pr-zkp2p
force-pushed
the
feat/physical-ios-e2e
branch
from
d843f14 to
311c61e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@swmansion/argent, and update packaged rules/skills so agents discover connected physical iPhonesWhy
Argent's existing physical-iOS experiment depended on a CoreDevice control surface that was unavailable for the tested iOS version and only covered a small subset of the simulator interface. Agents could discover the phone but could not reliably type, use multi-touch, inspect the foreground hierarchy, profile protected apps, or run realistic end-to-end workflows.
The new backend keeps the existing tool contracts and normalized coordinates, so callers select the
kind: "device"target returned bylist-devicesand use the normal Argent tools.Validation
npm run buildawait-ui-elementsuite: 37/37 tests passinggit diff --checkLive end-to-end validation used an iPhone 15 Pro on iOS 26.2 and an iPhone 17 simulator. Apple Maps was driven through search, keyboard input, tap, swipe, hold, interpolated pan, pinch, and rotation while unified logs and native profiling ran concurrently. Three-pass physical control registration remained within 5% of the simulator for every measured gesture (largest difference: tap at 4.59%).
Platform notes
argent enable physical-ios-devices.