ci: device E2E smoke tests for every hosted-CI platform by latekvo · Pull Request #371 · software-mansion/argent

latekvo · 2026-06-18T13:35:37Z

What

wayland-e2e.yml boots an Android AVD on Linux under headless Weston and asserts the full pipeline works without a display. It guards exactly one cell of the support matrix. This adds the same boot→screenshot→gesture-tap smoke test for every other (target × host) cell that can run on GitHub-hosted runners.

target	host	job	status
Android emulator	Linux (KVM)	`wayland-e2e.yml` (existing)	✅
iOS simulator	macOS	`ios-sim-macos`	✅
Chromium / Electron	Linux (Xvfb)	`chromium-linux`	✅
Chromium / Electron	macOS	`chromium-macos`	✅

How

scripts/e2e/drive-device.sh — shared driver. Boots one device through the tool-server, then asserts booted:true → screenshot has real (non-blank) pixels → gesture-tap round-trips. Every tool takes the device id as udid and screenshots come back as the same data.image.hostPath envelope on all platforms, so the body is uniform; each job just supplies the cell-specific boot JSON + device id.
packages/tool-server/test/fixtures/electron-smoke-app/ — minimal self-contained Electron app the Chromium jobs point electronAppPath at. It vendors its own electron via its own lockfile (kept out of the repo root lockfile); CI npm cis it so boot-electron resolves ./node_modules/.bin/electron.
.github/workflows/e2e-device-smoke.yml — three jobs, workflow_dispatch + path-filtered pull_request. iOS downloads the darwin native binaries (injection must succeed for bootIos to report booted:true); Chromium-Linux runs under Xvfb with --no-sandbox. The tool-server is started with TS_NODE_TRANSPILE_ONLY=1 so cold ts-node startup doesn't flakily exceed the readiness poll on loaded macOS runners (types are still enforced by the typecheck job).

Known gaps (supported on real machines, not runnable on hosted runners)

Android emulator on macOS — the arm64 emulator needs HVF (-enable-hvf), but hosted macOS runners are themselves VMs with no nested virtualization. Confirmed by running it: qemu dies with HVF error: HV_UNSUPPORTED ~18s into boot, regardless of GPU mode or RAM. The Android boot/screenshot/tap path is already regression-guarded on Linux via KVM (wayland-e2e.yml), so macOS adds no runnable coverage.
Physical devices — real iOS (CoreDevice HID) and real Android (adb-USB) have no attached hardware on hosted runners.

Both would need self-hosted runners (a Mac exposing HVF / wired-up devices). Documented in the workflow header.

Verification

All checks green: the 3 device jobs + ESLint + Prettier + Unit tests.
Locally (before CI): ran drive-device.sh end-to-end against a real tool-server → boot booted:true, screenshot 142 KB (floor 20 KB), tap tapped:true, exit 0; and confirmed via the Argent stack that a tap at (0.5, 0.5) lands on the fixture's centered button and increments its counter.

Extends the Android-on-Linux coverage in wayland-e2e.yml to the rest of the supported (target x host) matrix that can run on GitHub-hosted runners: - iOS simulator on macOS - Chromium/Electron on Linux (Xvfb) - Chromium/Electron on macOS Each job boots one device through the tool-server and asserts the headless interaction pipeline via a shared scripts/e2e/drive-device.sh: booted:true -> screenshot returns real pixels -> gesture-tap round-trips. The Chromium cells point at a minimal self-contained Electron fixture that vendors its own electron (kept out of the root lockfile). Documented as known gaps (supported on real machines, not runnable on hosted runners): Android emulator on macOS (the arm64 emulator needs HVF, but hosted macOS runners are VMs with no nested virtualization -> HV_UNSUPPORTED; the Android path is already guarded on Linux via KVM), and physical iOS/Android devices (no attached hardware).

@v4

…docs - The chromium-linux (ubuntu) job carried the macOS jobs' verbatim rationale ("loaded macOS arm runner"); reword the shared TS_NODE_TRANSPILE_ONLY comment to be OS-neutral so it is accurate in all three jobs. - Pin actions/upload-artifact@v7 to match the rest of the repo (was @v4). - Fixture package.json description said CI runs `npm install`, but the workflow runs `npm ci`; correct the wording.

…ecks, screenshot retry, wider path filter

hubgan · 2026-06-26T06:45:38Z

+          path: |
+            ${{ runner.temp }}/smoke-shot.png
+            ${{ runner.temp }}/smoke-shot.json
+            ~/.argent/tool-server.log


actions/upload-artifact resolves path: with @actions/glob, which does not do tilde expansion, so ~/.argent/tool-server.log matches nothing and if-no-files-found: warn silently swallows it. The shots above work only because they use ${{ runner.temp }}. Net effect: on a failed job the one artifact you actually need is missing.

The same goes for line 185 and 242

I checked this before changing it, and upload-artifact@v7 does expand a leading ~/. It bundles @actions/glob@^0.6.1, whose pattern normalizer rewrites a leading ~/ segment to os.homedir() — the branch that checks pattern === '~' || pattern.startsWith('~' + path.sep) in its bundled dist/index.js.

I confirmed it empirically against 0.6.1: glob.create('~/.argent/tool-server.log').glob() resolves the search path to /Users/<me>/.argent/tool-server.log and matches the file. So ~/.argent/tool-server.log does get uploaded — leaving it as-is. (The .png / .json use ${{ runner.temp }} only because that's where drive-device.sh writes them.)

hubgan · 2026-06-26T06:46:27Z

+#
+# Manual + on-change trigger only — the macOS jobs are slow and billed.
+
+on:


Add a concurrency group to cap billed macOS minutes. The header notes the macOS jobs are "slow and billed," but pull_request fires two of them on every push to a matching PR with no cancellation, so rapid pushes stack up.

Fixed in 6ab5fe1 — added a workflow-level concurrency group keyed on ${{ github.workflow }}-${{ github.ref }} with cancel-in-progress: true, so rapid pushes cancel superseded (billed macOS) runs.

hubgan · 2026-06-26T06:47:43Z

+      - name: Start tool-server
+        run: |
+          mkdir -p ~/.argent
+          : > ~/.argent/tool-server.log
+          cd packages/tool-server
+          # TS_NODE_TRANSPILE_ONLY skips ts-node's whole-program type-check (the
+          # typecheck job already enforces types). Without it a cold start on a
+          # loaded CI runner can take >60s to bind and flakily times out.
+          nohup env ARGENT_PORT=3033 TS_NODE_TRANSPILE_ONLY=1 npx ts-node src/index.ts start \
+            > ~/.argent/tool-server.log 2>&1 &
+          for i in $(seq 1 60); do
+            curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:3033/tools | grep -q 200 \
+              && { echo "tool-server up t+${i}s"; exit 0; }
+            sleep 1
+          done
+          echo "tool-server failed to start"; cat ~/.argent/tool-server.log; exit 1


This block is duplicated verbatim in all three jobs (here, 154, 210), as is the npm ci + rollup-install + tsc --build boilerplate. Worth extracting into a composite action (e.g. .github/actions/start-tool-server) so the cells can't drift out of sync the TS_NODE_TRANSPILE_ONLY / port / poll logic only needs to live in one place.

Fixed in 6ab5fe1 — extracted the install/build + tool-server-start block into a shared composite action (.github/actions/start-tool-server) that all three jobs now use, so the TS_NODE_TRANSPILE_ONLY / port / readiness-poll logic lives in exactly one place. The rollup native binary is selected by $RUNNER_OS.

hubgan · 2026-06-26T06:49:51Z

+}
+
+echo "::group::gesture-tap"
+TAP_RESP=$(curl -sS -m 30 -X POST "${BASE_URL}/gesture-tap" \


gesture-tap has no retry, unlike screenshot's 3x loop just above (line 59). Low risk because the screenshot succeeded immediately before, so the device is painted but if the same first-frame/transport race (#391) ever surfaces on the tap, it's an unguarded hard fail. Either wrap it in the same small retry, or add a one-line comment stating the tap relies on the preceding successful screenshot as its readiness gate.

Fixed in 6ab5fe1 — gesture-tap now uses the same 3× retry as the screenshot, with robust JSON parsing so an error/non-JSON reply retries instead of raising a raw Python traceback.

…action, tap retry - Add a workflow-level concurrency group (group per workflow+ref, cancel-in-progress) so rapid pushes to a PR don't stack up the slow, billed macOS jobs. - Extract the duplicated install/build + tool-server-start block into a shared composite action (.github/actions/start-tool-server) so the TS_NODE_TRANSPILE_ONLY / port / readiness-poll logic lives in one place and the three cells can't drift out of sync; the rollup native binary is picked by RUNNER_OS. iOS native-binary download moves after the action so Node is set up for its `node -p` manifest read (binaries are only needed at boot time). - Give gesture-tap the same 3x retry as the screenshot, parsing the response JSON robustly so an error/non-JSON reply retries instead of blowing up with a raw python traceback.

latekvo · 2026-06-26T15:44:58Z

Merging due to purely CI changes. Holding this PR open is somewhat inconvenient for #421 and others

latekvo force-pushed the e2e-device-smoke branch 5 times, most recently from 05126ff to 2296695 Compare June 18, 2026 14:10

latekvo force-pushed the e2e-device-smoke branch from 2296695 to bf27031 Compare June 18, 2026 14:15

latekvo marked this pull request as ready for review June 18, 2026 15:41

latekvo marked this pull request as draft June 18, 2026 15:43

latekvo added 2 commits June 18, 2026 17:45

ci: shorten device E2E job names to 'Host + Platform E2E'

fb5f217

latekvo marked this pull request as ready for review June 22, 2026 16:15

hubgan reviewed Jun 25, 2026

View reviewed changes

Comment thread scripts/e2e/drive-device.sh Outdated

Comment thread scripts/e2e/drive-device.sh

Comment thread .github/workflows/e2e-device-smoke.yml

Comment thread scripts/e2e/drive-device.sh Outdated

fix(ci): harden device-smoke driver — error detection, robust JSON ch…

c44f3a4

…ecks, screenshot retry, wider path filter

latekvo requested a review from hubgan June 25, 2026 14:26

hubgan reviewed Jun 26, 2026

View reviewed changes

latekvo requested a review from hubgan June 26, 2026 15:38

latekvo merged commit 05e619e into main Jun 26, 2026
8 checks passed

latekvo deleted the e2e-device-smoke branch June 26, 2026 15:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: device E2E smoke tests for every hosted-CI platform#371

ci: device E2E smoke tests for every hosted-CI platform#371
latekvo merged 5 commits into
mainfrom
e2e-device-smoke

latekvo commented Jun 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hubgan Jun 26, 2026

Uh oh!

latekvo Jun 26, 2026

Uh oh!

hubgan Jun 26, 2026

Uh oh!

latekvo Jun 26, 2026

Uh oh!

hubgan Jun 26, 2026

Uh oh!

latekvo Jun 26, 2026

Uh oh!

hubgan Jun 26, 2026

Uh oh!

latekvo Jun 26, 2026

Uh oh!

Uh oh!

latekvo commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

latekvo commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Known gaps (supported on real machines, not runnable on hosted runners)

Verification

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

latekvo commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

latekvo commented Jun 18, 2026 •

edited

Loading