Skip to content

feat: CLI runner for runtime tests, one library at a time#9930

Open
tjzel wants to merge 1 commit into
@tjzel/manual-runtime-testsfrom
@tjzel/runtime-tests-cli
Open

feat: CLI runner for runtime tests, one library at a time#9930
tjzel wants to merge 1 commit into
@tjzel/manual-runtime-testsfrom
@tjzel/runtime-tests-cli

Conversation

@tjzel

@tjzel tjzel commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Note

This PR description was AI-generated.

Stacked on #9929. Part 4 of the runtime-tests spec.

Summary

I added a CLI runner that builds, installs and launches the fabric-example app in a dedicated DebugRuntimeTests configuration, auto-runs the runtime tests for exactly one library, streams logs back over WebSocket, and exits with the run's status — yarn runtime-tests --library worklets (or reanimated).

How a run works. scripts/runtime-tests-server.js starts a WebSocket server (Metro port + 1), ensures Metro is up (reusing a healthy one or spawning yarn start), resolves the simulator (--udid / --simulator, falling back to the first available iPhone), builds with xcodebuild -scheme DebugRuntimeTests, installs via simctl, writes RCT_jsLocation so the app finds the right Metro, and launches with SIMCTL_CHILD_RUNTIME_TESTS_LIBRARY=<library>. The app connects back, sends a hello naming its library and declared suites (the server validates both), receives start (optionally with an --only suite filter), runs the tests, and reports done with a summary; the server exits 0/1 accordingly. --skip-build reuses the previous build for fast iteration.

One library at a time, at the native level. The RUNTIME_TESTS compilation condition makes the AppDelegate pick the bundle root index.runtimeTests.<library> from the RUNTIME_TESTS_LIBRARY launch environment variable. Each entry registers the same app module but with its own component:

  • the Reanimated entry's first statement is import 'react-native-reanimated', so Reanimated's commit hook installs at boot, before the first React commit — this removes the known race that crashed the experimental LayoutAnimations commit hook when Reanimated was lazy-loaded mid-run, without any of the C++ workarounds explored earlier;
  • the Worklets entry contains no Reanimated anywhere in its module graph, and the runner asserts after the run that Reanimated was never loaded (checking its init globals), appending a synthetic [isolation] failure if it was.

Making ReJest Reanimated-free for Worklets runs. The framework used to hard-import Reanimated in nine files, which would have dragged it into every run. I refactored so Reanimated only loads when a Reanimated-specific feature is used:

  • CallTrackerRegistry and mockConsole (rawMatchers) now use createSynchronizable from react-native-worklets instead of makeMutable — these two are captured by worklets, so a lazy require wasn't an option;
  • color formatting (stringFormatUtils, Comparators, drawSnapshotTable) uses React Native's own processColor via a small colorUtils helper;
  • the genuinely Reanimated-only paths (TestComponent.getAnimatedStyle, ValueRegistry.getRegisteredValue, UpdatesContainer.createUpdatesContainer, AnimationUpdatesRecorder.waitForAnimationUpdates) lazy-require Reanimated at first use.

Framework improvements (from the earlier auto-run exploration, plus TODO.md items): runTests() returns a {passed, failed, skipped, failedTests, durationMs} summary; a progress hook drives an on-device "Running test X of Y" display; uncaught exceptions from a test body, beforeEach, afterEach, or the per-test cleanup are recorded against the test instead of aborting the whole run; the server prints a friendly message on a busy port instead of an EADDRINUSE stack.

Test plan

  • yarn type:check:native in apps/common-app passes.
  • End-to-end on iPhone 17 Pro simulator (results below):
    • yarn runtime-tests --library worklets1347 passed, 3 failed, 1 skipped in 104 s. The 3 failures (throws when trying to serialize a Promise, createSerializableHostObject ×2) are the known error-path issues from TODO.md — the CLI's global error handler surfaces the runOnUISync/ErrorUtils.reportFatalError fast path that the matcher can't see; they are queued for the test-fixes part of the spec.
    • yarn runtime-tests --library reanimated --skip-build211 passed, 1 failed in 204 s. The failure is an animation snapshot test (Animate hwb(70, 50%, 0%) → …) — a different test than the one that flaked in the manual-run verification, consistent with animation-snapshot timing flakiness under simulator load; also queued for the test-fixes part.
  • The Worklets run's isolation assertion confirms react-native-reanimated never entered the runtime.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant