feat: CLI runner for runtime tests, one library at a time#9930
Open
tjzel wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This PR description was AI-generated.
Stacked on #9929. Part 4 of the runtime-tests spec.
Summary
I added a CLI runner that builds, installs and launches the fabric-example app in a dedicated
DebugRuntimeTestsconfiguration, auto-runs the runtime tests for exactly one library, streams logs back over WebSocket, and exits with the run's status —yarn runtime-tests --library worklets(orreanimated).How a run works.
scripts/runtime-tests-server.jsstarts a WebSocket server (Metro port + 1), ensures Metro is up (reusing a healthy one or spawningyarn start), resolves the simulator (--udid/--simulator, falling back to the first available iPhone), builds withxcodebuild -scheme DebugRuntimeTests, installs viasimctl, writesRCT_jsLocationso the app finds the right Metro, and launches withSIMCTL_CHILD_RUNTIME_TESTS_LIBRARY=<library>. The app connects back, sends ahellonaming its library and declared suites (the server validates both), receivesstart(optionally with an--onlysuite filter), runs the tests, and reportsdonewith a summary; the server exits 0/1 accordingly.--skip-buildreuses the previous build for fast iteration.One library at a time, at the native level. The
RUNTIME_TESTScompilation condition makes the AppDelegate pick the bundle rootindex.runtimeTests.<library>from theRUNTIME_TESTS_LIBRARYlaunch environment variable. Each entry registers the same app module but with its own component:import 'react-native-reanimated', so Reanimated's commit hook installs at boot, before the first React commit — this removes the known race that crashed the experimental LayoutAnimations commit hook when Reanimated was lazy-loaded mid-run, without any of the C++ workarounds explored earlier;[isolation]failure if it was.Making ReJest Reanimated-free for Worklets runs. The framework used to hard-import Reanimated in nine files, which would have dragged it into every run. I refactored so Reanimated only loads when a Reanimated-specific feature is used:
CallTrackerRegistryandmockConsole(rawMatchers) now usecreateSynchronizablefromreact-native-workletsinstead ofmakeMutable— these two are captured by worklets, so a lazyrequirewasn't an option;stringFormatUtils,Comparators,drawSnapshotTable) uses React Native's ownprocessColorvia a smallcolorUtilshelper;TestComponent.getAnimatedStyle,ValueRegistry.getRegisteredValue,UpdatesContainer.createUpdatesContainer,AnimationUpdatesRecorder.waitForAnimationUpdates) lazy-requireReanimated at first use.Framework improvements (from the earlier auto-run exploration, plus TODO.md items):
runTests()returns a{passed, failed, skipped, failedTests, durationMs}summary; a progress hook drives an on-device "Running test X of Y" display; uncaught exceptions from a test body,beforeEach,afterEach, or the per-test cleanup are recorded against the test instead of aborting the whole run; the server prints a friendly message on a busy port instead of anEADDRINUSEstack.Test plan
yarn type:check:nativeinapps/common-apppasses.yarn runtime-tests --library worklets— 1347 passed, 3 failed, 1 skipped in 104 s. The 3 failures (throws when trying to serialize a Promise,createSerializableHostObject×2) are the known error-path issues from TODO.md — the CLI's global error handler surfaces therunOnUISync/ErrorUtils.reportFatalErrorfast path that the matcher can't see; they are queued for the test-fixes part of the spec.yarn runtime-tests --library reanimated --skip-build— 211 passed, 1 failed in 204 s. The failure is an animation snapshot test (Animate hwb(70, 50%, 0%) → …) — a different test than the one that flaked in the manual-run verification, consistent with animation-snapshot timing flakiness under simulator load; also queued for the test-fixes part.react-native-reanimatednever entered the runtime.