feat: event-driven app state sync & spawner startup

[MorningLightMountain713]

Summary

Replaces the timer-based spawner startup (125-minute fixed wait) with an event-driven architecture that syncs all ephemeral app state from peers in seconds. A restarting node now reaches spawner readiness in ~30 seconds instead of 2+ hours. Also fixes a broken TTL on the install errors collection and enables the spawner to skip apps with network-wide install failures.

Key changes:

• AppSyncOrchestrator: State machine coordinating hash sync, DB rebuild, and spawner startup via events instead of timers
• Signed broadcast sync protocol: Four binary message types (0x20-0x23) for syncing temp messages, apprunning locations, appinstalling locations, and install errors from peers over WebSocket
• Spawner expiration filter: Pipeline-level check prevents spawning expired/cancelled apps (the original bug fix)
• Immediate promotion: Temp messages arriving for known hashes are promoted instantly instead of waiting for the 30-minute sweep
• Network-wide error check: Spawner skips apps with 5+ install failures across distinct nodes
• Install errors TTL fix: Broken cachedAt index replaced with working broadcastedAt (24-hour TTL)

The Problem

Cancelled enterprise apps (expire: 100) were being installed on nodes that restarted after the cancellation. Six root causes identified:

1. No expiration check in the spawner aggregation pipeline
2. Stale globalAppsInformation rebuilt at boot before hash sync
3. Slow hash sync: 30-minute loop, 500 at a time, one peer
4. 95% threshold skipping bulk sync for nodes down briefly
5. Missed gossip: P2P gossip is fire-and-forget, no replay
6. Timer-based spawner: starts on fixed clock regardless of DB state

Additionally, the install errors collection had a broken TTL (index on cachedAt which was never set — 19,000+ records accumulating indefinitely on production nodes) and an unsigned HTTP bulk fetch (getPeerAppsInstallingErrorMessages) that trusted one peer's aggregate data without verification.

Architecture

Before (timer-based)

T+0        Boot, rebuild DB from incomplete data
T+15-30m   Hash sync starts (30-min loop)
T+62m      Enterprise spawner (TIMER) — may use stale data
T+125m     Non-enterprise spawner (TIMER) — may use stale data

After (event-driven)

T+0        Boot, AppSyncOrchestrator starts
T+~30s     Peer threshold (12) → sync all 4 data types from peers
T+~31s     All syncs complete, location data ready
T+varies   Explorer syncs → hash sync → DB rebuild
T+varies   Node confirmed → spawnerReady

State Machine

INITIALIZING ──► SYNCING ──► READY
                   ▲            |
                   |            v
                RESYNCING ◄── DEGRADED

Signed Broadcast Sync Protocol

Message	Code	Data	Chunk Size
Temp Messages	0x20	App registrations/updates not yet on-chain	2000
App Running	0x21	Node → running apps mapping	2000
App Installing	0x22	In-progress installations	2000
Install Errors	0x23	Failed installations per node per app hash	2000

How it works

1. Receiver sends 9-byte binary request: [type:1][sinceTimestamp:8]
2. Sender opens MongoDB cursor (consistent snapshot), streams in chunks of 2000 — caps memory usage regardless of dataset size
3. Each chunk is a signed broadcast containing an array of inner signed broadcasts
4. Receiver verifies each inner broadcast signature against deterministic node list
5. Verified broadcasts bulk-written to both signed and location collections
6. Final chunk has done: true → receiver knows sync is complete

Security

• Two-layer verification: Outer broadcast proves a real node sent the response. Inner broadcasts are individually verified — pubkey checked against deterministic node list, signature verified via bitcoinjs-message
• Rate limiting: 5-minute cooldown per peer per sync type
• Clock offset adjustment: sinceTimestamp adjusted by peer.remoteClockOffsetMs before sender queries
• No trust in aggregates: Unlike the old getPeerAppsInstallingErrorMessages (which fetched unsigned data via HTTP from one peer), every synced record is cryptographically verified back to its originating node

Capability

Single appStateSync capability covers all four sync types. Defined once in FluxPeerSocket.js as FLUX_CAPABILITIES, used by both WebSocket client (outbound request headers) and server (inbound response headers). Previously capabilities were duplicated in two files with different values — fixed.

Sender-Side Filtering

Each sync type filters by validity on the sender side to avoid sending records that have technically expired but haven't been cleaned up yet by MongoDB's TTL thread (~60s lag):

• Running: Only sends records newer than 125 minutes (matches TTL)
• Installing: Only sends records newer than 15 minutes (matches TTL)
• Install errors: Only sends records newer than 24 hours (matches TTL)

Dual-Collection Storage

Each ephemeral data type stored in two collections:

• Signed collection (new): Full broadcast with pubKey, signature, data — used for serving sync requests
• Location collection (existing, unchanged): Existing flat rows — used by all existing queries (spawner, advancedWorkflows, etc.)

Both written on every gossip message. Installing collections cleaned up when app transitions to running.

Why duplicate instead of replacing? There are 15+ direct queries against the existing location collections scattered across the codebase (messageStore, peerNotification, fluxCommunication, nodeStatusMonitor, stoppedAppsRecovery, advancedWorkflows, appController, syncthingMonitor). A separate branch centralizes all these queries through registryManager. Changing the document shape in this PR would conflict with that work. The signed collections add ~1.7MB overhead (running) — trivial. Once the centralization branch is rebased on top of this, the dual-collection pattern can be collapsed into a single collection with an aggregation-based appLocation() query.

Signed	Location	TTL
fluxapprunningbroadcasts	zelappslocation	125 min
fluxappinstallingbroadcasts	appsinstallinglocations	15 min
fluxappinstallingerrorsbroadcasts	appsInstallingErrorsLocations	24 hours

Validity Windows

Data Type	Gossip Validity	Sync Validity	TTL
Running	5 min	125 min	125 min
Installing	5 min	15 min	15 min
Install Errors	5 min	24 hours	24 hours

Gossip validity is short because gossip messages are relayed — stale messages indicate network issues. Sync validity matches TTL because sync is a point-in-time snapshot of the sender's DB.

Install Errors Fix

What was broken

• TTL index on cachedAt — field never set in any document, nothing ever expired
• 19,304 records on production (chud), oldest 3 weeks, 98% had expireAt: null
• Null-expiry logic: when 5+ errors accumulated for same app+hash, set expireAt: null (persist forever)
• removeDocumentsFromCollection({}) wiped collection on every restart (inconsistent with other ephemeral collections)
• getPeerAppsInstallingErrorMessages fetched unsigned error data via HTTP from one random peer

What was fixed

• TTL index changed from cachedAt to broadcastedAt with 86400s (24-hour) expiry
• Removed null-expiry logic
• Removed startup wipe (consistent with other collections, TTL handles cleanup)
• Removed getPeerAppsInstallingErrorMessages — replaced by signed 0x23 sync
• Gossip validity reduced from 60 minutes to 5 minutes (consistent with installing messages)

Spawner network-wide error check

Previously commented out (appSpawner.js:336-340). Re-enabled with new logic:

const errorCount = await registryManager.countAppInstallingErrors(appHash);
if (errorCount >= 5) { // 5+ distinct nodes failed → broken spec
  globalState.spawnErrorsLongerAppCache.set(appHash, '');
  throw new Error(`...has ${errorCount} network-wide install failures, skipping`);
}

Two-layer error handling:

• Network DB (24h TTL): Prevents untried nodes from wasting time. ~5 nodes retry per day as errors expire
• Per-node in-memory (7 days): spawnErrorsLongerAppCache prevents nodes that already failed from retrying for a week

Other Changes

• Capability unification: FLUX_CAPABILITIES extracted to FluxPeerSocket.js as single source of truth (was duplicated between socketServer.js and fluxCommunication.js with different values)
• X-Flux-Uptime: Now sent in both inbound and outbound connection headers (was outbound only)
• peerNotification.js: Simplified from 10-parameter injection to direct globalState imports
• messageVerifier.js: Removed ~165 lines of duplicate code
• appHashSyncService.js: Rewritten — multi-peer parallel fetch, no 95% threshold skip
• Sync guard: #syncInProgress flag prevents overlapping syncs in orchestrator

Rollout Behavior

During rollout (mixed network): Nodes running new code will find no appStateSync peers and fall back to the block-count timer — 125 minutes for non-enterprise, 62 minutes for enterprise. This is the same behavior as today. All other improvements (hash sync, DB rebuild sequencing, spawner expiration filter, immediate promotion, network error check) apply immediately regardless.

After full network upgrade: Restarting nodes sync all location data from peers in ~1.5 seconds. Spawner can start as soon as explorer syncs + hash sync + DB rebuild completes — typically 2-5 minutes after restart instead of 2+ hours.

Test Results

Tested on two dedicated test nodes.

Test 1: Temp Message Sync

• Stopped sandwich, registered 2 test apps while it was down
• Restarted sandwich, peered to squidward
• Sandwich received 3 temp messages (chunkymonkey + 2 test apps), all processed
• Verified via /apps/temporarymessages API

Test 2: App Running Sync (full, from clean slate)

handleAppRunningSyncResponse - Received 2000 broadcasts (done: false)
handleAppRunningSyncResponse - Received 1336 broadcasts (done: true)
handleAppRunningSyncResponse - Stored 2000 of 2000 verified broadcasts
handleAppRunningSyncResponse - Stored 1335 of 1335 verified broadcasts

1 broadcast failed: nodeNotFound (node dropped off network between storage and verification — expected)

Test 3: App Installing Sync

handleAppInstallingSyncResponse - Received 57 broadcasts (done: true)
handleAppInstallingSyncResponse - Stored 57 of 57 verified broadcasts

Test 4: Install Errors Sync

handleAppInstallingErrorsSyncResponse - Received 3 broadcasts (done: true)
handleAppInstallingErrorsSyncResponse - Stored 3 of 3 verified broadcasts

Test 5: Collection Consistency

running broadcasts: 3335  apps: 5165  locations: 5166  match: ~✓ (1 from gossip)
installing broadcasts: 56  locations: 56  match: true
error broadcasts: 3  error locations: 3  match: true

Test 6: Spawner Error Check

Inserted 6 test error records for a fake hash. Verified:

countDocuments({hash: "testhash..."}) = 6 → would skip (>=5): true
countDocuments({hash: "nonexistent"}) = 0 → would skip (>=5): false

Test 7: Capability Advertisement

Fixed and verified: both inbound (server response) and outbound (client request) now advertise appStateSync. Previously server-side was missing new capabilities.

Test 8: Installing Validity Window

Initial test showed 132 received, 44 stored. Root cause: sender was sending records up to 15 minutes old (TTL window) but receiver only accepted 5 minutes (gossip validity). Fixed: sender now filters to validity window, 100% stored on subsequent tests.

Performance

T+0.0s   Boot
T+28.0s  Peer threshold reached (12 peers)
T+28.0s  All 4 sync requests sent
T+28.2s  Temp: 1 received, processed
T+28.3s  Installing: 57 received, verified, stored
T+28.3s  Errors: 3 received, verified, stored
T+28.7s  Running chunk 1: 2000 received
T+29.1s  Running chunk 2: 1336 received
T+29.7s  All syncs complete

Total sync time: ~1.5 seconds for full network state

Files Changed

File	Change
config/default.js	New collection names, peer thresholds
utils/peerCodec.js	0x20 sinceTimestamp, 0x21, 0x22, 0x23 encode/decode
utils/FluxPeerSocket.js	FLUX_CAPABILITIES constant
utils/FluxPeerManager.js	Binary handlers for 0x21-0x23, eligibility methods, threshold events
utils/globalState.js	appRunningSyncComplete, spawnerPaused
lib/socketServer.js	Uses shared FLUX_CAPABILITIES, adds X-Flux-Uptime
appMessaging/messageStore.js	Signed broadcast storage, batch operations, cleanup, error TTL fix
appMessaging/appSyncOrchestrator.js	Event-driven coordinator
appMessaging/appHashSyncService.js	Rewritten hash sync
appMessaging/messageVerifier.js	Removed duplicate code
appMessaging/peerNotification.js	Direct imports, simplified signature
fluxCommunication.js	Sync handlers, dispatch routes, capability, X-Flux-Uptime
fluxCommunicationMessagesSender.js	Cursor-based streaming responders
serviceManager.js	Collection indexes, orchestrator wiring, TTL fixes
appLifecycle/appSpawner.js	Event-driven startup, pause/resume, network error check
appLifecycle/advancedWorkflows.js	Removed getPeerAppsInstallingErrorMessages
appDatabase/registryManager.js	countAppInstallingErrors, cleaned projections

Test plan

• Deploy on 2 test nodes, verify full sync lifecycle from clean slate
• Verify all 4 sync types: temp messages, running, installing, errors
• Verify collection consistency after sync and gossip
• Verify installing cleanup: both collections cleaned on app→running transition
• Verify capability advertisement in both connection directions
• Verify sender-side validity filtering (installing, errors)
• Verify spawner error check logic (count >= 5 threshold)
• Verify spawner starts after sync instead of after 125-minute timer (requires full network upgrade)
• Verify cancelled app is not reinstalled after sync
• Verify fallback: no appStateSync peers → block-count timer activates
• Run on mixed network (new + old nodes) — new nodes fall back gracefully
• Pre-PR: Change uptime threshold from 60s to 7500s (done, 4 locations — matches apprunning TTL)

Before merging

• Final code review
• Extended testing on test nodes (leave running for 24+ hours, verify TTL cleanup, error accumulation, spawner behavior)
• Review all commit messages and squash if needed

[codecov]

Codecov Report

❌ Patch coverage is 35.02890% with 562 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.08%. Comparing base (8412550) to head (f2eb3f3).
⚠️ Report is 32 commits behind head on development.

Files with missing lines	Patch %	Lines
...k/src/services/appLifecycle/stoppedAppsRecovery.js	5.26%	126 Missing ⚠️
ZelBack/src/services/fluxCommunication.js	3.66%	105 Missing ⚠️
ZelBack/src/services/appMessaging/messageStore.js	20.38%	82 Missing ⚠️
...ck/src/services/fluxCommunicationMessagesSender.js	2.38%	82 Missing ⚠️
...ck/src/services/appMessaging/appHashSyncService.js	48.71%	60 Missing ⚠️
ZelBack/src/services/utils/FluxPeerManager.js	26.00%	37 Missing ⚠️
...Back/src/services/appMessaging/peerNotification.js	13.04%	20 Missing ⚠️
ZelBack/src/services/appLifecycle/appSpawner.js	34.78%	15 Missing ⚠️
...elBack/src/services/appDatabase/registryManager.js	7.14%	13 Missing ⚠️
...k/src/services/appMessaging/appSyncOrchestrator.js	92.39%	13 Missing ⚠️
... and 4 more

Additional details and impacted files

@@               Coverage Diff               @@
##           development    #1724      +/-   ##
===============================================
+ Coverage        55.02%   55.08%   +0.05%     
===============================================
  Files              135      139       +4     
  Lines            27734    28442     +708     
===============================================
+ Hits             15260    15666     +406     
- Misses           12474    12776     +302     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[MorningLightMountain713]

Superseded by #1726 which includes the event log approach.