qbft: clarify round-changes by iurii-ssv · Pull Request #2835 · ssvlabs/ssv

iurii-ssv · 2026-04-27T16:26:27Z

See Greptile summary below.

Closes https://github.com/ssvlabs/ssv-node-board/issues/1004

Closes ssvlabs/ssv-node-board#1052

codecov · 2026-04-27T16:27:57Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.3%. Comparing base (0fa3b4a) to head (82c4b6b).

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

greptile-apps · 2026-04-27T16:28:43Z

Greptile Summary

This PR clarifies round-change semantics in the QBFT protocol implementation, with two primary improvements: (1) it replaces specqbft.HashDataRoot (which returned an unreachable error since SHA-256 never fails) with a new local qbft.HashDataRoot that directly wraps sha256.Sum256, simplifying callsites throughout the codebase; and (2) it fixes round-advancement logic so that bumpToRound now includes both the guard (round > i.State.Round) and the timer reset in a single atomic step, and the round bump now happens before broadcast in both the timeout and partial-quorum paths (explicitly "best-effort" semantics).

Notable behavioral change: in UponRoundTimeout, when a node reaches CutOffRound-1, the round is now bumped to CutOffRound before Broadcast is called, causing the broadcast to fail with "instance is no longer considered relevant" — whereas previously the round-change message was successfully broadcast then the round was bumped via a deferred call. The test TestUponRoundTimeoutStopsProcessingAfterReachingCutOffRound explicitly validates this new behavior.

Confidence Score: 5/5

Safe to merge; changes are well-reasoned refactors with intentional behavioral adjustments and matching test updates.

No P0 or P1 issues found. The HashDataRoot refactoring correctly eliminates an unreachable error (SHA-256 never fails). The bumpToRound fix — guard + timer in one place — is a clean improvement and directly resolves the previous review finding. The order-of-operations change in UponRoundTimeout (bump before broadcast) is intentional and validated by updated tests. Only a single P2 observation about lost observability on the reasonJustified metric.

No files require special attention beyond the minor observability note in observability.go.

Important Files Changed

Filename	Overview
protocol/v2/qbft/instance/instance.go	bumpToRound gains guard (round > current) + timer scheduling; CanProcessMessages renamed to IsRelevant; HashDataRoot error handling removed
protocol/v2/qbft/instance/timeout.go	Removes deferred round bump — now bumps round before broadcast; timer folded into bumpToRound; moves RecordRoundChange before CreateRoundChange
protocol/v2/qbft/instance/round_change.go	Removes reasonJustified metric, adds ctx/span to uponChangeRoundPartialQuorum, removes i.roundTimer.TimeoutForRound (now in bumpToRound); replaces prevRound with i.State.Round inline (safe since round never changes in that path)
protocol/v2/qbft/instance/proposal.go	Captures currentRound before bumpToRound for correct metrics; fixes timer reset (now inside bumpToRound); pre-allocates slices with capacity; HashDataRoot error removed
protocol/v2/qbft/messages.go	New file: local HashDataRoot wrapping sha256.Sum256 with no error return, simplifying all callsites
protocol/v2/qbft/instance/observability.go	Removes reasonJustified constant; only reasonTimeout and reasonPartialQuorum remain
protocol/v2/qbft/instance/timeout_test.go	Updates assertions to validate new pre-broadcast bump behavior; CutOffRound test now expects error on first timeout (round bumped to CutOffRound, broadcast rejected)

Sequence Diagram

sequenceDiagram
    participant I as Instance
    participant N as Network

    Note over I: UponRoundTimeout (NEW order)
    I->>I: prevRound = i.State.Round
    I->>I: bumpToRound(prevRound+1) sets Round + TimeoutForRound
    I->>I: ProposalAcceptedForCurrentRound = nil
    I->>I: CreateRoundChange(newRound)
    I->>N: Broadcast(roundChange) - fails if Round >= CutOffRound

    Note over I: uponChangeRoundPartialQuorum (NEW)
    I->>I: bumpToRound(newRound) sets Round + TimeoutForRound
    I->>I: ProposalAcceptedForCurrentRound = nil
    I->>I: CreateRoundChange(newRound)
    I->>N: Broadcast(roundChange)

    Note over I: uponProposal (FIXED)
    I->>I: currentRound = i.State.Round
    I->>I: ProposalAcceptedForCurrentRound = msg
    I->>I: bumpToRound(msgRound) guard + timer
    I->>I: CreatePrepare(msgRound, root)
    I->>N: Broadcast(prepare)

_{Reviews (2): Last reviewed commit: "qbft: clarify round-changes" | Re-trigger Greptile}

iurii-ssv · 2026-04-27T17:02:35Z

@greprile pls re-review

iurii-ssv · 2026-04-28T07:58:37Z

Notable behavioral change: in UponRoundTimeout, when a node reaches CutOffRound-1, the round is now bumped to CutOffRound before Broadcast is called, causing the broadcast to fail with "instance is no longer considered relevant" — whereas previously the round-change message was successfully broadcast then the round was bumped via a deferred call. The test TestUponRoundTimeoutStopsProcessingAfterReachingCutOffRound explicitly validates this new behavior.

This ^ is slight behavior improvement/correction, since there is no point to broadcast a message with CutOffRound as it will simply be ignored/dropped by other nodes.

Only a single P2 observation about lost observability on the reasonJustified metric.

That reasonJustified metric was incorrect btw, it was recorded "as if" it was a round-change ... while by the time that branch executes, we are already in that new round, and just proposing. So the original implementation/classification for this metric was a bit mislead by the complex handling code.

momosh-ssv · 2026-04-30T10:23:03Z

+	// Always move on to the next round. The round-change message broadcast is a best-effort thing, the QBFT
+	// cluster as a whole can progress further even if our round-change message cannot be created/broadcast
+	// for whatever reason.
+	i.bumpToRound(newRound)


Could this lead to a liveness regression at cutoff? The bump now runs before Broadcast, so when prevRound + 1 == CutOffRound the bump succeeds but i.Broadcast calls IsRelevant() which returns false (State.Round < CutOffRound no longer holds), and the round-change is silently dropped. A node that hits cutoff one tick before its peers will not send the RC that could have helped them advance. The new test on lines 62-74 actually encodes this behavior. Maybe broadcast before bumping (revert to the defer-style ordering), or relax IsRelevant for Broadcast so the cutoff round can still emit its final RC?

Intentional. Round timers are deterministic and started in lockstep at the start of a duty — once a node hits CutOffRound, peers running the same schedule are at most one round-tick behind and will reach it on their own shortly. The marginal value of broadcasting one extra RC at cutoff is essentially zero, and we preferred the cleaner bump-before-broadcast ordering for consistency with the partial-quorum and justified paths.

momosh-ssv · 2026-04-30T10:23:03Z

 	}

 	if justifiedRoundChangeMsg != nil {
+		i.metrics.EndStage(ctx, i.State.Round)


Should we consider only advancing the stage after the broadcast actually succeeds? EndStage / StartStage now run before CreateProposal / Broadcast here (and similarly at lines 110-112 for the partial-quorum path). I fear if CreateProposal or MarshalJustifications errors, the metric reports stageProposal while nothing was emitted, which can make dashboards misleading.

Keeping consistent with the "best-effort broadcast" framing this PR uses throughout: the protocol stage advances when we move on (we've already built the new message and the state machine has progressed); broadcast failures are tracked independently. Gating stage advancement on a successful broadcast would create a different inconsistency — on broadcast failure the instance would still be in the new round but the dashboard would show the old stage.

momosh-ssv · 2026-04-30T10:23:03Z

@@ -31,7 +31,6 @@ type roundChangeReason string
 const (
 	reasonTimeout       roundChangeReason = "timeout"
 	reasonPartialQuorum roundChangeReason = "partial-quorum"


What do you think about keeping reasonJustified? Once the leader-path round-bump issue (open thread on round_change.go) is addressed and a real round change happens in the justified path, there will be no labeled metric for it. Operators currently can distinguish 'round changed because of justified RC quorum' from 'timeout' / 'partial-quorum' — losing that makes it harder to diagnose why a cluster keeps round-changing.

Fair concern. After digging in, this metric was effectively double-counting in pre-PR code: by the time the leader-on-justified-quorum branch fired, State.Round had already been advanced (and recorded as a round change) via reasonTimeout or reasonPartialQuorum. The reasoning:

RoundChangeContainer is populated only inside uponRoundChange:34 — single call site (grep-verified).

Inside uponRoundChange, partial-quorum is checked after AddFirstMsgForSignerAndRound on every call.

Partial-quorum threshold (f+1) is strictly less than full quorum (2f+1) by definition.

processMsgF.Run() serializes processing.

So as RCs stream in, the (f+1)-th always triggers uponChangeRoundPartialQuorum → bumps State.Round — before the (2f+1)-th arrives and the leader branch fires. By the time the leader branch executes, bumpToRound(newRound) is always a no-op (State.Round == newRound already). No real round change happens in this branch.

Soft confirmation: if this future-round leader case were a real scenario in normal flow, ssv-spec tests would have a scenario for it — they don't. Leaving the metric removed.

(Julien's bumpToRound fix in 7ddb771 is still worth keeping for defense-in-depth against future refactors, but the metric would just be a perpetual zero.)

momosh-ssv · 2026-04-30T10:23:03Z

+
+// HashDataRoot hashes input data to root.
+func HashDataRoot(data []byte) [32]byte {
+	return sha256.Sum256(data)


Worth considering a guard against silent divergence from spec? The new helper is byte-identical to specqbft.HashDataRoot today and messages_test.go checks it on a few inputs at fix-time. Seems that nothing in protocol/v2/qbft/spectest/ exercises this code path though, so a future ssv-spec hash change would let nodes hash differently from spec while make spec-test still passes. Maybe keep this as a thin wrapper (r, _ := specqbft.HashDataRoot(data); return r) or assert equivalence against the spec helper in CI?

Spec-divergence risk is covered by TestHashDataRootMatchesSpec — it asserts equivalence against specqbft.HashDataRoot on several inputs, so if the spec ever changes its hash output, make unit-test fails before anything ships.

y0sher · 2026-05-11T10:48:08Z

@momosh-ssv @Roy-blox @iurii-ssv
this change QBFT code in critical paths, lets finish with code review and test thoroughly if we want it in the version. we'll want some mileage in staging env.

hasReceivedProposalJustificationForLeadingRound accepts future-round RC quorums, but CreateProposal uses i.State.Round and MessagesForRound(i.State.Round) selects the round-change justifications - without bumping first, the leader would broadcast a stale-round proposal with empty justifications. The case is structurally unreachable via natural message flow (partial-quorum f+1 fires before full quorum 2f+1 on each uponRoundChange call), so the fix closes a latent inconsistency rather than a live bug. Test bypasses natural flow by preloading the container, kept as defense-in-depth.

The partial-quorum branch (f+1 < 2f+1) always fires on an earlier uponRoundChange call and advances State.Round before the leader branch can see a full quorum for that round - so bumpToRound here is always a no-op on natural message flow. Kept as defense-in-depth.

iurii-ssv · 2026-05-12T13:41:49Z

Note: force-pushed rebased commits as my and Momo's interleaving git merge produced an awkward commit history (pulling 4f78c80 in to this PR, which is fixed now)

momosh-ssv · 2026-05-12T17:38:06Z

Deferring to v2.4.4.

Not enough time remaining in the v2.4.3 window to soak this properly — touches the consensus path, so we want a full stage-soak rather than rushing the cut. Rolling to the next minor.

Tracking issue moved to milestone v2.4.4: ssvlabs/ssv-node-board#1052.

iurii-ssv · 2026-05-14T15:34:40Z

/test cluster=304,305,306,307 tests=(sanity,e2m,threshold),(bulk,e2m,threshold,5) network=hoodi_stage

ssv-test-bot · 2026-05-14T15:34:44Z

🔴 Test run for PR #2835

Branch: qbft-correct-instance-message-retries
Commit: 8db85bb

Status

2026-05-14T15:34:43Z Starting test run
2026-05-14T15:34:45Z Test run submitted successfully for commit 8db85bb
2026-05-14T15:34:46Z container image for commit 8db85bb5b74ac1f47187bd847c35a3f23207b1ba is build and present in container registry
2026-05-14T15:34:47Z using user defined node range cluster=304,305,306,307 on network=hoodi_stage
2026-05-14T15:34:49Z deploying commit 8db85bb5b74ac1f47187bd847c35a3f23207b1ba to ssv nodes 304,305,306,307
2026-05-14T15:35:35Z commit 8db85bb5b74ac1f47187bd847c35a3f23207b1ba is deployed to ssv nodes 304,305,306,307
2026-05-14T15:35:37Z started test executor with tests=(sanity,e2m,threshold),(bulk,e2m,threshold,5) and network=hoodi_stage

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2026-05-14T15:35:40.126065037Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2026-05-14T15:35:40.132130812Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2026-05-14T15:35:40.132167323Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[304,305,306,307],"tests":"(sanity,e2m,threshold),(bulk,e2m,threshold,5)"}
{"time":"2026-05-14T15:35:40.187319415Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"61411fa17765df34d0b299d94991b37b"}
{"time":"2026-05-14T15:35:40.18806221Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2026-05-14T15:35:40.188096607Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[304,305,306,386],"test_type":{"Sanity":{"ShouldRun":true,"E2M":true,"Threshold":true,"BulkAmount":0},"Bulk":{"ShouldRun":true,"E2M":true,"Threshold":true,"BulkAmount":5},"Isolated":{"Tests":null}}}
{"time":"2026-05-14T15:35:43.346360585Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2026-05-14T15:35:43.346439804Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2026-05-14T15:35:43.346450231Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2026-05-14T15:35:43.346469331Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":true}
{"time":"2026-05-14T15:35:43.346485777Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2026-05-14T15:35:43.346498887Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2026-05-14T15:35:43.346515372Z","level":"INFO","msg":"Sanity: config values","test":"sanity","cluster_owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","account_public_key":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","operator_ids":[304,305,306,386]}
{"time":"2026-05-14T15:35:43.346592602Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2026-05-14T15:35:43.396906984Z","level":"INFO","msg":"Available slots","test":"sanity","node":304,"slots":3000}
{"time":"2026-05-14T15:35:43.403422588Z","level":"INFO","msg":"Available slots","test":"sanity","node":305,"slots":1011}
{"time":"2026-05-14T15:35:43.410284273Z","level":"INFO","msg":"Available slots","test":"sanity","node":306,"slots":3000}
{"time":"2026-05-14T15:35:43.418445387Z","level":"INFO","msg":"Available slots","test":"sanity","node":386,"slots":1325}
{"time":"2026-05-14T15:36:15.624338399Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-14T15:36:15.624397391Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-14T15:36:15.630543893Z","level":"WARN","msg":"⚠️  Sanity preflight found pre-registered validator - attempting cleanup before starting the flow","test":"sanity","validator_pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27"}
{"time":"2026-05-14T15:36:20.031637017Z","level":"INFO","msg":"Trying to remove validator...","test":"sanity","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","validator_pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","nodes":[304,305,306,386]}
{"time":"2026-05-14T15:36:20.060112455Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"removeValidator","gas_estimate_error":"gas estimation failed: execution reverted","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-14T15:36:20.072580278Z","level":"INFO","msg":"4️⃣ Removing validator","test":"sanity","module":"contract","tx_hash":"0x043309e922ffb992421b95bb66ca8465c799f0ce2f0fa19982b5e613ffe5d5da"}
{"time":"2026-05-14T15:36:20.075523255Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x043309e922ffb992421b95bb66ca8465c799f0ce2f0fa19982b5e613ffe5d5da"}
{"time":"2026-05-14T15:36:30.086460395Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x043309e922ffb992421b95bb66ca8465c799f0ce2f0fa19982b5e613ffe5d5da"}
{"time":"2026-05-14T15:36:50.094392996Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x043309e922ffb992421b95bb66ca8465c799f0ce2f0fa19982b5e613ffe5d5da"}
{"time":"2026-05-14T15:36:50.094452298Z","level":"ERROR","msg":"Error checking transaction success","test":"sanity","txHash":"0x043309e922ffb992421b95bb66ca8465c799f0ce2f0fa19982b5e613ffe5d5da"}
{"time":"2026-05-14T15:36:50.094467621Z","level":"WARN","msg":"remove validator retry failed","test":"sanity","attempt":1,"max_attempts":3,"error":"transaction verification failed"}
{"time":"2026-05-14T15:36:56.401841252Z","level":"INFO","msg":"Trying to remove validator...","test":"sanity","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","validator_pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","nodes":[304,305,306,386]}
{"time":"2026-05-14T15:36:56.407072917Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"removeValidator","gas_estimate_error":"gas estimation failed: execution reverted","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-14T15:36:56.421555969Z","level":"INFO","msg":"4️⃣ Removing validator","test":"sanity","module":"contract","tx_hash":"0xfa9bf6796e03499b46ce117c9008cca73542b05f648170b5c27e66220aaa6cd6"}
{"time":"2026-05-14T15:36:56.425047201Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xfa9bf6796e03499b46ce117c9008cca73542b05f648170b5c27e66220aaa6cd6"}
{"time":"2026-05-14T15:37:06.432508086Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xfa9bf6796e03499b46ce117c9008cca73542b05f648170b5c27e66220aaa6cd6"}
{"time":"2026-05-14T15:37:06.432572491Z","level":"ERROR","msg":"Error checking transaction success","test":"sanity","txHash":"0xfa9bf6796e03499b46ce117c9008cca73542b05f648170b5c27e66220aaa6cd6"}
{"time":"2026-05-14T15:37:06.432587356Z","level":"WARN","msg":"remove validator retry failed","test":"sanity","attempt":2,"max_attempts":3,"error":"transaction verification failed"}
{"time":"2026-05-14T15:37:15.018036735Z","level":"INFO","msg":"Trying to remove validator...","test":"sanity","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","validator_pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","nodes":[304,305,306,386]}
{"time":"2026-05-14T15:37:15.022643155Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"removeValidator","gas_estimate_error":"gas estimation failed: execution reverted","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-14T15:37:15.045914061Z","level":"INFO","msg":"4️⃣ Removing validator","test":"sanity","module":"contract","tx_hash":"0xd3cd4202f89cf22cddf2ff929fbf4a97090b970c884bbbd136c36befb6eb9758"}
{"time":"2026-05-14T15:37:15.063237012Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xd3cd4202f89cf22cddf2ff929fbf4a97090b970c884bbbd136c36befb6eb9758"}
{"time":"2026-05-14T15:37:25.076143977Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xd3cd4202f89cf22cddf2ff929fbf4a97090b970c884bbbd136c36befb6eb9758"}
{"time":"2026-05-14T15:37:25.076231865Z","level":"ERROR","msg":"Error checking transaction success","test":"sanity","txHash":"0xd3cd4202f89cf22cddf2ff929fbf4a97090b970c884bbbd136c36befb6eb9758"}
{"time":"2026-05-14T15:37:25.076248622Z","level":"WARN","msg":"remove validator retry failed","test":"sanity","attempt":3,"max_attempts":3,"error":"transaction verification failed"}
{"time":"2026-05-14T15:37:25.07627344Z","level":"WARN","msg":"ℹ️ Tracking unresolved cleanup action","test":"sanity","test_type":"sanity","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","cluster":[304,305,306,386],"pubkeys":["0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754"],"failure_stage":"preflight_existing_validator","desired_action":"remove_validator"}
{"time":"2026-05-14T15:37:25.088147481Z","level":"ERROR","msg":"❌ FAILURE: Sanity Test Suite Failed","module":"main","error":"failed to remove pre-registered sanity validator before run: remove validator retries exhausted: transaction verification failed"}
{"time":"2026-05-14T15:37:25.088190035Z","level":"INFO","msg":"🔑 KEY STEP: Starting Bulk Test Suite","module":"main"}
{"time":"2026-05-14T15:37:25.088211219Z","level":"INFO","msg":"Bulk: config values","test":"bulk","cluster_owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","account_public_key":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","operator_ids":[304,305,306,386]}
{"time":"2026-05-14T15:37:25.117798874Z","level":"INFO","msg":"Starting cluster cleanup replay before running tests","test":"bulk","events_count":1,"network":"hoodi_stage","test_type":"bulk"}
{"time":"2026-05-14T15:37:25.117856402Z","level":"INFO","msg":"🧹 Cluster cleanup replay finished","test":"bulk","test_type":"bulk","replayed_count":0}
{"time":"2026-05-14T15:37:25.128784751Z","level":"INFO","msg":"Available slots","test":"bulk","node":304,"slots":3000}
{"time":"2026-05-14T15:37:25.135855947Z","level":"INFO","msg":"Available slots","test":"bulk","node":305,"slots":1011}
{"time":"2026-05-14T15:37:25.142381392Z","level":"INFO","msg":"Available slots","test":"bulk","node":306,"slots":3000}
{"time":"2026-05-14T15:37:25.149841487Z","level":"INFO","msg":"Available slots","test":"bulk","node":386,"slots":1325}
{"time":"2026-05-14T15:38:13.254451788Z","level":"INFO","msg":"Extracted keyshares path for bulk keyshare","test":"bulk","module":"keys","path":"/app/shared/data/keyshares/keyshares-1778773093.json"}
{"time":"2026-05-14T15:38:17.735054476Z","level":"INFO","msg":"Cluster asset type before write","test":"bulk","asset_type":1,"asset_err":null,"asset_label":"ETH"}
{"time":"2026-05-14T15:38:22.033398217Z","level":"WARN","msg":"⚠️  Bulk preflight found pre-registered validators - attempting cleanup before starting","test":"bulk","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","cluster":[304,305,306,386],"pre_registered_count":3}
{"time":"2026-05-14T15:38:26.469545512Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"bulkRemoveValidator","gas_estimate_error":"gas estimation failed: execution reverted","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-14T15:38:26.482928164Z","level":"INFO","msg":"4️⃣ Bulk Removing validators","test":"bulk","module":"contract","tx_hash":"0x083da524e0b17db5601dbba8d81bf69dd74408d8726143384c7d76cf1a846195"}
{"time":"2026-05-14T15:38:26.488783604Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x083da524e0b17db5601dbba8d81bf69dd74408d8726143384c7d76cf1a846195"}
{"time":"2026-05-14T15:38:36.492974055Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x083da524e0b17db5601dbba8d81bf69dd74408d8726143384c7d76cf1a846195"}
{"time":"2026-05-14T15:38:56.501625214Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":3,"sleep":40,"tx_hash":"0x083da524e0b17db5601dbba8d81bf69dd74408d8726143384c7d76cf1a846195"}
{"time":"2026-05-14T15:39:36.54148207Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x083da524e0b17db5601dbba8d81bf69dd74408d8726143384c7d76cf1a846195"}
{"time":"2026-05-14T15:39:36.541578643Z","level":"WARN","msg":"bulk remove retry failed","test":"bulk","attempt":1,"max_attempts":3,"error":"transaction verification failed","pubkeys_count":3}
{"time":"2026-05-14T15:39:42.972906982Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"bulkRemoveValidator","gas_estimate_error":"gas estimation failed: execution reverted","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-14T15:39:42.9884942Z","level":"INFO","msg":"4️⃣ Bulk Removing validators","test":"bulk","module":"contract","tx_hash":"0x2e4b9f51c01340e207538d54b3026bc2b79dd75cd4ed33f97ac9e77fc3d5a972"}
{"time":"2026-05-14T15:39:42.992431194Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x2e4b9f51c01340e207538d54b3026bc2b79dd75cd4ed33f97ac9e77fc3d5a972"}
{"time":"2026-05-14T15:39:53.006382827Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x2e4b9f51c01340e207538d54b3026bc2b79dd75cd4ed33f97ac9e77fc3d5a972"}
{"time":"2026-05-14T15:40:13.031758338Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x2e4b9f51c01340e207538d54b3026bc2b79dd75cd4ed33f97ac9e77fc3d5a972"}
{"time":"2026-05-14T15:40:13.031804074Z","level":"WARN","msg":"bulk remove retry failed","test":"bulk","attempt":2,"max_attempts":3,"error":"transaction verification failed","pubkeys_count":3}
{"time":"2026-05-14T15:40:21.19844796Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"bulkRemoveValidator","gas_estimate_error":"gas estimation failed: execution reverted","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-14T15:40:21.214393576Z","level":"INFO","msg":"4️⃣ Bulk Removing validators","test":"bulk","module":"contract","tx_hash":"0x51c17c245b786cf5d9eca2505f1a6928fc3630435981fe5fd31533c463fa1ed6"}
{"time":"2026-05-14T15:40:21.218168116Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x51c17c245b786cf5d9eca2505f1a6928fc3630435981fe5fd31533c463fa1ed6"}
{"time":"2026-05-14T15:40:31.227312503Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x51c17c245b786cf5d9eca2505f1a6928fc3630435981fe5fd31533c463fa1ed6"}
{"time":"2026-05-14T15:40:31.227396819Z","level":"WARN","msg":"bulk remove retry failed","test":"bulk","attempt":3,"max_attempts":3,"error":"transaction verification failed","pubkeys_count":3}
{"time":"2026-05-14T15:40:31.227479474Z","level":"WARN","msg":"ℹ️ Tracking unresolved cleanup action","test":"bulk","test_type":"bulk","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","cluster":[304,305,306,386],"pubkeys":["0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","0x874e9b0b577bb6c9dbd7bad54e19306422288cc9d9cced8522fbb3990a55c0c82586a9b0c8eca964aa38306179b42856","0xa3a4b31b4ab322e432cec4719f3795d46e707f82f3e4a0930550edf09643712e61e947eea441ee037bb947868e9b0898"],"failure_stage":"preflight_existing_validators","desired_action":"bulk_remove"}
{"time":"2026-05-14T15:40:31.238119705Z","level":"ERROR","msg":"❌ FAILURE: Bulk Test Suite Failed","module":"main","error":"failed to remove pre-registered bulk validators before run: bulk remove retries exhausted: transaction verification failed. Stored cleanup event in cluster_cleanup_events table"}
{"time":"2026-05-14T15:40:31.23817551Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2026-05-14T15:40:31.238210298Z","level":"ERROR","msg":"❌ FAILURE: Test Execution Completed with Failures","module":"main","passed_suites":null,"failed_suites":["sanity","bulk"],"skipped_suites":["isolated"]}
{"time":"2026-05-14T15:40:31.238250788Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"sanity","error":"failed to remove pre-registered sanity validator before run: remove validator retries exhausted: transaction verification failed"}
{"time":"2026-05-14T15:40:31.238261608Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"bulk","error":"failed to remove pre-registered bulk validators before run: bulk remove retries exhausted: transaction verification failed. Stored cleanup event in cluster_cleanup_events table"}
{"time":"2026-05-14T15:40:31.238268931Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"61411fa17765df34d0b299d94991b37b"}
{"time":"2026-05-14T15:40:31.240360119Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"61411fa17765df34d0b299d94991b37b"}
{"time":"2026-05-14T15:40:31.269906674Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"61411fa17765df34d0b299d94991b37b"}

2026-05-14T15:40:35Z 🟠 The test run failed. Resources are NOT deleted yet — held for 15 minutes for debugging (until 2026-05-14T15:55:34Z UTC). Please do NOT rerun this test until cleanup completes. To force immediate cleanup, comment /test cancel.

Result - 🔴 Failure

2026-05-14T15:55:43Z 🔴 Resources cleaned up — test run is finalised. You can rerun the test now.

AlexM-ssv-labs · 2026-05-15T08:12:19Z

/test cluster=304,305,306,307 tests=(sanity,e2m),(bulk,e2m)

ssv-test-bot · 2026-05-15T08:12:23Z

🔴 Test run for PR #2835

Branch: qbft-correct-instance-message-retries
Commit: 8db85bb

Status

2026-05-15T08:12:22Z Starting test run
2026-05-15T08:12:24Z Test run submitted successfully for commit 8db85bb
2026-05-15T08:12:25Z container image for commit 8db85bb5b74ac1f47187bd847c35a3f23207b1ba is build and present in container registry
2026-05-15T08:12:26Z using user defined node range cluster=304,305,306,307 on network=hoodi_stage
2026-05-15T08:12:28Z deploying commit 8db85bb5b74ac1f47187bd847c35a3f23207b1ba to ssv nodes 304,305,306,307
2026-05-15T08:13:45Z commit 8db85bb5b74ac1f47187bd847c35a3f23207b1ba is deployed to ssv nodes 304,305,306,307
2026-05-15T08:13:46Z started test executor with tests=(sanity,e2m),(bulk,e2m) and network=hoodi_stage

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2026-05-15T08:13:49.567666073Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2026-05-15T08:13:49.578819338Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2026-05-15T08:13:49.57888215Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[304,305,306,307],"tests":"(sanity,e2m),(bulk,e2m)"}
{"time":"2026-05-15T08:13:49.627572623Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"76b29e68939ec21d2cf391af9acd6d3f"}
{"time":"2026-05-15T08:13:49.628169058Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2026-05-15T08:13:49.628206282Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[304,305,306,307],"test_type":{"Sanity":{"ShouldRun":true,"E2M":true,"Threshold":false,"BulkAmount":0},"Bulk":{"ShouldRun":true,"E2M":true,"Threshold":false,"BulkAmount":3},"Isolated":{"Tests":null}}}
{"time":"2026-05-15T08:13:52.863539497Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2026-05-15T08:13:52.863705638Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2026-05-15T08:13:52.863720894Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2026-05-15T08:13:52.86374107Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":true}
{"time":"2026-05-15T08:13:52.863754569Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2026-05-15T08:13:52.863768067Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2026-05-15T08:13:52.863784512Z","level":"INFO","msg":"Sanity: config values","test":"sanity","cluster_owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","account_public_key":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","operator_ids":[304,305,306,307]}
{"time":"2026-05-15T08:13:52.863932593Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2026-05-15T08:13:52.89840708Z","level":"INFO","msg":"Starting cluster cleanup replay before running tests","test":"sanity","events_count":2,"network":"hoodi_stage","test_type":"sanity"}
{"time":"2026-05-15T08:13:52.898458242Z","level":"INFO","msg":"🔁 Replaying cleanup event","test":"sanity","event_id":"a309abf4e36024ee717adff105700422","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","cluster":[304,305,306,386],"pubkeys":["0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754"],"desired_action":"remove_validator","test_type":"sanity"}
{"time":"2026-05-15T08:13:57.29263856Z","level":"INFO","msg":"Trying to remove validator...","test":"sanity","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","validator_pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","nodes":[304,305,306,307]}
{"time":"2026-05-15T08:13:57.292760991Z","level":"ERROR","msg":"❌ FAILURE: Validator Removal Failed","test":"sanity","txHash":""}
{"time":"2026-05-15T08:13:57.292781486Z","level":"WARN","msg":"remove validator retry failed","test":"sanity","attempt":1,"max_attempts":3,"error":"failed to read used directory: open /app/shared/data/keyshares/: no such file or directory"}
{"time":"2026-05-15T08:14:03.672329161Z","level":"INFO","msg":"Trying to remove validator...","test":"sanity","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","validator_pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","nodes":[304,305,306,307]}
{"time":"2026-05-15T08:14:03.672394555Z","level":"ERROR","msg":"❌ FAILURE: Validator Removal Failed","test":"sanity","txHash":""}
{"time":"2026-05-15T08:14:03.672407445Z","level":"WARN","msg":"remove validator retry failed","test":"sanity","attempt":2,"max_attempts":3,"error":"failed to read used directory: open /app/shared/data/keyshares/: no such file or directory"}
{"time":"2026-05-15T08:14:12.084994589Z","level":"INFO","msg":"Trying to remove validator...","test":"sanity","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","validator_pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","nodes":[304,305,306,307]}
{"time":"2026-05-15T08:14:12.085052854Z","level":"ERROR","msg":"❌ FAILURE: Validator Removal Failed","test":"sanity","txHash":""}
{"time":"2026-05-15T08:14:12.085075665Z","level":"WARN","msg":"remove validator retry failed","test":"sanity","attempt":3,"max_attempts":3,"error":"failed to read used directory: open /app/shared/data/keyshares/: no such file or directory"}
{"time":"2026-05-15T08:14:12.085091889Z","level":"WARN","msg":"cleanup replay event still unresolved","test":"sanity","event_id":"a309abf4e36024ee717adff105700422","error":"remove validator retries exhausted: failed to read used directory: open /app/shared/data/keyshares/: no such file or directory"}
{"time":"2026-05-15T08:14:12.085098738Z","level":"INFO","msg":"🧹 Cluster cleanup replay finished","test":"sanity","test_type":"sanity","replayed_count":0}
{"time":"2026-05-15T08:14:12.115134679Z","level":"INFO","msg":"Available slots","test":"sanity","node":304,"slots":3000}
{"time":"2026-05-15T08:14:12.120713473Z","level":"INFO","msg":"Available slots","test":"sanity","node":305,"slots":1011}
{"time":"2026-05-15T08:14:12.126430498Z","level":"INFO","msg":"Available slots","test":"sanity","node":306,"slots":3000}
{"time":"2026-05-15T08:14:12.131615957Z","level":"INFO","msg":"Available slots","test":"sanity","node":307,"slots":1325}
{"time":"2026-05-15T08:14:50.614654598Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-15T08:14:50.614697032Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-15T08:14:50.618574541Z","level":"WARN","msg":"⚠️  Sanity preflight found pre-registered validator - attempting cleanup before starting the flow","test":"sanity","validator_pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27"}
{"time":"2026-05-15T08:14:55.042011114Z","level":"INFO","msg":"Trying to remove validator...","test":"sanity","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","validator_pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","nodes":[304,305,306,307]}
{"time":"2026-05-15T08:14:55.05716493Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"removeValidator","estimated_gas":144494,"gas_buffer_pct":1000,"gas_limit":1589434}
{"time":"2026-05-15T08:14:55.071266725Z","level":"INFO","msg":"4️⃣ Removing validator","test":"sanity","module":"contract","tx_hash":"0x6d95fb5fcebb84a0a0537e5155a1abcbec46371e36f39fb2cb4508a72d4258c8"}
{"time":"2026-05-15T08:14:55.07517779Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x6d95fb5fcebb84a0a0537e5155a1abcbec46371e36f39fb2cb4508a72d4258c8"}
{"time":"2026-05-15T08:15:05.087701489Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x6d95fb5fcebb84a0a0537e5155a1abcbec46371e36f39fb2cb4508a72d4258c8"}
{"time":"2026-05-15T08:15:05.087766739Z","level":"INFO","msg":"Sanity preflight removed pre-registered validator - continuing with the sanity flow","test":"sanity","validator_pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754"}
{"time":"2026-05-15T08:16:29.935821653Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-15T08:17:04.184332192Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x86d6aa42d120173270727af286965c371f919f423558997f914b30b20bba45b041561650b531d386ed85f65828e0bde1.json"}
{"time":"2026-05-15T08:17:04.193130419Z","level":"INFO","msg":"Sanity preflight: switched to a different validator after cleanup","test":"sanity","previous_validator_pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","new_validator_pubkey":"0x86d6aa42d120173270727af286965c371f919f423558997f914b30b20bba45b041561650b531d386ed85f65828e0bde1"}
{"time":"2026-05-15T08:17:08.469846001Z","level":"INFO","msg":"Cluster asset type before write","test":"sanity","asset_type":1,"asset_err":null,"asset_label":"ETH"}
{"time":"2026-05-15T08:17:12.848570545Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"registerValidator","estimated_gas":246425,"gas_buffer_pct":1000,"gas_limit":2710675}
{"time":"2026-05-15T08:17:12.865235247Z","level":"INFO","msg":"1️⃣ Registering validator","test":"sanity","module":"contract","tx_hash":"0x604c031550c4256e5d361798c324385e156ed7d58c15ece91ac72bfaf68b1218"}
{"time":"2026-05-15T08:17:12.876324617Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x604c031550c4256e5d361798c324385e156ed7d58c15ece91ac72bfaf68b1218"}
{"time":"2026-05-15T08:17:22.880217993Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x604c031550c4256e5d361798c324385e156ed7d58c15ece91ac72bfaf68b1218"}
{"time":"2026-05-15T08:17:42.889412795Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x604c031550c4256e5d361798c324385e156ed7d58c15ece91ac72bfaf68b1218"}
{"time":"2026-05-15T08:17:42.889511204Z","level":"INFO","msg":"📜 CONTRACT: Contract Register Successful","test":"sanity","txHash":"0x604c031550c4256e5d361798c324385e156ed7d58c15ece91ac72bfaf68b1218","block_hash":"0x67b0c9032c39e93a14533a2c1ed98bbd982421fe7db6b640473697dde81a869e"}
{"time":"2026-05-15T08:17:42.897734184Z","level":"INFO","msg":"Exporter: validator not indexed yet (or index=0), retrying until non-zero index appears or max attempts is reached","test":"sanity","module":"helpers","pubkey":"0x86d6aa42d120173270727af286965c371f919f423558997f914b30b20bba45b041561650b531d386ed85f65828e0bde1","delay_seconds":15,"max_attempts":20}
{"time":"2026-05-15T08:19:12.953159661Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"sanity","module":"helpers","pubkey":"0x86d6aa42d120173270727af286965c371f919f423558997f914b30b20bba45b041561650b531d386ed85f65828e0bde1"}
{"time":"2026-05-15T08:19:12.964875592Z","level":"INFO","msg":"E2M registration: polling until assigned/executed match validator count","module":"main","module":"e2m-validator","validation_type":"registration","registered_epoch":95363,"indices":"1092007","poll_interval_seconds":15,"fail_if_current_epoch_gt":95367}
{"time":"2026-05-15T08:19:13.200472245Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorAdded"}
{"time":"2026-05-15T08:19:13.200571665Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"sanity","indices":["1092007"],"validators":1,"validate-type":"registration"}
{"time":"2026-05-15T08:21:13.289444534Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1092007"],"registered_epoch":95363,"current_epoch":95363,"assigned":0,"executed":0,"missed":0}
{"time":"2026-05-15T08:23:13.586237663Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1092007"],"registered_epoch":95363,"current_epoch":95363,"assigned":0,"executed":0,"missed":0}
{"time":"2026-05-15T08:25:13.86565515Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1092007"],"registered_epoch":95363,"current_epoch":95363,"assigned":0,"executed":0,"missed":0}
{"time":"2026-05-15T08:26:14.032681773Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1092007"],"registered_epoch":95363,"validated_epoch":95364,"epochs_waited":1}
{"time":"2026-05-15T08:26:14.032742818Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1092007"],"validate-type":"registration"}
{"time":"2026-05-15T08:26:14.032794441Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_indices":["1092007"],"registered_epoch":95363,"max_epoch_offset":8}
{"time":"2026-05-15T08:26:14.042760011Z","level":"INFO","msg":"Consensus: Loki committee/aggregate lines restricted to logs at or after registration","module":"main","module":"loki-validator","registration_block_hash":"0x67b0c9032c39e93a14533a2c1ed98bbd982421fe7db6b640473697dde81a869e","not_before_utc":"2026-05-15T08:17:24Z"}
{"time":"2026-05-15T08:26:34.062212358Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"95363","validator_indices":["1092007"]}
{"time":"2026-05-15T08:26:34.068559102Z","level":"WARN","msg":"No attesters found in duties response for one or more validators","module":"main","module":"loki-validator","epoch_offset":0,"epoch":"95363"}
{"time":"2026-05-15T08:26:44.068901245Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":1,"current_epoch":"95364","validator_indices":["1092007"]}
{"time":"2026-05-15T08:26:44.077476442Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95364-s3051653","final_epoch":"95364","slot":"3051653","validator_index":"1092007"}
{"time":"2026-05-15T08:26:44.218868166Z","level":"INFO","msg":"Consensus: QBFT start logs","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95364-s3051653","qbft_start_log_count":4,"canonical_round":1,"canonical_height":3051653,"canonical_leader":"305","container":"","canonical_log_time":"2026-05-15T08:20:37.047359Z"}
{"time":"2026-05-15T08:26:44.218925343Z","level":"INFO","msg":"Consensus: leader matches QBFT round-robin (height, round, eth epoch term)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95364-s3051653","round":1,"height":3051653,"qbft_eth_epoch":95364,"leader":305}
{"time":"2026-05-15T08:26:44.218935875Z","level":"INFO","msg":"Consensus leader validation passed","module":"main","module":"loki-validator","round":1,"leader":305}
{"time":"2026-05-15T08:26:44.311560674Z","level":"WARN","msg":"Consensus: aggregate duty in Loki but no timing metrics yet - retrying with backoff...)","module":"main","module":"loki-validator","aggregate_duty_ids":["AGGREGATOR_COMMITTEE-304_305_306_307-e95364-s3051653","AGGREGATOR-e95364-s3051653-v1092007"],"max_attempts":12,"retry_interval_seconds":20}
{"time":"2026-05-15T08:30:24.847083141Z","level":"WARN","msg":"Consensus timing skipped: no aggregate submission log lines found for aggregate duty id after retries","module":"main","module":"loki-validator","aggregate_duty_ids":["AGGREGATOR_COMMITTEE-304_305_306_307-e95364-s3051653","AGGREGATOR-e95364-s3051653-v1092007"],"last_aggregate_duty_id":"AGGREGATOR-e95364-s3051653-v1092007","attempts":12,"loki_lines":32,"aggregate_submission_logs":0}
{"time":"2026-05-15T08:30:24.847172236Z","level":"WARN","msg":"Consensus timing skipped: aggregate metrics missing after retries","module":"main","module":"loki-validator","committee_duty_id":"COMMITTEE-304_305_306_307-e95364-s3051653","aggregate_duty_ids":["AGGREGATOR_COMMITTEE-304_305_306_307-e95364-s3051653","AGGREGATOR-e95364-s3051653-v1092007"]}
{"time":"2026-05-15T08:30:24.8472216Z","level":"INFO","msg":"Consensus validation completed successfully","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95364-s3051653","final_epoch":"95364"}
{"time":"2026-05-15T08:30:24.847246564Z","level":"INFO","msg":"Consensus timing validation successful","test":"sanity","indices":["1092007"],"epoch":95363}
{"time":"2026-05-15T08:30:24.847288703Z","level":"INFO","msg":"✅ SUCCESS: Registration Validation Success","test":"sanity","validate_type":"registration","e2m_enabled":true,"e2m_success":true,"loki_success":true}
{"time":"2026-05-15T08:30:29.328753138Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"liquidate","estimated_gas":148924,"gas_buffer_pct":1000,"gas_limit":1638164}
{"time":"2026-05-15T08:30:29.353850548Z","level":"INFO","msg":"2️⃣ Liquidating cluster","test":"sanity","module":"contract","tx_hash":"0xa5fb1ca11bc5a3201f14abd8309211294289cc0721bf80855854e92f5a9a4f5b"}
{"time":"2026-05-15T08:30:29.362792855Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xa5fb1ca11bc5a3201f14abd8309211294289cc0721bf80855854e92f5a9a4f5b"}
{"time":"2026-05-15T08:30:39.371058432Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xa5fb1ca11bc5a3201f14abd8309211294289cc0721bf80855854e92f5a9a4f5b"}
{"time":"2026-05-15T08:30:59.375183663Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xa5fb1ca11bc5a3201f14abd8309211294289cc0721bf80855854e92f5a9a4f5b"}
{"time":"2026-05-15T08:30:59.375243739Z","level":"INFO","msg":"📜 CONTRACT: Cluster Liquidation Successful","test":"sanity","txHash":"0xa5fb1ca11bc5a3201f14abd8309211294289cc0721bf80855854e92f5a9a4f5b","block_hash":"0x93cdc34ff23965a9ca7fc9233c4896553e01cec8c2fe3f034ced41196c28e732"}
{"time":"2026-05-15T08:30:59.389134058Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2026-05-15T08:30:59.422394255Z","level":"INFO","msg":"Loki: no matching logs yet, retrying until data appears or max attempts is reached","module":"main","module":"loki-validator","attempts":1,"retry_delay":"20s","start_unix":1778823059,"end_unix":1778833859}
{"time":"2026-05-15T08:31:19.402428107Z","level":"INFO","msg":"Liquidation tx epoch: polling E2M until validators are all missed or have zero assigned duties","module":"main","module":"e2m-validator","validation_type":"liquidation","epoch":95365,"indices":"1092007","poll_interval_seconds":15,"fail_if_current_epoch_gte":95367,"fail_after_epochs_past_liquidation_tx":2}
{"time":"2026-05-15T08:31:34.441132767Z","level":"INFO","msg":"Waiting for E2M current epoch to reach liquidation tx epoch before checking liquidation stats","module":"main","module":"e2m-validator","epoch":95364,"indices":["1092007"],"liquidation_tx_epoch":95365}
{"time":"2026-05-15T08:32:34.597877245Z","level":"INFO","msg":"Validator still active after liquidation wait","module":"main","module":"e2m-validator","epoch":95365,"indices":["1092007"],"assigned":1,"executed":1,"missed":0}
{"time":"2026-05-15T08:32:39.594391397Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterLiquidated"}
{"time":"2026-05-15T08:32:39.59447176Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"sanity","indices":["1092007"],"validators":1,"validate-type":"liquidation"}
{"time":"2026-05-15T08:32:39.594481699Z","level":"INFO","msg":"E2M validation still in progress: waiting to confirm whether validators are no longer active on-chain duties (polling each 15s, next log may take several minutes)","test":"sanity","indices":["1092007"],"validators":1,"validate-type":"liquidation"}
{"time":"2026-05-15T08:34:34.899531873Z","level":"INFO","msg":"Waiting for liquidation criteria (all missed or zero assigned; same epoch, still polling E2M)","module":"main","module":"e2m-validator","epoch":95365,"indices":["1092007"],"liquidated_epoch":95365,"assigned":1,"executed":1,"missed":0}
{"time":"2026-05-15T08:36:35.184134453Z","level":"INFO","msg":"Waiting for liquidation criteria (all missed or zero assigned; same epoch, still polling E2M)","module":"main","module":"e2m-validator","epoch":95365,"indices":["1092007"],"liquidated_epoch":95365,"assigned":1,"executed":1,"missed":0}
{"time":"2026-05-15T08:38:35.449816214Z","level":"INFO","msg":"Waiting for liquidation criteria (all missed or zero assigned; same epoch, still polling E2M)","module":"main","module":"e2m-validator","epoch":95365,"indices":["1092007"],"liquidated_epoch":95365,"assigned":1,"executed":1,"missed":0}
{"time":"2026-05-15T08:38:50.490302411Z","level":"INFO","msg":"E2M liquidation validation successful - validators are liquidated","module":"main","module":"e2m-validator","validation_type":"liquidation","indices":["1092007"],"liquidated_epoch":95365,"validated_epoch":95366,"epochs_waited":1,"assigned":0,"missed":0}
{"time":"2026-05-15T08:38:50.490363769Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1092007"],"validate-type":"liquidation"}
{"time":"2026-05-15T08:38:50.490397861Z","level":"INFO","msg":"✅ SUCCESS: Liquidation Validation Success","test":"sanity","validate_type":"liquidation","e2m_enabled":true,"e2m_success":true,"loki_success":true}
{"time":"2026-05-15T08:38:54.874022625Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"reactivate","estimated_gas":140820,"gas_buffer_pct":1000,"gas_limit":1549020}
{"time":"2026-05-15T08:38:54.894261122Z","level":"INFO","msg":"3️⃣ Reactivating cluster","test":"sanity","module":"contract","tx_hash":"0xbc8fd48ead5dd74eb635e6cb7a31395384f9c3a38c394be3b685c5ce87b57e16"}
{"time":"2026-05-15T08:38:54.905192494Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xbc8fd48ead5dd74eb635e6cb7a31395384f9c3a38c394be3b685c5ce87b57e16"}
{"time":"2026-05-15T08:39:04.918028349Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xbc8fd48ead5dd74eb635e6cb7a31395384f9c3a38c394be3b685c5ce87b57e16"}
{"time":"2026-05-15T08:39:24.922847801Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xbc8fd48ead5dd74eb635e6cb7a31395384f9c3a38c394be3b685c5ce87b57e16"}
{"time":"2026-05-15T08:39:24.922907595Z","level":"INFO","msg":"📜 CONTRACT: Cluster Reactivation Successful","test":"sanity","txHash":"0xbc8fd48ead5dd74eb635e6cb7a31395384f9c3a38c394be3b685c5ce87b57e16","block_hash":"0x4480e4e95415ea961695e44f2f71c70dd4d7ed0ad7e21ff5f73d4182a6547540"}
{"time":"2026-05-15T08:39:24.922961683Z","level":"INFO","msg":"Validating reactivation - checking that validators resume attesting","module":"main","module":"e2m-validator","indices":["1092007"],"block_hash":"0x4480e4e95415ea961695e44f2f71c70dd4d7ed0ad7e21ff5f73d4182a6547540"}
{"time":"2026-05-15T08:39:24.930649929Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2026-05-15T08:39:24.981172973Z","level":"INFO","msg":"Loki: no matching logs yet, retrying until data appears or max attempts is reached","module":"main","module":"loki-validator","attempts":1,"retry_delay":"20s","start_unix":1778823564,"end_unix":1778834364}
{"time":"2026-05-15T08:39:44.94450821Z","level":"INFO","msg":"E2M registration: polling until assigned/executed match validator count","module":"main","module":"e2m-validator","validation_type":"registration","registered_epoch":95367,"indices":"1092007","poll_interval_seconds":15,"fail_if_current_epoch_gt":95371}
{"time":"2026-05-15T08:41:05.190226109Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterReactivated"}
{"time":"2026-05-15T08:41:05.190292085Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"sanity","indices":["1092007"],"validators":1,"validate-type":"reactivation"}
{"time":"2026-05-15T08:41:45.299633509Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1092007"],"registered_epoch":95367,"current_epoch":95366,"assigned":0,"executed":0,"missed":0}
{"time":"2026-05-15T08:43:45.597808795Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1092007"],"registered_epoch":95367,"current_epoch":95366,"assigned":0,"executed":0,"missed":0}
{"time":"2026-05-15T08:45:45.902789472Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1092007"],"registered_epoch":95367,"current_epoch":95367,"assigned":0,"executed":0,"missed":0}
{"time":"2026-05-15T08:47:46.202543685Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1092007"],"registered_epoch":95367,"current_epoch":95367,"assigned":0,"executed":0,"missed":0}
{"time":"2026-05-15T08:49:46.515595987Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1092007"],"registered_epoch":95367,"current_epoch":95367,"assigned":0,"executed":0,"missed":0}
{"time":"2026-05-15T08:51:46.787547626Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1092007"],"registered_epoch":95367,"current_epoch":95367,"assigned":0,"executed":0,"missed":0}
{"time":"2026-05-15T08:53:01.991067964Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1092007"],"registered_epoch":95367,"validated_epoch":95368,"epochs_waited":1}
{"time":"2026-05-15T08:53:01.991122217Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1092007"],"validate-type":"reactivation"}
{"time":"2026-05-15T08:53:01.991176197Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_indices":["1092007"],"registered_epoch":95367,"max_epoch_offset":8}
{"time":"2026-05-15T08:53:02.001595927Z","level":"INFO","msg":"Consensus: Loki committee/aggregate lines restricted to logs at or after registration","module":"main","module":"loki-validator","registration_block_hash":"0x4480e4e95415ea961695e44f2f71c70dd4d7ed0ad7e21ff5f73d4182a6547540","not_before_utc":"2026-05-15T08:39:12Z"}
{"time":"2026-05-15T08:53:22.002754711Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"95367","validator_indices":["1092007"]}
{"time":"2026-05-15T08:53:22.006527164Z","level":"WARN","msg":"No attesters found in duties response for one or more validators","module":"main","module":"loki-validator","epoch_offset":0,"epoch":"95367"}
{"time":"2026-05-15T08:53:32.015864198Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":1,"current_epoch":"95368","validator_indices":["1092007"]}
{"time":"2026-05-15T08:53:32.025239868Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95368-s3051807","final_epoch":"95368","slot":"3051807","validator_index":"1092007"}
{"time":"2026-05-15T08:53:32.133239631Z","level":"INFO","msg":"Consensus: QBFT start logs","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95368-s3051807","qbft_start_log_count":4,"canonical_round":1,"canonical_height":3051807,"canonical_leader":"307","container":"","canonical_log_time":"2026-05-15T08:51:26.24057Z"}
{"time":"2026-05-15T08:53:32.133285892Z","level":"INFO","msg":"Consensus: leader matches QBFT round-robin (height, round, eth epoch term)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95368-s3051807","round":1,"height":3051807,"qbft_eth_epoch":95368,"leader":307}
{"time":"2026-05-15T08:53:32.133298828Z","level":"INFO","msg":"Consensus leader validation passed","module":"main","module":"loki-validator","round":1,"leader":307}
{"time":"2026-05-15T08:53:32.210312356Z","level":"WARN","msg":"Consensus: aggregate duty in Loki but no timing metrics yet - retrying with backoff...)","module":"main","module":"loki-validator","aggregate_duty_ids":["AGGREGATOR_COMMITTEE-304_305_306_307-e95368-s3051807","AGGREGATOR-e95368-s3051807-v1092007"],"max_attempts":12,"retry_interval_seconds":20}
{"time":"2026-05-15T08:57:12.764547311Z","level":"WARN","msg":"Consensus timing skipped: no aggregate submission log lines found for aggregate duty id after retries","module":"main","module":"loki-validator","aggregate_duty_ids":["AGGREGATOR_COMMITTEE-304_305_306_307-e95368-s3051807","AGGREGATOR-e95368-s3051807-v1092007"],"last_aggregate_duty_id":"AGGREGATOR-e95368-s3051807-v1092007","attempts":12,"loki_lines":32,"aggregate_submission_logs":0}
{"time":"2026-05-15T08:57:12.764637352Z","level":"WARN","msg":"Consensus timing skipped: aggregate metrics missing after retries","module":"main","module":"loki-validator","committee_duty_id":"COMMITTEE-304_305_306_307-e95368-s3051807","aggregate_duty_ids":["AGGREGATOR_COMMITTEE-304_305_306_307-e95368-s3051807","AGGREGATOR-e95368-s3051807-v1092007"]}
{"time":"2026-05-15T08:57:12.764677169Z","level":"INFO","msg":"Consensus validation completed successfully","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95368-s3051807","final_epoch":"95368"}
{"time":"2026-05-15T08:57:12.764688544Z","level":"INFO","msg":"Consensus timing validation successful","test":"sanity","indices":["1092007"],"epoch":95367}
{"time":"2026-05-15T08:57:12.764711639Z","level":"INFO","msg":"✅ SUCCESS: Reactivation Validation Success","test":"sanity","validate_type":"reactivation","e2m_enabled":true,"e2m_success":true,"loki_success":true}
{"time":"2026-05-15T08:57:17.297079872Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"removeValidator","estimated_gas":144346,"gas_buffer_pct":1000,"gas_limit":1587806}
{"time":"2026-05-15T08:57:17.32450019Z","level":"INFO","msg":"4️⃣ Removing validator","test":"sanity","module":"contract","tx_hash":"0xabc765fc8b00a64ebd5d36a2523e651ecce45eb915a3744ada24992b1136d226"}
{"time":"2026-05-15T08:57:17.332768249Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xabc765fc8b00a64ebd5d36a2523e651ecce45eb915a3744ada24992b1136d226"}
{"time":"2026-05-15T08:57:27.338287243Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xabc765fc8b00a64ebd5d36a2523e651ecce45eb915a3744ada24992b1136d226"}
{"time":"2026-05-15T08:57:27.338333976Z","level":"INFO","msg":"📜 CONTRACT: Contract Removal Successful","test":"sanity","txHash":"0xabc765fc8b00a64ebd5d36a2523e651ecce45eb915a3744ada24992b1136d226","block_hash":"0xc48a2436374ffeeee0100b6418cb5a4eaea073c812a8cab49b201ec3eabdb16e"}
{"time":"2026-05-15T08:57:27.35146286Z","level":"INFO","msg":"waiting for block to be found","module":"main","module":"e2m-validator","module":"e2m"}
{"time":"2026-05-15T08:57:27.395731759Z","level":"INFO","msg":"Loki: no matching logs yet, retrying until data appears or max attempts is reached","module":"main","module":"loki-validator","attempts":1,"retry_delay":"20s","start_unix":1778824647,"end_unix":1778835447}
{"time":"2026-05-15T08:57:47.357756981Z","level":"INFO","msg":"Removal tx epoch: polling E2M until attestations.Assigned is 0 for these validators (on-chain removed, indexer may lag)","module":"main","module":"e2m-validator","validation_type":"removal","epoch":95369,"indices":"1092007","poll_interval_seconds":15,"fail_if_current_epoch_gte":95371,"fail_after_epochs_past_removal_tx":2}
{"time":"2026-05-15T08:58:02.411545762Z","level":"INFO","msg":"E2M removal check: validators still have duties in E2M, waiting for 'assigned' to drop to zero","module":"main","module":"e2m-validator","validation_type":"removal","epoch":95369,"indices":["1092007"],"assigned":1,"executed":1,"missed":0}
{"time":"2026-05-15T08:59:07.558899339Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorRemoved"}
{"time":"2026-05-15T08:59:07.558991593Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"sanity","indices":["1092007"],"validators":1,"validate-type":"removal"}
{"time":"2026-05-15T08:59:07.559008507Z","level":"INFO","msg":"E2M validation still in progress: waiting to confirm whether validators are no longer active on-chain duties (polling each 15s, next log may take several minutes)","test":"sanity","indices":["1092007"],"validators":1,"validate-type":"removal"}
{"time":"2026-05-15T09:00:02.705111292Z","level":"INFO","msg":"E2M removal check: same epoch, still polling until 'assigned' is zero (validators should stop, not attest)","module":"main","module":"e2m-validator","validation_type":"removal","epoch":95369,"indices":["1092007"],"assigned":1,"executed":1,"missed":0}
{"time":"2026-05-15T09:02:02.990359956Z","level":"INFO","msg":"E2M removal check: same epoch, still polling until 'assigned' is zero (validators should stop, not attest)","module":"main","module":"e2m-validator","validation_type":"removal","epoch":95369,"indices":["1092007"],"assigned":1,"executed":1,"missed":0}
{"time":"2026-05-15T09:04:03.383712948Z","level":"INFO","msg":"E2M removal check: same epoch, still polling until 'assigned' is zero (validators should stop, not attest)","module":"main","module":"e2m-validator","validation_type":"removal","epoch":95369,"indices":["1092007"],"assigned":1,"executed":1,"missed":0}
{"time":"2026-05-15T09:04:33.44790294Z","level":"INFO","msg":"E2M removal check: validators still have duties in E2M, waiting for 'assigned' to drop to zero","module":"main","module":"e2m-validator","validation_type":"removal","epoch":95370,"indices":["1092007"],"assigned":1,"executed":1,"missed":0}
{"time":"2026-05-15T09:06:33.728370134Z","level":"INFO","msg":"E2M removal check: same epoch, still polling until 'assigned' is zero (validators should stop, not attest)","module":"main","module":"e2m-validator","validation_type":"removal","epoch":95370,"indices":["1092007"],"assigned":1,"executed":1,"missed":0}
{"time":"2026-05-15T09:08:33.979304341Z","level":"INFO","msg":"E2M removal check: same epoch, still polling until 'assigned' is zero (validators should stop, not attest)","module":"main","module":"e2m-validator","validation_type":"removal","epoch":95370,"indices":["1092007"],"assigned":1,"executed":1,"missed":0}
{"time":"2026-05-15T09:10:34.214458321Z","level":"INFO","msg":"E2M removal check: same epoch, still polling until 'assigned' is zero (validators should stop, not attest)","module":"main","module":"e2m-validator","validation_type":"removal","epoch":95370,"indices":["1092007"],"assigned":1,"executed":1,"missed":0}
{"time":"2026-05-15T09:11:04.307556975Z","level":"INFO","msg":"E2M removal validation successful - validators stopped attesting","module":"main","module":"e2m-validator","validation_type":"removal","indices":["1092007"],"removed_epoch":95369,"validated_epoch":95371,"epochs_waited":2}
{"time":"2026-05-15T09:11:04.307688561Z","level":"INFO","msg":"E2M validation successful","test":"sanity","indices":["1092007"],"validate-type":"removal"}
{"time":"2026-05-15T09:11:04.307719766Z","level":"INFO","msg":"✅ SUCCESS: Removal Validation Success","test":"sanity","validate_type":"removal","e2m_enabled":true,"e2m_success":true,"loki_success":true}
{"time":"2026-05-15T09:11:04.319329378Z","level":"INFO","msg":"✅ SUCCESS: Sanity Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":true,"Loki":true},"liquidate":{"Contract":true,"E2M":true,"Loki":true},"reactivate":{"Contract":true,"E2M":true,"Loki":true},"remove":{"Contract":true,"E2M":true,"Loki":true}}
{"time":"2026-05-15T09:11:04.319371269Z","level":"INFO","msg":"🔑 KEY STEP: Starting Bulk Test Suite","module":"main"}
{"time":"2026-05-15T09:11:04.319387965Z","level":"INFO","msg":"Bulk: config values","test":"bulk","cluster_owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","account_public_key":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","operator_ids":[304,305,306,307]}
{"time":"2026-05-15T09:11:04.348610223Z","level":"INFO","msg":"Starting cluster cleanup replay before running tests","test":"bulk","events_count":2,"network":"hoodi_stage","test_type":"bulk"}
{"time":"2026-05-15T09:11:04.348749663Z","level":"INFO","msg":"🔁 Replaying cleanup event","test":"bulk","event_id":"ca18a92a3738667dbe232341adacb38e","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","cluster":[304,305,306,386],"pubkeys":["0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","0x874e9b0b577bb6c9dbd7bad54e19306422288cc9d9cced8522fbb3990a55c0c82586a9b0c8eca964aa38306179b42856","0xa3a4b31b4ab322e432cec4719f3795d46e707f82f3e4a0930550edf09643712e61e947eea441ee037bb947868e9b0898"],"desired_action":"bulk_remove","test_type":"bulk"}
{"time":"2026-05-15T09:11:08.888232972Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"bulkRemoveValidator","gas_estimate_error":"gas estimation failed: execution reverted","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-15T09:11:08.903155691Z","level":"INFO","msg":"4️⃣ Bulk Removing validators","test":"bulk","module":"contract","tx_hash":"0xc7061cc20cbb29842ccf2cec9ca9f6425f22a184b93a9e4300188d0caca2a693"}
{"time":"2026-05-15T09:11:08.90855516Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xc7061cc20cbb29842ccf2cec9ca9f6425f22a184b93a9e4300188d0caca2a693"}
{"time":"2026-05-15T09:11:18.917860592Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0xc7061cc20cbb29842ccf2cec9ca9f6425f22a184b93a9e4300188d0caca2a693"}
{"time":"2026-05-15T09:11:18.917920143Z","level":"WARN","msg":"bulk remove retry failed","test":"bulk","attempt":1,"max_attempts":3,"error":"transaction verification failed","pubkeys_count":3}
{"time":"2026-05-15T09:11:25.353632374Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"bulkRemoveValidator","gas_estimate_error":"gas estimation failed: execution reverted","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-15T09:11:25.367201911Z","level":"INFO","msg":"4️⃣ Bulk Removing validators","test":"bulk","module":"contract","tx_hash":"0x4fd0ecee5e282466b8772b7546ac1fcc3ff3a55faf86578b1cc1e95085eb2c6f"}
{"time":"2026-05-15T09:11:25.372187165Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x4fd0ecee5e282466b8772b7546ac1fcc3ff3a55faf86578b1cc1e95085eb2c6f"}
{"time":"2026-05-15T09:11:35.377000351Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x4fd0ecee5e282466b8772b7546ac1fcc3ff3a55faf86578b1cc1e95085eb2c6f"}
{"time":"2026-05-15T09:11:55.386653569Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x4fd0ecee5e282466b8772b7546ac1fcc3ff3a55faf86578b1cc1e95085eb2c6f"}
{"time":"2026-05-15T09:11:55.386711943Z","level":"WARN","msg":"bulk remove retry failed","test":"bulk","attempt":2,"max_attempts":3,"error":"transaction verification failed","pubkeys_count":3}
{"time":"2026-05-15T09:12:03.893832452Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"bulkRemoveValidator","gas_estimate_error":"gas estimation failed: execution reverted","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-15T09:12:03.912511518Z","level":"INFO","msg":"4️⃣ Bulk Removing validators","test":"bulk","module":"contract","tx_hash":"0xfb6f2e6d781bfa44099b2f11a9c03d3297f149928930a02a2c1b99e1dd477a97"}
{"time":"2026-05-15T09:12:03.919284418Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xfb6f2e6d781bfa44099b2f11a9c03d3297f149928930a02a2c1b99e1dd477a97"}
{"time":"2026-05-15T09:12:13.929098626Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xfb6f2e6d781bfa44099b2f11a9c03d3297f149928930a02a2c1b99e1dd477a97"}
{"time":"2026-05-15T09:12:33.94662998Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0xfb6f2e6d781bfa44099b2f11a9c03d3297f149928930a02a2c1b99e1dd477a97"}
{"time":"2026-05-15T09:12:33.946689873Z","level":"WARN","msg":"bulk remove retry failed","test":"bulk","attempt":3,"max_attempts":3,"error":"transaction verification failed","pubkeys_count":3}
{"time":"2026-05-15T09:12:33.946705542Z","level":"WARN","msg":"cleanup replay event still unresolved","test":"bulk","event_id":"ca18a92a3738667dbe232341adacb38e","error":"bulk remove retries exhausted: transaction verification failed"}
{"time":"2026-05-15T09:12:33.946712377Z","level":"INFO","msg":"🧹 Cluster cleanup replay finished","test":"bulk","test_type":"bulk","replayed_count":0}
{"time":"2026-05-15T09:12:33.966196816Z","level":"INFO","msg":"Available slots","test":"bulk","node":304,"slots":3000}
{"time":"2026-05-15T09:12:33.976253298Z","level":"INFO","msg":"Available slots","test":"bulk","node":305,"slots":1011}
{"time":"2026-05-15T09:12:33.983230646Z","level":"INFO","msg":"Available slots","test":"bulk","node":306,"slots":3000}
{"time":"2026-05-15T09:12:33.990226094Z","level":"INFO","msg":"Available slots","test":"bulk","node":307,"slots":1325}
{"time":"2026-05-15T09:13:17.089291306Z","level":"INFO","msg":"Extracted keyshares path for bulk keyshare","test":"bulk","module":"keys","path":"/app/shared/data/keyshares/keyshares-1778836397.json"}
{"time":"2026-05-15T09:13:21.284128782Z","level":"INFO","msg":"Cluster asset type before write","test":"bulk","asset_type":1,"asset_err":null,"asset_label":"ETH"}
{"time":"2026-05-15T09:13:25.580070059Z","level":"WARN","msg":"⚠️  Bulk preflight found pre-registered validators - attempting cleanup before starting","test":"bulk","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","cluster":[304,305,306,307],"pre_registered_count":1}
{"time":"2026-05-15T09:13:30.058198326Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"bulkRemoveValidator","estimated_gas":145590,"gas_buffer_pct":1000,"gas_limit":1601490}
{"time":"2026-05-15T09:13:30.084636906Z","level":"INFO","msg":"4️⃣ Bulk Removing validators","test":"bulk","module":"contract","tx_hash":"0x1f5b85213fba93702947481205cb4a3f18e8f50fc827996d927d2d2c71ecd21f"}
{"time":"2026-05-15T09:13:30.089464296Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x1f5b85213fba93702947481205cb4a3f18e8f50fc827996d927d2d2c71ecd21f"}
{"time":"2026-05-15T09:13:40.104971504Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x1f5b85213fba93702947481205cb4a3f18e8f50fc827996d927d2d2c71ecd21f"}
{"time":"2026-05-15T09:13:40.105027877Z","level":"INFO","msg":"Bulk preflight removed pre-registered validators on-chain - continuing with bulk flow","test":"bulk","removed_count":1}
{"time":"2026-05-15T09:13:40.105256446Z","level":"WARN","msg":"Bulk preflight: failed converting rotated keystore window to keyshares, falling back to original batch","test":"bulk","offset":3,"bulk_amount":3,"error":"requested keystore window out of bounds: offset=3 amount=3 total=3"}
{"time":"2026-05-15T09:13:44.506219048Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"bulkRegisterValidator","estimated_gas":371733,"gas_buffer_pct":1000,"gas_limit":4089063}
{"time":"2026-05-15T09:13:44.521673647Z","level":"INFO","msg":"1️⃣ Bulk Registering validators","test":"bulk","module":"contract","tx_hash":"0xb4116947485b1effa64d89914c4e6c43c84037ec6dccf67ec69906b0f4927758"}
{"time":"2026-05-15T09:13:44.526863804Z","level":"INFO","msg":"Bulk registration transaction sent","test":"bulk","txHash":"0xb4116947485b1effa64d89914c4e6c43c84037ec6dccf67ec69906b0f4927758"}
{"time":"2026-05-15T09:13:44.536201948Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xb4116947485b1effa64d89914c4e6c43c84037ec6dccf67ec69906b0f4927758"}
{"time":"2026-05-15T09:13:54.544912341Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0xb4116947485b1effa64d89914c4e6c43c84037ec6dccf67ec69906b0f4927758"}
{"time":"2026-05-15T09:13:54.545055197Z","level":"INFO","msg":"📜 CONTRACT: Bulk Registration Successful","test":"bulk","blockHash":"0x8b0a4bd8c2e9ea416e64a42c6a932861cb16d41b7e63a094093aab137b067ec6","Validators Amount":3}
{"time":"2026-05-15T09:13:54.553308196Z","level":"INFO","msg":"Exporter: validator not indexed yet (or index=0), retrying until non-zero index appears or max attempts is reached","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754","delay_seconds":15,"max_attempts":20}
{"time":"2026-05-15T09:15:39.617840149Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754"}
{"time":"2026-05-15T09:15:39.622166666Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0x86d6aa42d120173270727af286965c371f919f423558997f914b30b20bba45b041561650b531d386ed85f65828e0bde1"}
{"time":"2026-05-15T09:15:39.625147276Z","level":"INFO","msg":"Successfully retrieved validator index from exporter","test":"bulk","module":"helpers","pubkey":"0xa3a4b31b4ab322e432cec4719f3795d46e707f82f3e4a0930550edf09643712e61e947eea441ee037bb947868e9b0898"}
{"time":"2026-05-15T09:15:39.636650982Z","level":"INFO","msg":"E2M registration: polling until assigned/executed match validator count","module":"main","module":"e2m-validator","validation_type":"registration","registered_epoch":95372,"indices":"1102420,1092007,1104098","poll_interval_seconds":15,"fail_if_current_epoch_gt":95376}
{"time":"2026-05-15T09:15:39.776910648Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":3,"event":"ValidatorAdded"}
{"time":"2026-05-15T09:15:39.776991648Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"bulk","indices":["1102420","1092007","1104098"],"validators":3,"validate-type":"registration"}
{"time":"2026-05-15T09:17:39.926339992Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420","1092007","1104098"],"registered_epoch":95372,"current_epoch":95372,"assigned":1,"executed":0,"missed":1}
{"time":"2026-05-15T09:19:40.237783401Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420","1092007","1104098"],"registered_epoch":95372,"current_epoch":95372,"assigned":1,"executed":0,"missed":1}
{"time":"2026-05-15T09:21:40.524278631Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420","1092007","1104098"],"registered_epoch":95372,"current_epoch":95372,"assigned":1,"executed":0,"missed":1}
{"time":"2026-05-15T09:23:40.787330154Z","level":"INFO","msg":"E2M registration check: still polling for assigned/executed to match validator count","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420","1092007","1104098"],"registered_epoch":95372,"current_epoch":95373,"assigned":3,"executed":3,"missed":0}
{"time":"2026-05-15T09:23:40.787431458Z","level":"INFO","msg":"E2M registration validation successful - validators are attesting","module":"main","module":"e2m-validator","validation_type":"registration","indices":["1102420","1092007","1104098"],"registered_epoch":95372,"validated_epoch":95373,"epochs_waited":1}
{"time":"2026-05-15T09:23:40.787458726Z","level":"INFO","msg":"E2M validation successful","test":"bulk","indices":["1102420","1092007","1104098"],"validate-type":"registration"}
{"time":"2026-05-15T09:23:40.78753554Z","level":"INFO","msg":"Starting consensus validation with epoch iteration logic","module":"main","module":"loki-validator","validator_indices":["1102420","1092007","1104098"],"registered_epoch":95372,"max_epoch_offset":8}
{"time":"2026-05-15T09:23:40.797852925Z","level":"INFO","msg":"Consensus: Loki committee/aggregate lines restricted to logs at or after registration","module":"main","module":"loki-validator","registration_block_hash":"0x8b0a4bd8c2e9ea416e64a42c6a932861cb16d41b7e63a094093aab137b067ec6","not_before_utc":"2026-05-15T09:13:48Z"}
{"time":"2026-05-15T09:24:00.799116634Z","level":"INFO","msg":"Checking epoch for duties","module":"main","module":"loki-validator","epoch_offset":0,"current_epoch":"95372","validator_indices":["1102420","1092007","1104098"]}
{"time":"2026-05-15T09:24:00.812074317Z","level":"INFO","msg":"Successfully found duties, building duty ID for consensus validation","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","final_epoch":"95372","slot":"3051929","validator_index":"1104098"}
{"time":"2026-05-15T09:24:00.966775357Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":1,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:24:10.996227763Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":2,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:24:21.025957874Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":3,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:24:31.058412007Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":4,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:24:41.080593973Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":5,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:24:51.108825097Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":6,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:25:01.145519917Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":7,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:25:11.170020663Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":8,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:25:21.204554235Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":9,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:25:31.234016656Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":10,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:25:41.257173622Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":11,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:25:51.280752237Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":12,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:26:01.306897172Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":13,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:26:11.330390643Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":14,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:26:21.351075541Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":15,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:26:31.372762763Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":16,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:26:41.404624447Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":17,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:26:51.429963667Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":18,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:27:01.463224622Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":19,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:27:11.492681661Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":20,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:27:21.515962174Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":21,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:27:31.535926524Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":22,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:27:41.564304356Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":23,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:27:51.588795898Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":24,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:28:01.613772182Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":25,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:28:11.637637418Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":26,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:28:21.663327734Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":27,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:28:31.682567856Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":28,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:28:41.704330824Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":29,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:28:51.730919683Z","level":"WARN","msg":"Consensus: committee duty in Loki but no QBFT start lines yet (will retry)","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","attempt":30,"max_attempts":30,"loki_lines":0,"qbft_start_lines":0}
{"time":"2026-05-15T09:28:51.731079368Z","level":"ERROR","msg":"Consensus failed: no post-registration QBFT start logs after retries","module":"main","module":"loki-validator","duty_id":"COMMITTEE-304_305_306_307-e95372-s3051929","not_before_utc":"2026-05-15T09:13:48Z","reason":"consensus qbft start logs missing after retries: duty_id=COMMITTEE-304_305_306_307-e95372-s3051929 attempts=30"}
{"time":"2026-05-15T09:28:51.73112419Z","level":"ERROR","msg":"Consensus timing validation failed","test":"bulk","indices":["1102420","1092007","1104098"],"epoch":95372,"error":"missing post-registration QBFT start logs for duty COMMITTEE-304_305_306_307-e95372-s3051929: consensus qbft start logs missing after retries: duty_id=COMMITTEE-304_305_306_307-e95372-s3051929 attempts=30"}
{"time":"2026-05-15T09:28:51.731164407Z","level":"ERROR","msg":"❌ FAILURE: Registration Validation Fail","test":"bulk","validate_type":"registration","e2m_enabled":true,"e2m_success":true,"loki_success":false}
{"time":"2026-05-15T09:28:51.740250889Z","level":"WARN","msg":"failed to report run progress","test":"bulk","error":"progress update failed: no active usage tracking found with ID 76b29e68939ec21d2cf391af9acd6d3f"}
{"time":"2026-05-15T09:28:51.740299431Z","level":"WARN","msg":"🧹 Starting cluster cleanup for failed bulk action","test":"bulk","stage":"bulk_register_validation","owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","nodes":[304,305,306,307],"pubkeys_count":3,"liquidated":false,"reactivated":false}
{"time":"2026-05-15T09:28:56.094806243Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"bulkRemoveValidator","estimated_gas":173261,"gas_buffer_pct":1000,"gas_limit":1905871}
{"time":"2026-05-15T09:28:56.107970308Z","level":"INFO","msg":"4️⃣ Bulk Removing validators","test":"bulk","module":"contract","tx_hash":"0x899bc5d4bc6b6c9fce9e17f3ac92bb22f489baec761963f23c492db9c3224da6"}
{"time":"2026-05-15T09:28:56.111839176Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x899bc5d4bc6b6c9fce9e17f3ac92bb22f489baec761963f23c492db9c3224da6"}
{"time":"2026-05-15T09:29:06.118290687Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x899bc5d4bc6b6c9fce9e17f3ac92bb22f489baec761963f23c492db9c3224da6"}
{"time":"2026-05-15T09:29:06.118368587Z","level":"ERROR","msg":"❌ FAILURE: Bulk Test Suite Failed","module":"main","error":"bulk register step failed: bulk register validation failed"}
{"time":"2026-05-15T09:29:06.118378996Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2026-05-15T09:29:06.118415688Z","level":"ERROR","msg":"❌ FAILURE: Test Execution Completed with Failures","module":"main","passed_suites":["sanity"],"failed_suites":["bulk"],"skipped_suites":["isolated"]}
{"time":"2026-05-15T09:29:06.118446459Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"bulk","error":"bulk register step failed: bulk register validation failed"}
{"time":"2026-05-15T09:29:06.118459044Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"76b29e68939ec21d2cf391af9acd6d3f"}
{"time":"2026-05-15T09:29:06.121990586Z","level":"INFO","msg":"Resource usage already cleared, skipping teardown attempts","module":"main","usage_id":"76b29e68939ec21d2cf391af9acd6d3f"}

2026-05-15T09:29:11Z 🟠 The test run failed. Resources are NOT deleted yet — held for 15 minutes for debugging (until 2026-05-15T09:44:10Z UTC). Please do NOT rerun this test until cleanup completes. To force immediate cleanup, comment /test cancel.

Result - 🔴 Failure

2026-05-15T09:44:15Z 🔴 Resources cleaned up — test run is finalised. You can rerun the test now.

iurii-ssv · 2026-05-18T09:51:05Z

/test

ssv-test-bot · 2026-05-18T09:51:09Z

🔴 Test run for PR #2835

Branch: qbft-correct-instance-message-retries
Commit: 82c4b6b

Status

2026-05-18T09:51:08Z Starting test run
2026-05-18T09:51:10Z Test run submitted successfully for commit 82c4b6b
2026-05-18T09:52:12Z container image for commit 82c4b6b23a1098f0964c84b4304650796388ca02 is build and present in container registry
2026-05-18T09:52:14Z automatically allocated node range cluster=300,301,302,303 for test run on network=hoodi_stage
2026-05-18T09:52:15Z deploying commit 82c4b6b23a1098f0964c84b4304650796388ca02 to ssv nodes 300,301,302,303
2026-05-18T09:53:02Z commit 82c4b6b23a1098f0964c84b4304650796388ca02 is deployed to ssv nodes 300,301,302,303
2026-05-18T09:53:03Z started test executor with tests=(sanity) and network=hoodi_stage
2026-05-18T09:54:00Z 🟠 The test run failed. Resources are NOT deleted yet — held for 15 minutes for debugging (until 2026-05-18T10:09:00Z UTC). Please do NOT rerun this test until cleanup completes. To force immediate cleanup, comment /test cancel.

Result - 🔴 Failure

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\ \/ /| __|/ __|| | | ||_   _|/ _ \ | _ \
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\_\|___|\___| \___/   |_|  \___/ |_|_\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2026-05-18T09:53:06.171923432Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2026-05-18T09:53:06.176655082Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2026-05-18T09:53:06.176675584Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[300,301,302,303],"tests":"(sanity)"}
{"time":"2026-05-18T09:53:06.225798528Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"ad8a37c1f2879aad7427c5ca05523a9d"}
{"time":"2026-05-18T09:53:06.226167508Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2026-05-18T09:53:06.226181952Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[300,301,302,303],"test_type":{"Sanity":{"ShouldRun":true,"E2M":false,"Threshold":false,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"Threshold":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2026-05-18T09:53:08.501575822Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2026-05-18T09:53:08.50165243Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2026-05-18T09:53:08.501660032Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2026-05-18T09:53:08.501673296Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":false}
{"time":"2026-05-18T09:53:08.501687906Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2026-05-18T09:53:08.50169769Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2026-05-18T09:53:08.501711338Z","level":"INFO","msg":"Sanity: config values","test":"sanity","cluster_owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","account_public_key":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","operator_ids":[300,301,302,303]}
{"time":"2026-05-18T09:53:08.50182463Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2026-05-18T09:53:08.546579665Z","level":"INFO","msg":"Available slots","test":"sanity","node":300,"slots":3000}
{"time":"2026-05-18T09:53:08.555041048Z","level":"INFO","msg":"Available slots","test":"sanity","node":301,"slots":1011}
{"time":"2026-05-18T09:53:08.562582782Z","level":"INFO","msg":"Available slots","test":"sanity","node":302,"slots":3000}
{"time":"2026-05-18T09:53:08.571462793Z","level":"INFO","msg":"Available slots","test":"sanity","node":303,"slots":1325}
{"time":"2026-05-18T09:53:37.573555728Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-18T09:53:37.573580123Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-18T09:53:41.709360398Z","level":"INFO","msg":"Cluster asset type before write","test":"sanity","asset_type":1,"asset_err":null,"asset_label":"ETH"}
{"time":"2026-05-18T09:53:45.705327259Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"registerValidator","gas_estimate_error":"gas estimation failed: execution reverted","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-18T09:53:45.721660682Z","level":"INFO","msg":"1️⃣ Registering validator","test":"sanity","module":"contract","tx_hash":"0xd164216e80fab3924a4fe7b85648af58994fedf15ca3be7bca3ad9af0e4f9b53"}
{"time":"2026-05-18T09:53:45.730077256Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xd164216e80fab3924a4fe7b85648af58994fedf15ca3be7bca3ad9af0e4f9b53"}
{"time":"2026-05-18T09:53:55.741304129Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xd164216e80fab3924a4fe7b85648af58994fedf15ca3be7bca3ad9af0e4f9b53"}
{"time":"2026-05-18T09:53:55.741343419Z","level":"ERROR","msg":"❌ FAILURE: Contract Registration Failed - Transaction verification failed","test":"sanity","txHash":"0xd164216e80fab3924a4fe7b85648af58994fedf15ca3be7bca3ad9af0e4f9b53"}
{"time":"2026-05-18T09:53:55.741371632Z","level":"ERROR","msg":"❌ FAILURE: Sanity Test Suite Failed","module":"main","error":"sanity register step failed: transaction verification failed"}
{"time":"2026-05-18T09:53:55.741375696Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2026-05-18T09:53:55.741388697Z","level":"ERROR","msg":"❌ FAILURE: Test Execution Completed with Failures","module":"main","passed_suites":null,"failed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2026-05-18T09:53:55.741397105Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"sanity","error":"sanity register step failed: transaction verification failed"}
{"time":"2026-05-18T09:53:55.741401717Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"ad8a37c1f2879aad7427c5ca05523a9d"}
{"time":"2026-05-18T09:53:55.743737952Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"ad8a37c1f2879aad7427c5ca05523a9d"}
{"time":"2026-05-18T09:53:55.759018216Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"ad8a37c1f2879aad7427c5ca05523a9d"}

2026-05-18T10:09:05Z 🔴 Resources cleaned up — test run is finalised. You can rerun the test now.

julienh-ssv

All review comments addressed, lgtm

AlexM-ssv-labs · 2026-05-19T08:58:26Z

/test cluster=304,305,306,307 tests=(sanity),(bulk)

ssv-test-bot · 2026-05-19T08:58:29Z

🟢 Test run for PR #2835

Branch: qbft-correct-instance-message-retries
Commit: 82c4b6b

Status

2026-05-19T08:58:29Z Starting test run
2026-05-19T08:58:31Z Test run submitted successfully for commit 82c4b6b
2026-05-19T08:58:31Z container image for commit 82c4b6b23a1098f0964c84b4304650796388ca02 is build and present in container registry
2026-05-19T08:58:33Z using user defined node range cluster=304,305,306,307 on network=hoodi_stage
2026-05-19T08:58:34Z deploying commit 82c4b6b23a1098f0964c84b4304650796388ca02 to ssv nodes 304,305,306,307
2026-05-19T08:59:36Z commit 82c4b6b23a1098f0964c84b4304650796388ca02 is deployed to ssv nodes 304,305,306,307
2026-05-19T08:59:37Z started test executor with tests=(sanity),(bulk) and network=hoodi_stage

Click to expand executor logs

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2026-05-19T08:59:40.715066675Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2026-05-19T08:59:40.719735708Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2026-05-19T08:59:40.719757297Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[304,305,306,307],"tests":"(sanity),(bulk)"}
{"time":"2026-05-19T08:59:40.768271106Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"bbc05c559467bb578b2048415ad3400c"}
{"time":"2026-05-19T08:59:40.768557842Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2026-05-19T08:59:40.768568911Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[304,305,306,307],"test_type":{"Sanity":{"ShouldRun":true,"E2M":false,"Threshold":false,"BulkAmount":0},"Bulk":{"ShouldRun":true,"E2M":false,"Threshold":false,"BulkAmount":3},"Isolated":{"Tests":null}}}
{"time":"2026-05-19T08:59:43.503931654Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2026-05-19T08:59:43.50400308Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2026-05-19T08:59:43.504012439Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2026-05-19T08:59:43.504027734Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":false}
{"time":"2026-05-19T08:59:43.504036853Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2026-05-19T08:59:43.504050083Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2026-05-19T08:59:43.504062987Z","level":"INFO","msg":"Sanity: config values","test":"sanity","cluster_owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","account_public_key":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","operator_ids":[304,305,306,307]}
{"time":"2026-05-19T08:59:43.504171699Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2026-05-19T08:59:43.558911564Z","level":"INFO","msg":"Available slots","test":"sanity","node":304,"slots":3000}
{"time":"2026-05-19T08:59:43.569641261Z","level":"INFO","msg":"Available slots","test":"sanity","node":305,"slots":1010}
{"time":"2026-05-19T08:59:43.577521648Z","level":"INFO","msg":"Available slots","test":"sanity","node":306,"slots":3000}
{"time":"2026-05-19T08:59:43.584024598Z","level":"INFO","msg":"Available slots","test":"sanity","node":307,"slots":1324}
{"time":"2026-05-19T09:00:54.552727053Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-19T09:00:54.553246767Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-19T09:00:58.374280087Z","level":"INFO","msg":"Cluster asset type before write","test":"sanity","asset_type":1,"asset_err":null,"asset_label":"ETH"}
{"time":"2026-05-19T09:01:02.356274501Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"registerValidator","estimated_gas":244454,"gas_buffer_pct":1000,"gas_limit":2688994}
{"time":"2026-05-19T09:01:02.371729637Z","level":"INFO","msg":"1️⃣ Registering validator","test":"sanity","module":"contract","tx_hash":"0x658347bdc31458cb966558ca453d72dccdf23469a00a75e6d665e7dcc6d4f96a"}
{"time":"2026-05-19T09:01:02.382230764Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x658347bdc31458cb966558ca453d72dccdf23469a00a75e6d665e7dcc6d4f96a"}
{"time":"2026-05-19T09:01:12.389877551Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x658347bdc31458cb966558ca453d72dccdf23469a00a75e6d665e7dcc6d4f96a"}
{"time":"2026-05-19T09:01:32.394715164Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x658347bdc31458cb966558ca453d72dccdf23469a00a75e6d665e7dcc6d4f96a"}
{"time":"2026-05-19T09:01:32.394784468Z","level":"INFO","msg":"📜 CONTRACT: Contract Register Successful","test":"sanity","txHash":"0x658347bdc31458cb966558ca453d72dccdf23469a00a75e6d665e7dcc6d4f96a","block_hash":"0xa39be59ff917a0459edc3da9a75318846d528d9fbfd26cc028436bfda5376eb1"}
{"time":"2026-05-19T09:01:32.748386868Z","level":"INFO","msg":"Loki: no matching logs yet, retrying until data appears or max attempts is reached","module":"main","module":"loki-validator","attempts":1,"retry_delay":"20s","start_unix":1779170492,"end_unix":1779181292}
{"time":"2026-05-19T09:02:52.873063093Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorAdded"}
{"time":"2026-05-19T09:02:52.873112133Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"sanity","indices":[],"validators":1,"validate-type":"registration"}
{"time":"2026-05-19T09:02:52.873134834Z","level":"INFO","msg":"✅ SUCCESS: Registration Validation Success","test":"sanity","validate_type":"registration","e2m_enabled":false,"e2m_success":false,"loki_success":true}
{"time":"2026-05-19T09:02:56.496485084Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"liquidate","estimated_gas":149318,"gas_buffer_pct":1000,"gas_limit":1642498}
{"time":"2026-05-19T09:02:56.507376007Z","level":"INFO","msg":"2️⃣ Liquidating cluster","test":"sanity","module":"contract","tx_hash":"0x661439e5c3a0df123c9b9066d38476bc1b2a017d47af305e2056b18aba894593"}
{"time":"2026-05-19T09:02:56.516479542Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x661439e5c3a0df123c9b9066d38476bc1b2a017d47af305e2056b18aba894593"}
{"time":"2026-05-19T09:03:06.525950842Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x661439e5c3a0df123c9b9066d38476bc1b2a017d47af305e2056b18aba894593"}
{"time":"2026-05-19T09:03:06.525996475Z","level":"INFO","msg":"📜 CONTRACT: Cluster Liquidation Successful","test":"sanity","txHash":"0x661439e5c3a0df123c9b9066d38476bc1b2a017d47af305e2056b18aba894593","block_hash":"0xf7d4e2cb8a9c86a36e30c9f6274cc569275cfd1c5ed7ebdb4f1849ec03d384c8"}
{"time":"2026-05-19T09:03:06.630571723Z","level":"INFO","msg":"Loki: no matching logs yet, retrying until data appears or max attempts is reached","module":"main","module":"loki-validator","attempts":1,"retry_delay":"20s","start_unix":1779170586,"end_unix":1779181386}
{"time":"2026-05-19T09:05:06.883430099Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterLiquidated"}
{"time":"2026-05-19T09:05:06.883506956Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"sanity","indices":[],"validators":0,"validate-type":"liquidation"}
{"time":"2026-05-19T09:05:06.883535856Z","level":"INFO","msg":"✅ SUCCESS: Liquidation Validation Success","test":"sanity","validate_type":"liquidation","e2m_enabled":false,"e2m_success":false,"loki_success":true}
{"time":"2026-05-19T09:05:10.820603465Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"reactivate","estimated_gas":140820,"gas_buffer_pct":1000,"gas_limit":1549020}
{"time":"2026-05-19T09:05:10.830426118Z","level":"INFO","msg":"3️⃣ Reactivating cluster","test":"sanity","module":"contract","tx_hash":"0xa5af17aa32b37a82ab7cc6e3e5ea816bc0d54f9cd55e9ffcc7cbe67d969d37b3"}
{"time":"2026-05-19T09:05:10.838515004Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xa5af17aa32b37a82ab7cc6e3e5ea816bc0d54f9cd55e9ffcc7cbe67d969d37b3"}
{"time":"2026-05-19T09:05:20.851489655Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xa5af17aa32b37a82ab7cc6e3e5ea816bc0d54f9cd55e9ffcc7cbe67d969d37b3"}
{"time":"2026-05-19T09:05:40.856083076Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xa5af17aa32b37a82ab7cc6e3e5ea816bc0d54f9cd55e9ffcc7cbe67d969d37b3"}
{"time":"2026-05-19T09:05:40.856155799Z","level":"INFO","msg":"📜 CONTRACT: Cluster Reactivation Successful","test":"sanity","txHash":"0xa5af17aa32b37a82ab7cc6e3e5ea816bc0d54f9cd55e9ffcc7cbe67d969d37b3","block_hash":"0xf8bf3ec82b85319a04b0f817eaeefe206de0d916a9e7511ec42c5924a6c3b34c"}
{"time":"2026-05-19T09:05:40.941138458Z","level":"INFO","msg":"Loki: no matching logs yet, retrying until data appears or max attempts is reached","module":"main","module":"loki-validator","attempts":1,"retry_delay":"20s","start_unix":1779170740,"end_unix":1779181540}
{"time":"2026-05-19T09:07:21.052565323Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterReactivated"}
{"time":"2026-05-19T09:07:21.052632063Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"sanity","indices":[],"validators":1,"validate-type":"reactivation"}
{"time":"2026-05-19T09:07:21.052651858Z","level":"INFO","msg":"✅ SUCCESS: Reactivation Validation Success","test":"sanity","validate_type":"reactivation","e2m_enabled":false,"e2m_success":false,"loki_success":true}
{"time":"2026-05-19T09:07:25.093892942Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"removeValidator","estimated_gas":144470,"gas_buffer_pct":1000,"gas_limit":1589170}
{"time":"2026-05-19T09:07:25.10625682Z","level":"INFO","msg":"4️⃣ Removing validator","test":"sanity","module":"contract","tx_hash":"0xccc629782f7bd8e5ad2cb53a3fbf17087d835d65b17926f94298f0e1dfd9e791"}
{"time":"2026-05-19T09:07:25.114150007Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xccc629782f7bd8e5ad2cb53a3fbf17087d835d65b17926f94298f0e1dfd9e791"}
{"time":"2026-05-19T09:07:35.122669175Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xccc629782f7bd8e5ad2cb53a3fbf17087d835d65b17926f94298f0e1dfd9e791"}
{"time":"2026-05-19T09:07:35.122727175Z","level":"INFO","msg":"📜 CONTRACT: Contract Removal Successful","test":"sanity","txHash":"0xccc629782f7bd8e5ad2cb53a3fbf17087d835d65b17926f94298f0e1dfd9e791","block_hash":"0xec55d20ab4fe32f3fe99be5f2186b279a289ab8f2782fde73e07185aec2054a6"}
{"time":"2026-05-19T09:07:35.402173028Z","level":"INFO","msg":"Loki: no matching logs yet, retrying until data appears or max attempts is reached","module":"main","module":"loki-validator","attempts":1,"retry_delay":"20s","start_unix":1779170855,"end_unix":1779181655}
{"time":"2026-05-19T09:09:35.551633051Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":1,"event":"ValidatorRemoved"}
{"time":"2026-05-19T09:09:35.551671061Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"sanity","indices":[],"validators":1,"validate-type":"removal"}
{"time":"2026-05-19T09:09:35.55168985Z","level":"INFO","msg":"✅ SUCCESS: Removal Validation Success","test":"sanity","validate_type":"removal","e2m_enabled":false,"e2m_success":false,"loki_success":true}
{"time":"2026-05-19T09:09:35.562504469Z","level":"INFO","msg":"✅ SUCCESS: Sanity Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":false,"Loki":true},"liquidate":{"Contract":true,"E2M":false,"Loki":true},"reactivate":{"Contract":true,"E2M":false,"Loki":true},"remove":{"Contract":true,"E2M":false,"Loki":true}}
{"time":"2026-05-19T09:09:35.562539084Z","level":"INFO","msg":"🔑 KEY STEP: Starting Bulk Test Suite","module":"main"}
{"time":"2026-05-19T09:09:35.562557747Z","level":"INFO","msg":"Bulk: config values","test":"bulk","cluster_owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","account_public_key":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","operator_ids":[304,305,306,307]}
{"time":"2026-05-19T09:09:35.59730123Z","level":"INFO","msg":"Available slots","test":"bulk","node":304,"slots":3000}
{"time":"2026-05-19T09:09:35.602986318Z","level":"INFO","msg":"Available slots","test":"bulk","node":305,"slots":1010}
{"time":"2026-05-19T09:09:35.609161049Z","level":"INFO","msg":"Available slots","test":"bulk","node":306,"slots":3000}
{"time":"2026-05-19T09:09:35.61534846Z","level":"INFO","msg":"Available slots","test":"bulk","node":307,"slots":1324}
{"time":"2026-05-19T09:10:11.360792926Z","level":"INFO","msg":"Extracted keyshares path for bulk keyshare","test":"bulk","module":"keys","path":"/app/shared/data/keyshares/keyshares-1779181811.json"}
{"time":"2026-05-19T09:10:15.386437677Z","level":"INFO","msg":"Cluster asset type before write","test":"bulk","asset_type":1,"asset_err":null,"asset_label":"ETH"}
{"time":"2026-05-19T09:10:19.233136677Z","level":"WARN","msg":"failed to report run progress","test":"bulk","error":"progress update failed: no active usage tracking found with ID bbc05c559467bb578b2048415ad3400c"}
{"time":"2026-05-19T09:10:22.960674743Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"bulkRegisterValidator","estimated_gas":385812,"gas_buffer_pct":1000,"gas_limit":4243932}
{"time":"2026-05-19T09:10:22.978543201Z","level":"INFO","msg":"1️⃣ Bulk Registering validators","test":"bulk","module":"contract","tx_hash":"0xea3251d49cb8ebcac7495640e1562b4a8859f56ce116055efba331f316aa7c48"}
{"time":"2026-05-19T09:10:22.980028831Z","level":"WARN","msg":"failed to report run progress","test":"bulk","error":"progress update failed: no active usage tracking found with ID bbc05c559467bb578b2048415ad3400c"}
{"time":"2026-05-19T09:10:22.980056574Z","level":"INFO","msg":"Bulk registration transaction sent","test":"bulk","txHash":"0xea3251d49cb8ebcac7495640e1562b4a8859f56ce116055efba331f316aa7c48"}
{"time":"2026-05-19T09:10:22.983570217Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xea3251d49cb8ebcac7495640e1562b4a8859f56ce116055efba331f316aa7c48"}
{"time":"2026-05-19T09:10:32.987960007Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xea3251d49cb8ebcac7495640e1562b4a8859f56ce116055efba331f316aa7c48"}
{"time":"2026-05-19T09:10:52.995048228Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0xea3251d49cb8ebcac7495640e1562b4a8859f56ce116055efba331f316aa7c48"}
{"time":"2026-05-19T09:10:52.995097497Z","level":"INFO","msg":"📜 CONTRACT: Bulk Registration Successful","test":"bulk","blockHash":"0xf7fe79258618925dbb2eb64df485756a18e8604910e6bd3f03dd38d171f0447f","Validators Amount":3}
{"time":"2026-05-19T09:10:53.295952619Z","level":"INFO","msg":"Loki: no matching logs yet, retrying until data appears or max attempts is reached","module":"main","module":"loki-validator","attempts":1,"retry_delay":"20s","start_unix":1779171052,"end_unix":1779181852}
{"time":"2026-05-19T09:12:53.440337338Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":3,"event":"ValidatorAdded"}
{"time":"2026-05-19T09:12:53.440445305Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"bulk","indices":[],"validators":3,"validate-type":"registration"}
{"time":"2026-05-19T09:12:53.440498778Z","level":"INFO","msg":"✅ SUCCESS: Registration Validation Success","test":"bulk","validate_type":"registration","e2m_enabled":false,"e2m_success":false,"loki_success":true}
{"time":"2026-05-19T09:12:53.440529284Z","level":"INFO","msg":"Bulk: registration validation summary","test":"bulk","bulk_amount":3,"registered_pubkeys":3,"indices_for_e2m":0,"e2m_success":false,"loki_success":true}
{"time":"2026-05-19T09:12:57.479639443Z","level":"WARN","msg":"failed to report run progress","test":"bulk","error":"progress update failed: no active usage tracking found with ID bbc05c559467bb578b2048415ad3400c"}
{"time":"2026-05-19T09:12:57.490371327Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"liquidate","estimated_gas":149318,"gas_buffer_pct":1000,"gas_limit":1642498}
{"time":"2026-05-19T09:12:57.500552508Z","level":"INFO","msg":"2️⃣ Liquidating cluster","test":"bulk","module":"contract","tx_hash":"0x6efe869b4410114efb3f003c59863212b7c5273f2615f31946841e92c5b478d5"}
{"time":"2026-05-19T09:12:57.502990382Z","level":"WARN","msg":"failed to report run progress","test":"bulk","error":"progress update failed: no active usage tracking found with ID bbc05c559467bb578b2048415ad3400c"}
{"time":"2026-05-19T09:12:57.50605543Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x6efe869b4410114efb3f003c59863212b7c5273f2615f31946841e92c5b478d5"}
{"time":"2026-05-19T09:13:07.514004067Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x6efe869b4410114efb3f003c59863212b7c5273f2615f31946841e92c5b478d5"}
{"time":"2026-05-19T09:13:07.514047761Z","level":"INFO","msg":"📜 CONTRACT: Bulk Cluster Liquidation Successful","test":"bulk","txHash":"0x6efe869b4410114efb3f003c59863212b7c5273f2615f31946841e92c5b478d5","block_hash":"0xc443ff0567eb863f9a74dafe53dc1356e9bd88df7af6f0834c5200ad061c9eed"}
{"time":"2026-05-19T09:13:07.767760449Z","level":"INFO","msg":"Loki: no matching logs yet, retrying until data appears or max attempts is reached","module":"main","module":"loki-validator","attempts":1,"retry_delay":"20s","start_unix":1779171187,"end_unix":1779181987}
{"time":"2026-05-19T09:14:47.925738266Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterLiquidated"}
{"time":"2026-05-19T09:14:47.925816953Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"bulk","indices":[],"validators":0,"validate-type":"liquidation"}
{"time":"2026-05-19T09:14:47.925842194Z","level":"INFO","msg":"✅ SUCCESS: Liquidation Validation Success","test":"bulk","validate_type":"liquidation","e2m_enabled":false,"e2m_success":false,"loki_success":true}
{"time":"2026-05-19T09:14:52.072734508Z","level":"WARN","msg":"failed to report run progress","test":"bulk","error":"progress update failed: no active usage tracking found with ID bbc05c559467bb578b2048415ad3400c"}
{"time":"2026-05-19T09:14:52.084407816Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"reactivate","estimated_gas":140820,"gas_buffer_pct":1000,"gas_limit":1549020}
{"time":"2026-05-19T09:14:52.098351832Z","level":"INFO","msg":"3️⃣ Reactivating cluster","test":"bulk","module":"contract","tx_hash":"0x8453f8a6a1ad5e21d2935f9a9d9fbd125a4808312a20ae7caeee29dc2bddf762"}
{"time":"2026-05-19T09:14:52.100336274Z","level":"WARN","msg":"failed to report run progress","test":"bulk","error":"progress update failed: no active usage tracking found with ID bbc05c559467bb578b2048415ad3400c"}
{"time":"2026-05-19T09:14:52.104127448Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x8453f8a6a1ad5e21d2935f9a9d9fbd125a4808312a20ae7caeee29dc2bddf762"}
{"time":"2026-05-19T09:15:02.12290069Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x8453f8a6a1ad5e21d2935f9a9d9fbd125a4808312a20ae7caeee29dc2bddf762"}
{"time":"2026-05-19T09:15:02.122947849Z","level":"INFO","msg":"📜 CONTRACT: Bulk Cluster Reactivation Successful","test":"bulk","txHash":"0x8453f8a6a1ad5e21d2935f9a9d9fbd125a4808312a20ae7caeee29dc2bddf762","block_hash":"0x80f63fbbfbfee5f82948707002133d6f33fe31bd34361303683418d523edbade"}
{"time":"2026-05-19T09:15:02.396504042Z","level":"INFO","msg":"Loki: no matching logs yet, retrying until data appears or max attempts is reached","module":"main","module":"loki-validator","attempts":1,"retry_delay":"20s","start_unix":1779171302,"end_unix":1779182102}
{"time":"2026-05-19T09:17:02.590906503Z","level":"INFO","msg":"Event validation successful","module":"main","module":"loki-validator","valid_entries":4,"event":"ClusterReactivated"}
{"time":"2026-05-19T09:17:02.590948992Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"bulk","indices":[],"validators":3,"validate-type":"reactivation"}
{"time":"2026-05-19T09:17:02.590968127Z","level":"INFO","msg":"✅ SUCCESS: Reactivation Validation Success","test":"bulk","validate_type":"reactivation","e2m_enabled":false,"e2m_success":false,"loki_success":true}
{"time":"2026-05-19T09:17:02.597066269Z","level":"WARN","msg":"failed to report run progress","test":"bulk","error":"progress update failed: no active usage tracking found with ID bbc05c559467bb578b2048415ad3400c"}
{"time":"2026-05-19T09:17:06.510925644Z","level":"INFO","msg":"📝 tx gas limit set (estimate + buffer)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"bulkRemoveValidator","estimated_gas":173224,"gas_buffer_pct":1000,"gas_limit":1905464}
{"time":"2026-05-19T09:17:06.525931804Z","level":"INFO","msg":"4️⃣ Bulk Removing validators","test":"bulk","module":"contract","tx_hash":"0xb4b7a81b89e5b1f592b7c1ce1cd0029d04e7dd15e6cef894df106115e8686c7e"}
{"time":"2026-05-19T09:17:06.530673275Z","level":"WARN","msg":"failed to report run progress","test":"bulk","error":"progress update failed: no active usage tracking found with ID bbc05c559467bb578b2048415ad3400c"}
{"time":"2026-05-19T09:17:06.53072747Z","level":"INFO","msg":"Bulk removal transaction sent","test":"bulk","txHash":"0xb4b7a81b89e5b1f592b7c1ce1cd0029d04e7dd15e6cef894df106115e8686c7e"}
{"time":"2026-05-19T09:17:06.534159657Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xb4b7a81b89e5b1f592b7c1ce1cd0029d04e7dd15e6cef894df106115e8686c7e"}
{"time":"2026-05-19T09:17:16.54292723Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":2,"sleep":20,"tx_hash":"0xb4b7a81b89e5b1f592b7c1ce1cd0029d04e7dd15e6cef894df106115e8686c7e"}
{"time":"2026-05-19T09:17:36.546717612Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":3,"sleep":40,"tx_hash":"0xb4b7a81b89e5b1f592b7c1ce1cd0029d04e7dd15e6cef894df106115e8686c7e"}
{"time":"2026-05-19T09:18:16.56380768Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0xb4b7a81b89e5b1f592b7c1ce1cd0029d04e7dd15e6cef894df106115e8686c7e"}
{"time":"2026-05-19T09:18:16.563852517Z","level":"INFO","msg":"📜 CONTRACT: Bulk Removal Successful","test":"bulk","blockHash":"0x42d17399cee1112ed3b3e8af679f37525ee6941fe59758f028cffc947912e538","Validators Amount":3}
{"time":"2026-05-19T09:18:16.660968822Z","level":"INFO","msg":"Loki: no matching logs yet, retrying until data appears or max attempts is reached","module":"main","module":"loki-validator","attempts":1,"retry_delay":"20s","start_unix":1779171496,"end_unix":1779182296}
{"time":"2026-05-19T09:19:16.76816576Z","level":"INFO","msg":"Validator event validation successful","module":"main","module":"loki-validator","validated_validators":3,"event":"ValidatorRemoved"}
{"time":"2026-05-19T09:19:16.768228616Z","level":"INFO","msg":"Loki: SSV node log validation successful","test":"bulk","indices":[],"validators":3,"validate-type":"removal"}
{"time":"2026-05-19T09:19:16.768252754Z","level":"INFO","msg":"✅ SUCCESS: Removal Validation Success","test":"bulk","validate_type":"removal","e2m_enabled":false,"e2m_success":false,"loki_success":true}
{"time":"2026-05-19T09:19:16.768273909Z","level":"INFO","msg":"Bulk: removal validation summary","test":"bulk","bulk_amount":3,"removed_pubkeys":3,"indices_for_e2m":0,"e2m_success":false,"loki_success":true}
{"time":"2026-05-19T09:19:16.774595048Z","level":"WARN","msg":"failed to report run progress","test":"bulk","error":"progress update failed: no active usage tracking found with ID bbc05c559467bb578b2048415ad3400c"}
{"time":"2026-05-19T09:19:16.774643823Z","level":"INFO","msg":"✅ SUCCESS: Bulk Test Suite Completed Successfully","module":"main","register":{"Contract":true,"E2M":false,"Loki":true},"liquidate":{"Contract":true,"E2M":false,"Loki":true},"reactivate":{"Contract":true,"E2M":false,"Loki":true},"remove":{"Contract":true,"E2M":false,"Loki":true}}
{"time":"2026-05-19T09:19:16.77464921Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2026-05-19T09:19:16.774670545Z","level":"INFO","msg":"✅ SUCCESS: All Test Suites Completed Successfully","module":"main","passed_suites":["sanity","bulk"],"skipped_suites":["isolated"]}
{"time":"2026-05-19T09:19:16.77467609Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"bbc05c559467bb578b2048415ad3400c"}
{"time":"2026-05-19T09:19:16.776445552Z","level":"INFO","msg":"Resource usage already cleared, skipping teardown attempts","module":"main","usage_id":"bbc05c559467bb578b2048415ad3400c"}

2026-05-19T09:19:20Z The test run finished successfully

Result - 🟢 Success

2026-05-19T09:19:26Z The test run finished successfully
2026-05-19T09:19:31Z Finished cleaning reserved resources

Adds the 4th-or-5th audit lane (alongside Devin / Qwen / Codex / Greptile, depending on whether Codex PR #234 lands first). Calibration-phase informational only — does NOT yet authorize Claude's standing in-thread merge delegation. Greptile is a SaaS GitHub App; there's no client to build. This PR ships the LABELER infrastructure that reads Greptile's reviews and applies our standard `greptile-audit-*` labels. Dormant until the Greptile App is installed on the repo. PR #233 calibration showed Devin / Qwen / Codex have complementary strengths (Devin = final-state QA, Codex = code tracing, Qwen = currently too noisy). Greptile is a 4th data point with a distinctive mechanic: graph-indexed whole-repo context + claims to learn from existing review-comment history. Worth a 10-20 PR bake-off to see if its findings are differentiated. | Aspect | Devin | Qwen | Codex | Greptile | |---|---|---|---|---| | Trigger | webhook | polling daemon | manual CLI | GitHub App auto-fires on every PR | | Where it runs | cloud | local LM Studio | local subprocess | SaaS | | Review style | final-state QA | per-file LLM | code tracing | graph-indexed | | Verdict signal | trailer in comment | trailer in comment | trailer in comment | P0/P1/P2/P3 badges in inline review comments | | Event type | issue_comment | issue_comment | issue_comment | pull_request_review | | Our infra | bridge + labeler | bridge + CLI + labeler | CLI + labeler | labeler only | tools/greptile_audit_labeler.py (NEW, ~380 lines) Fires on pull_request_review events from greptile-apps[bot]. Four defensive gates per Codex's tightening of the scope: 1. Opt-in: only flip labels on PRs that currently carry `needs-greptile-audit`. Greptile auto-reviews every PR via the App; only opted-in PRs get labeled. 2. Stale-HEAD: compare review.commit_id to PR head SHA. If they differ, re-queue. 3. Severity parse: extract from `alt="P[N]"` (preferred) or badge URL pattern `greptile-static-assets.../p{N}.svg` (cross-check). Fail closed to needs-greptile-audit if inline comments exist but NO severity markers can be parsed — Greptile format drift becomes a re-queue, not a silent auto-PASS. 4. Verdict: P0 / P1 → blocked. P2 / P3-only → done (concerns surfaced but non-blocking). Clean review (no inline comments) → done. .github/workflows/greptile-audit-labeler.yml (NEW) Fires on pull_request_review:submitted|edited. Token permissions documented (Issues read+write + Pull requests read). tools/GREPTILE_AUDIT_PROTOCOL.md Refined from /tmp/greptile_scope.md (the Codex-input version). Documents the four gates, the four-reviewer comparison matrix, bake-off plan, and decommissioning instructions. tests/test_greptile_audit_labeler.py (NEW, 24 tests) Pure-function tests over severity parsing + verdict classification + decision logic. Uses a real Greptile inline comment body (captured from ssvlabs/ssv#2835, the only public Greptile-installed repo I could find) as the P1 fixture. Synthesizes P0/P2/P3 by patching the badge tier. Covers: Severity parsing: real P1, synthetic P0/P2/P3, alt-only, URL-only, alt+URL disagreement (alt wins), no markers (None) Classification: no findings = done, P3-only = done, P1 = blocked, P0 = blocked, mixed-severity = blocked-when-any-blocker, format drift = needs (fail closed), partial parse uses what we got Decision gates: non-submitted action skipped, non-Greptile author skipped, opt-in gate, stale-HEAD gate, clean → done, blocker → blocked, format drift → needs, P2/P3-only → done Author env override: defaults when unset, defaults when empty (#234 fix pattern), custom override tools/README.md Add Greptile + Qwen entries to the Infra section. - No cube-snap mirror — companion PR will land byte-identical per the existing infra-mirror convention. - No Greptile install on the repo (user-controlled; cube-snap is MIT and qualifies for the free OSS tier; ctvd is private and needs $30/mo Pro or skip). - No `.greptile/rules.md` configuration (deferred until install, per the protocol doc). - No labels created yet (manual `gh label create` post-merge). - No Claude-side merge-delegation change — Devin still owns merge authority. Greptile stays informational. The labeler tests use ONE real Greptile payload (P1, captured from ssvlabs/ssv#2835). Other severities (P0, P2, P3) are synthesized by patching the badge tier; the actual Greptile output for those tiers is expected to follow the same shape but hasn't been independently verified. Codex's scoping called for fixtures of all 4 shapes (no-findings, P1, P2/P3-only, stale-SHA) plus the clean-review case; those are captured-on-install in the protocol doc as a follow-up. 24 tests pass (`pytest tests/test_greptile_audit_labeler.py`) py_compile clean on the labeler Workflow YAML syntactically valid Severity parse handles the real Greptile P1 body correctly Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Greptile audit lane: labeler + workflow + protocol + tests Adds the 4th-or-5th audit lane (alongside Devin / Qwen / Codex / Greptile, depending on whether Codex PR #234 lands first). Calibration-phase informational only — does NOT yet authorize Claude's standing in-thread merge delegation. Greptile is a SaaS GitHub App; there's no client to build. This PR ships the LABELER infrastructure that reads Greptile's reviews and applies our standard `greptile-audit-*` labels. Dormant until the Greptile App is installed on the repo. PR #233 calibration showed Devin / Qwen / Codex have complementary strengths (Devin = final-state QA, Codex = code tracing, Qwen = currently too noisy). Greptile is a 4th data point with a distinctive mechanic: graph-indexed whole-repo context + claims to learn from existing review-comment history. Worth a 10-20 PR bake-off to see if its findings are differentiated. | Aspect | Devin | Qwen | Codex | Greptile | |---|---|---|---|---| | Trigger | webhook | polling daemon | manual CLI | GitHub App auto-fires on every PR | | Where it runs | cloud | local LM Studio | local subprocess | SaaS | | Review style | final-state QA | per-file LLM | code tracing | graph-indexed | | Verdict signal | trailer in comment | trailer in comment | trailer in comment | P0/P1/P2/P3 badges in inline review comments | | Event type | issue_comment | issue_comment | issue_comment | pull_request_review | | Our infra | bridge + labeler | bridge + CLI + labeler | CLI + labeler | labeler only | tools/greptile_audit_labeler.py (NEW, ~380 lines) Fires on pull_request_review events from greptile-apps[bot]. Four defensive gates per Codex's tightening of the scope: 1. Opt-in: only flip labels on PRs that currently carry `needs-greptile-audit`. Greptile auto-reviews every PR via the App; only opted-in PRs get labeled. 2. Stale-HEAD: compare review.commit_id to PR head SHA. If they differ, re-queue. 3. Severity parse: extract from `alt="P[N]"` (preferred) or badge URL pattern `greptile-static-assets.../p{N}.svg` (cross-check). Fail closed to needs-greptile-audit if inline comments exist but NO severity markers can be parsed — Greptile format drift becomes a re-queue, not a silent auto-PASS. 4. Verdict: P0 / P1 → blocked. P2 / P3-only → done (concerns surfaced but non-blocking). Clean review (no inline comments) → done. .github/workflows/greptile-audit-labeler.yml (NEW) Fires on pull_request_review:submitted|edited. Token permissions documented (Issues read+write + Pull requests read). tools/GREPTILE_AUDIT_PROTOCOL.md Refined from /tmp/greptile_scope.md (the Codex-input version). Documents the four gates, the four-reviewer comparison matrix, bake-off plan, and decommissioning instructions. tests/test_greptile_audit_labeler.py (NEW, 24 tests) Pure-function tests over severity parsing + verdict classification + decision logic. Uses a real Greptile inline comment body (captured from ssvlabs/ssv#2835, the only public Greptile-installed repo I could find) as the P1 fixture. Synthesizes P0/P2/P3 by patching the badge tier. Covers: Severity parsing: real P1, synthetic P0/P2/P3, alt-only, URL-only, alt+URL disagreement (alt wins), no markers (None) Classification: no findings = done, P3-only = done, P1 = blocked, P0 = blocked, mixed-severity = blocked-when-any-blocker, format drift = needs (fail closed), partial parse uses what we got Decision gates: non-submitted action skipped, non-Greptile author skipped, opt-in gate, stale-HEAD gate, clean → done, blocker → blocked, format drift → needs, P2/P3-only → done Author env override: defaults when unset, defaults when empty (#234 fix pattern), custom override tools/README.md Add Greptile + Qwen entries to the Infra section. - No cube-snap mirror — companion PR will land byte-identical per the existing infra-mirror convention. - No Greptile install on the repo (user-controlled; cube-snap is MIT and qualifies for the free OSS tier; ctvd is private and needs $30/mo Pro or skip). - No `.greptile/rules.md` configuration (deferred until install, per the protocol doc). - No labels created yet (manual `gh label create` post-merge). - No Claude-side merge-delegation change — Devin still owns merge authority. Greptile stays informational. The labeler tests use ONE real Greptile payload (P1, captured from ssvlabs/ssv#2835). Other severities (P0, P2, P3) are synthesized by patching the badge tier; the actual Greptile output for those tiers is expected to follow the same shape but hasn't been independently verified. Codex's scoping called for fixtures of all 4 shapes (no-findings, P1, P2/P3-only, stale-SHA) plus the clean-review case; those are captured-on-install in the protocol doc as a follow-up. 24 tests pass (`pytest tests/test_greptile_audit_labeler.py`) py_compile clean on the labeler Workflow YAML syntactically valid Severity parse handles the real Greptile P1 body correctly Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Address Codex review round 1: fail-closed-on-any-unparsed + paginate Codex review of PR #235 found 2 P2 findings, both real: ## P2 #1: mixed parsed + unparsed should fail closed Original `classify()` only failed-closed when ALL comments were unparseable — a P3-only review with ONE drifted-format unparsed comment would fall through to 'done' and silently ignore the potential hidden blocker. Fix: ANY unparsed comment triggers fail-closed (returns 'needs'). This matches the fail-closed gate documented in the protocol and the conservative posture the labeler is supposed to take. Updated `test_classify_mixed_parsed_and_unparsed_*` (was 'use what we got', now 'fail closed'). Added 2 more guards for one-unparsed and unparsed-alongside-P3. ## P2 #2: pagination `fetch_review_comments` only fetched page 1 of `/pulls/{n}/comments` (100 max). For a PR with >100 prior inline comments OR a Greptile review with >100 findings, target comments on page 2+ would be invisible — a P0/P1 there would be labeled `greptile-audit-done`. Fix: paginate up to 5 pages = 500 comments. Stop when chunk is empty or has <100 entries. Added `test_fetch_review_comments_paginates_until_empty` (3 pages, two full + one partial) and `test_fetch_review_comments_filters_by_review_id` (ensures the filter still applies across pages). ## Tests 28 passed (was 24; +4 net: 3 new pagination/fail-closed tests plus the updated mixed-parsed-unparsed test) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Address Codex review round 2: include done/blocked in opt-in + fail-closed pagination cap Codex round 2 found 2 P2 findings, both real: ## P2 #1: opt-in gate stuck after first review `needs-greptile-audit` is REMOVED by the first review's decision (replaced by done or blocked). Subsequent Greptile reviews on the same PR (e.g., after a fix push) would fail the opt-in gate and be silently ignored, leaving the now-stale done/blocked label. Fix: opt-in gate accepts ANY greptile-audit-* label (needs / done / blocked). Once a PR is opted in, it stays in the bake-off and state-machine transitions work normally. Tests: + test_decision_opt_in_gate_accepts_done_label_too (P1 found in follow-up review flips done→blocked) + test_decision_opt_in_gate_accepts_blocked_label_too (clean follow-up review flips blocked→done) Renamed: test_decision_opt_in_gate_skips_pr_without_needs_label → ..._without_any_greptile_label ## P2 #2: pagination cap silently truncated Original code stopped at page 5 (500 comments) and returned partial data. If a Greptile review had findings on page 6+ (degenerate but possible for very busy PRs), a P0/P1 there would be invisible and the audit labeled `done`. Fix: bump cap to 50 pages (5000 comments) and raise `ReviewCommentsTruncated` if cap is hit without exhaustion. `main()` catches that, applies `needs-greptile-audit` (fail closed), and logs the cause. Test: test_fetch_review_comments_raises_on_pagination_cap. ## Tests 31 passed (was 28; +3: 2 opt-in-gate guards + 1 pagination-cap raise guard). Codex round 3 next. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Address Codex review round 3: gate truncation fallback by author + opt-in Codex round 3 found one P3 — non-blocking per the severity policy but a real correctness fix worth applying. ## P3: truncation fallback bypassed author/opt-in gates The pagination-cap fail-closed branch in main() applied `needs-greptile-audit` before resolve_label_decision had a chance to check the review author or opt-in label. So a Codex review (or any non-Greptile review) on a PR with >5000 inline comments, OR a Greptile review on a non-opted-in PR, could trigger the fallback and incorrectly mutate Greptile labels. Fix: replicate the author check and opt-in label check inside the truncation fallback. Same gates as resolve_label_decision; same skip messages. If either check fails, skip the fallback entirely. No new tests (the integration test would need full event-fixture plumbing for the main() entrypoint; the pure-function tests already cover the gates in resolve_label_decision and the truncation raise in fetch_review_comments). 31 passed (unchanged; the fix is to main() which the existing test suite doesn't enter). Codex round 4 next; round 3 verdict via the audit-lane policy (P3-only = PASS) but I applied the fix anyway since it's real. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

iurii-ssv requested review from a team as code owners April 27, 2026 16:26

iurii-ssv marked this pull request as draft April 27, 2026 16:26

greptile-apps Bot reviewed Apr 27, 2026

View reviewed changes

Comment thread protocol/v2/qbft/instance/proposal.go Outdated

iurii-ssv changed the title ~~qbft: correct Instance message retries~~ qbft: clarify round-changes Apr 27, 2026

iurii-ssv force-pushed the qbft-correct-instance-message-retries branch from a3532ce to 612f35a Compare April 28, 2026 07:22

iurii-ssv marked this pull request as ready for review April 28, 2026 07:35

iurii-ssv force-pushed the qbft-correct-instance-message-retries branch from 9ccaf80 to 612f35a Compare April 28, 2026 07:49

julienh-ssv reviewed Apr 29, 2026

View reviewed changes

Comment thread protocol/v2/qbft/instance/round_change.go

momosh-ssv reviewed Apr 30, 2026

View reviewed changes

iurii-ssv requested review from julienh-ssv and momosh-ssv May 12, 2026 12:21

iurii-ssv added 4 commits May 12, 2026 16:36

qbft: clarify round-changes

f977043

cleanup

bcd0593

iurii-ssv force-pushed the qbft-correct-instance-message-retries branch from 8aba10d to 8db85bb Compare May 12, 2026 13:39

Merge branch 'stage' into qbft-correct-instance-message-retries

82c4b6b

julienh-ssv approved these changes May 19, 2026

View reviewed changes

Conversation

iurii-ssv commented Apr 27, 2026 • edited by momosh-ssv Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

Uh oh!

iurii-ssv commented Apr 27, 2026

Uh oh!

iurii-ssv commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

y0sher commented May 11, 2026

Uh oh!

iurii-ssv commented May 12, 2026

Uh oh!

momosh-ssv commented May 12, 2026

Uh oh!

iurii-ssv commented May 14, 2026

Uh oh!

ssv-test-bot Bot commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Test run for PR #2835

Status

Result - 🔴 Failure

Uh oh!

AlexM-ssv-labs commented May 15, 2026

Uh oh!

ssv-test-bot Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Test run for PR #2835

Status

Result - 🔴 Failure

Uh oh!

iurii-ssv commented May 18, 2026

Uh oh!

ssv-test-bot Bot commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Test run for PR #2835

Status

Result - 🔴 Failure

Uh oh!

julienh-ssv left a comment

Choose a reason for hiding this comment

Uh oh!

AlexM-ssv-labs commented May 19, 2026

Uh oh!

ssv-test-bot Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Test run for PR #2835

Status

Result - 🟢 Success

Uh oh!

Reviewers

iurii-ssv commented Apr 27, 2026 •

edited by momosh-ssv

Loading

codecov Bot commented Apr 27, 2026 •

edited

Loading

greptile-apps Bot commented Apr 27, 2026 •

edited

Loading

iurii-ssv commented Apr 28, 2026 •

edited

Loading

ssv-test-bot Bot commented May 14, 2026 •

edited

Loading

ssv-test-bot Bot commented May 15, 2026 •

edited

Loading

ssv-test-bot Bot commented May 18, 2026 •

edited

Loading

ssv-test-bot Bot commented May 19, 2026 •

edited

Loading