Skip to content

vaclav-test-ci-123 DEBUG CI#2856

Draft
vaclav-ssvlabs wants to merge 1 commit into
stagefrom
vaclav-test-ci-123
Draft

vaclav-test-ci-123 DEBUG CI#2856
vaclav-ssvlabs wants to merge 1 commit into
stagefrom
vaclav-test-ci-123

Conversation

@vaclav-ssvlabs
Copy link
Copy Markdown
Contributor

chore: fix incorrect comment in ssvNodeFilter function

@vaclav-ssvlabs
Copy link
Copy Markdown
Contributor Author

/test help

@ssv-test-bot
Copy link
Copy Markdown

ssv-test-bot Bot commented May 19, 2026

❓ Help

Available Commands

/test - Run Tests

To run tests, use the command /test followed by optional parameters.
Example:
/test cluster=300,301,302,303 tests=(sanity,e2m) network=hoodi_stage

Available parameters (all optional):

  • cluster — Comma-separated operator IDs (4 or 7 numbers in ascending order). If omitted (recommended), the workflow auto-allocates nodes. Valid range: 300-333.
  • tests — A list describing which tests to run. Default: (sanity,e2m)
  • network — Network configuration to use. Default: hoodi_stage
  • extraEnvs — Comma-separated KEY=value pairs injected as env vars into the deployed SSV node pods. Values containing commas must be quoted: KEY="a,b,c". Ignored on network=local_testnet.

🧪 TESTS parameter

Format:

(type[,options]),(type[,options]),...

Supported test types:

  • sanity[,e2m][,threshold] — Sanity tests (optional e2m validation, optional threshold testing)
  • bulk[,e2m][,threshold][,amount] — Bulk tests (optional e2m/threshold validation; optional validator count as amount - default 3, max 10)
  • vouch,key:value — Vouch tests (requires all keys)
  • fee-recipient,key:value — Fee recipient validation tests (requires newAddress, optional ownerAddress)

Isolated parameters for vouch (all required):

  • epochs:N — Number of epochs to examine
  • amountOfETHNodes:N — Number of ETH1 nodes vouch deployed with

Isolated parameters for fee-recipient:

  • newAddress:0x...Required - New fee recipient address to set and validate
  • ownerAddress:0x...Optional - Cluster owner address (defaults to cluster owner from config)

💡 Usage examples

  • Default (no cluster, no network, no tests; defaults kick in):
/test
  • Simple bulk run with amount only, default network (hoodi_stage):
/test cluster=300,301,302,303 tests=(bulk,5)
  • With specific cluster, e2m on both sanity and bulk, fixed amount:
/test cluster=300,301,302,303 tests=(sanity,e2m),(bulk,e2m,10) network=hoodi_stage
  • Sanity plus a vouch run:
/test cluster=300,301,302,303 tests=(sanity),(vouch,epochs:5,amountOfETHNodes:4) network=hoodi_stage
  • Simple bulk run with default validators amount (3):
/test cluster=300,301,302,303 tests=(bulk) network=hoodi_stage
  • Sanity with threshold testing:
/test cluster=300,301,302,303 tests=(sanity,threshold) network=hoodi_stage
  • Bulk with threshold and custom amount:
/test cluster=300,301,302,303 tests=(bulk,threshold,5) network=hoodi_stage
  • Combined e2m and threshold testing:
/test cluster=300,301,302,303 tests=(sanity,e2m,threshold),(bulk,e2m,threshold,10) network=hoodi_stage
  • Fee recipient validation test:
/test cluster=300,301,302,303 tests=(fee-recipient,newAddress:0xd2DBd02e4efe087d7d195de828b9Dd25f19A89C9) network=hoodi_stage
  • Fee recipient test with custom owner address:
/test cluster=300,301,302,303 tests=(fee-recipient,newAddress:0xd2DBd02e4efe087d7d195de828b9Dd25f19A89C9,ownerAddress:0x5cc0dde14e7256340cc820415a6022a7d1c93a35) network=hoodi_stage
  • Combined sanity and fee recipient tests:
/test cluster=300,301,302,303 tests=(sanity),(fee-recipient,newAddress:0xd2DBd02e4efe087d7d195de828b9Dd25f19A89C9) network=hoodi_stage
  • Inject extra env vars into the SSV node pods (e.g. raise log verbosity, toggle a feature flag):
/test cluster=300,301,302,303 extraEnvs=LOG_LEVEL=debug,FEATURE_X=1

/test cancel - Cancel an Active Test Run

Forces immediate cleanup of any active test run for this PR, skipping the 15-minute debug grace period.
Useful when you spotted a problem and want to free the SSV nodes for a rerun without waiting.

/test cancel

/test skip-ci - Skip CI Pipeline

⚠️ Restricted Command: Only members of authorized teams can use this command.

Bypasses the CI pipeline by setting the configured status check to success.
Example:

/test skip-ci

Important Notes:

  • Only members of authorized teams (configurable) can execute this command
  • Use with caution as it bypasses all automated checks

@vaclav-ssvlabs
Copy link
Copy Markdown
Contributor Author

/test cluster=300,301,302,303 extraEnvs=VACLAV_ENV=trueFEATURE_X=1

@ssv-test-bot
Copy link
Copy Markdown

ssv-test-bot Bot commented May 19, 2026

🔴 Test run for PR #2856

Branch: vaclav-test-ci-123
Commit: a756fcf

Status

2026-05-19T07:10:29Z Starting test run
2026-05-19T07:10:31Z Test run submitted successfully for commit a756fcf
2026-05-19T07:10:31Z container image for commit a756fcf327c91aae98dc43078a9ddb499be5e307 is build and present in container registry
2026-05-19T07:10:33Z using user defined node range cluster=300,301,302,303 on network=hoodi_stage
2026-05-19T07:10:34Z deploying commit a756fcf327c91aae98dc43078a9ddb499be5e307 to ssv nodes 300,301,302,303
2026-05-19T07:11:36Z commit a756fcf327c91aae98dc43078a9ddb499be5e307 is deployed to ssv nodes 300,301,302,303
2026-05-19T07:11:38Z started test executor with tests=(sanity) and network=hoodi_stage

Click to expand executor logs
═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2026-05-19T07:12:09.02194763Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2026-05-19T07:12:09.026027189Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2026-05-19T07:12:09.026051087Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[300,301,302,303],"tests":"(sanity)"}
{"time":"2026-05-19T07:12:09.071426423Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"42fa98abea74b45c079dd9d268bfe6b8"}
{"time":"2026-05-19T07:12:09.072424715Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2026-05-19T07:12:09.072440341Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[300,301,302,303],"test_type":{"Sanity":{"ShouldRun":true,"E2M":false,"Threshold":false,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"Threshold":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2026-05-19T07:12:12.398968469Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2026-05-19T07:12:12.399034591Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2026-05-19T07:12:12.399045411Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2026-05-19T07:12:12.399062378Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":false}
{"time":"2026-05-19T07:12:12.399074544Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2026-05-19T07:12:12.399085884Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2026-05-19T07:12:12.39909852Z","level":"INFO","msg":"Sanity: config values","test":"sanity","cluster_owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","account_public_key":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","operator_ids":[300,301,302,303]}
{"time":"2026-05-19T07:12:12.399231931Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2026-05-19T07:12:12.439716092Z","level":"INFO","msg":"Available slots","test":"sanity","node":300,"slots":3000}
{"time":"2026-05-19T07:12:12.445470391Z","level":"INFO","msg":"Available slots","test":"sanity","node":301,"slots":1008}
{"time":"2026-05-19T07:12:12.451413979Z","level":"INFO","msg":"Available slots","test":"sanity","node":302,"slots":3000}
{"time":"2026-05-19T07:12:12.456502467Z","level":"INFO","msg":"Available slots","test":"sanity","node":303,"slots":1325}
{"time":"2026-05-19T07:12:40.93684588Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-19T07:12:40.936868866Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-19T07:12:44.883610942Z","level":"INFO","msg":"Cluster asset type before write","test":"sanity","asset_type":1,"asset_err":null,"asset_label":"ETH"}
{"time":"2026-05-19T07:12:48.832396374Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"registerValidator","gas_estimate_error":"gas estimation failed: execution reverted: 0x95a0cf33","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-19T07:12:48.843871027Z","level":"INFO","msg":"1️⃣ Registering validator","test":"sanity","module":"contract","tx_hash":"0x0ffe70a41d8477c986aad86fe467b9613956f22018e7808ccbd8807214694c40"}
{"time":"2026-05-19T07:12:48.850093555Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x0ffe70a41d8477c986aad86fe467b9613956f22018e7808ccbd8807214694c40"}
{"time":"2026-05-19T07:12:58.858939881Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":2,"sleep":20,"tx_hash":"0x0ffe70a41d8477c986aad86fe467b9613956f22018e7808ccbd8807214694c40"}
{"time":"2026-05-19T07:13:18.86195546Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x0ffe70a41d8477c986aad86fe467b9613956f22018e7808ccbd8807214694c40"}
{"time":"2026-05-19T07:13:18.862005222Z","level":"ERROR","msg":"❌ FAILURE: Contract Registration Failed - Transaction verification failed","test":"sanity","txHash":"0x0ffe70a41d8477c986aad86fe467b9613956f22018e7808ccbd8807214694c40"}
{"time":"2026-05-19T07:13:18.862041729Z","level":"ERROR","msg":"❌ FAILURE: Sanity Test Suite Failed","module":"main","error":"sanity register step failed: transaction verification failed"}
{"time":"2026-05-19T07:13:18.862046554Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2026-05-19T07:13:18.862083218Z","level":"ERROR","msg":"❌ FAILURE: Test Execution Completed with Failures","module":"main","passed_suites":null,"failed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2026-05-19T07:13:18.862093171Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"sanity","error":"sanity register step failed: transaction verification failed"}
{"time":"2026-05-19T07:13:18.862097971Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"42fa98abea74b45c079dd9d268bfe6b8"}
{"time":"2026-05-19T07:13:18.866785166Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"42fa98abea74b45c079dd9d268bfe6b8"}
{"time":"2026-05-19T07:13:18.88528746Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"42fa98abea74b45c079dd9d268bfe6b8"}

2026-05-19T07:13:24Z 🟠 The test run failed. Resources are NOT deleted yet — held for 15 minutes for debugging (until 2026-05-19T07:28:23Z UTC). Please do NOT rerun this test until cleanup completes. To force immediate cleanup, comment /test cancel.

Result - 🔴 Failure

2026-05-19T07:18:42Z 🔴 Resources cleaned up — test run is finalised. You can rerun the test now.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 19, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.3%. Comparing base (0fa3b4a) to head (a756fcf).

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vaclav-ssvlabs
Copy link
Copy Markdown
Contributor Author

/test cancel

@ssv-test-bot
Copy link
Copy Markdown

ssv-test-bot Bot commented May 19, 2026

🛑 @vaclav-ssvlabs requested cancellation. Forced cleanup on 1 test run(s); resources will be released shortly.

@vaclav-ssvlabs
Copy link
Copy Markdown
Contributor Author

/test extraEnvs=VACLAV_ENV=true,FEATURE_X=1

@ssv-test-bot
Copy link
Copy Markdown

ssv-test-bot Bot commented May 19, 2026

🔴 Test run for PR #2856

Branch: vaclav-test-ci-123
Commit: a756fcf

Status

2026-05-19T07:19:19Z Starting test run
2026-05-19T07:19:20Z Test run submitted successfully for commit a756fcf
2026-05-19T07:19:21Z container image for commit a756fcf327c91aae98dc43078a9ddb499be5e307 is build and present in container registry
2026-05-19T07:19:22Z automatically allocated node range cluster=300,301,302,303 for test run on network=hoodi_stage
2026-05-19T07:19:24Z deploying commit a756fcf327c91aae98dc43078a9ddb499be5e307 to ssv nodes 300,301,302,303
2026-05-19T07:19:26Z commit a756fcf327c91aae98dc43078a9ddb499be5e307 is deployed to ssv nodes 300,301,302,303
2026-05-19T07:19:27Z started test executor with tests=(sanity) and network=hoodi_stage
2026-05-19T07:20:23Z 🟠 The test run failed. Resources are NOT deleted yet — held for 15 minutes for debugging (until 2026-05-19T07:35:23Z UTC). Please do NOT rerun this test until cleanup completes. To force immediate cleanup, comment /test cancel.

Result - 🔴 Failure

Click to expand executor logs
═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\ \/ /| __|/ __|| | | ||_   _|/ _ \ | _ \
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\_\|___|\___| \___/   |_|  \___/ |_|_\
                                        

{"time":"2026-05-19T07:19:31.425383227Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2026-05-19T07:19:31.430998485Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2026-05-19T07:19:31.431013496Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[300,301,302,303],"tests":"(sanity)"}
{"time":"2026-05-19T07:19:31.471792896Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"1d474e255a1bd65603065c022d8e242f"}
{"time":"2026-05-19T07:19:31.472196827Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2026-05-19T07:19:31.472219598Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[300,301,302,303],"test_type":{"Sanity":{"ShouldRun":true,"E2M":false,"Threshold":false,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"Threshold":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2026-05-19T07:19:33.900068019Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2026-05-19T07:19:33.900163291Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2026-05-19T07:19:33.900175663Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2026-05-19T07:19:33.900192367Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":false}
{"time":"2026-05-19T07:19:33.900225151Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2026-05-19T07:19:33.90024125Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2026-05-19T07:19:33.900264805Z","level":"INFO","msg":"Sanity: config values","test":"sanity","cluster_owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","account_public_key":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","operator_ids":[300,301,302,303]}
{"time":"2026-05-19T07:19:33.900338964Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2026-05-19T07:19:33.955576756Z","level":"INFO","msg":"Available slots","test":"sanity","node":300,"slots":3000}
{"time":"2026-05-19T07:19:33.96257608Z","level":"INFO","msg":"Available slots","test":"sanity","node":301,"slots":1008}
{"time":"2026-05-19T07:19:33.969022756Z","level":"INFO","msg":"Available slots","test":"sanity","node":302,"slots":3000}
{"time":"2026-05-19T07:19:33.975532572Z","level":"INFO","msg":"Available slots","test":"sanity","node":303,"slots":1325}
{"time":"2026-05-19T07:20:01.342595134Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-19T07:20:01.34262176Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-19T07:20:05.173641312Z","level":"INFO","msg":"Cluster asset type before write","test":"sanity","asset_type":1,"asset_err":null,"asset_label":"ETH"}
{"time":"2026-05-19T07:20:08.999473503Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"registerValidator","gas_estimate_error":"gas estimation failed: execution reverted","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-19T07:20:09.014067273Z","level":"INFO","msg":"1️⃣ Registering validator","test":"sanity","module":"contract","tx_hash":"0x10b275ed2549a65040caad6e1c75d81ffd52e6a1e6f28e20def64f20a48468f6"}
{"time":"2026-05-19T07:20:09.02447719Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x10b275ed2549a65040caad6e1c75d81ffd52e6a1e6f28e20def64f20a48468f6"}
{"time":"2026-05-19T07:20:19.029338897Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0x10b275ed2549a65040caad6e1c75d81ffd52e6a1e6f28e20def64f20a48468f6"}
{"time":"2026-05-19T07:20:19.029383927Z","level":"ERROR","msg":"❌ FAILURE: Contract Registration Failed - Transaction verification failed","test":"sanity","txHash":"0x10b275ed2549a65040caad6e1c75d81ffd52e6a1e6f28e20def64f20a48468f6"}
{"time":"2026-05-19T07:20:19.029419937Z","level":"ERROR","msg":"❌ FAILURE: Sanity Test Suite Failed","module":"main","error":"sanity register step failed: transaction verification failed"}
{"time":"2026-05-19T07:20:19.029426673Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2026-05-19T07:20:19.029441833Z","level":"ERROR","msg":"❌ FAILURE: Test Execution Completed with Failures","module":"main","passed_suites":null,"failed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2026-05-19T07:20:19.029451102Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"sanity","error":"sanity register step failed: transaction verification failed"}
{"time":"2026-05-19T07:20:19.02945624Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"1d474e255a1bd65603065c022d8e242f"}
{"time":"2026-05-19T07:20:19.033256066Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"1d474e255a1bd65603065c022d8e242f"}
{"time":"2026-05-19T07:20:19.049592022Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"1d474e255a1bd65603065c022d8e242f"}

2026-05-19T07:24:23Z 🔴 Resources cleaned up — test run is finalised. You can rerun the test now.

@vaclav-ssvlabs
Copy link
Copy Markdown
Contributor Author

/test cancel

@ssv-test-bot
Copy link
Copy Markdown

ssv-test-bot Bot commented May 19, 2026

🛑 @vaclav-ssvlabs requested cancellation. Forced cleanup on 1 test run(s); resources will be released shortly.

@vaclav-ssvlabs
Copy link
Copy Markdown
Contributor Author

/test extraEnvs=VACLAV_ENV=12345asd#

@ssv-test-bot
Copy link
Copy Markdown

ssv-test-bot Bot commented May 19, 2026

🔴 Test run for PR #2856

Branch: vaclav-test-ci-123
Commit: a756fcf

Status

2026-05-19T07:32:16Z Starting test run
2026-05-19T07:32:18Z container image for commit a756fcf327c91aae98dc43078a9ddb499be5e307 is build and present in container registry
2026-05-19T07:32:19Z automatically allocated node range cluster=300,301,302,303 for test run on network=hoodi_stage
2026-05-19T07:32:21Z automatically allocated node range cluster=300,301,302,303 for test run on network=hoodi_stage
2026-05-19T07:32:23Z deploying commit a756fcf327c91aae98dc43078a9ddb499be5e307 to ssv nodes 300,301,302,303
2026-05-19T07:33:09Z commit a756fcf327c91aae98dc43078a9ddb499be5e307 is deployed to ssv nodes 300,301,302,303
2026-05-19T07:33:11Z started test executor with tests=(sanity) and network=hoodi_stage
2026-05-19T07:34:09Z 🟠 The test run failed. Resources are NOT deleted yet — held for 15 minutes for debugging (until 2026-05-19T07:49:08Z UTC). Please do NOT rerun this test until cleanup completes. To force immediate cleanup, comment /test cancel.

Result - 🔴 Failure

Click to expand executor logs
{"time":"2026-05-19T07:33:14.565181289Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\ \/ /| __|/ __|| | | ||_   _|/ _ \ | _ \
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\_\|___|\___| \___/   |_|  \___/ |_|_\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2026-05-19T07:33:14.569450087Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2026-05-19T07:33:14.56948029Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[300,301,302,303],"tests":"(sanity)"}
{"time":"2026-05-19T07:33:14.610751556Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"0d85e898730b40cf93db5bd05b39d9ce"}
{"time":"2026-05-19T07:33:14.611255037Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2026-05-19T07:33:14.611273112Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[300,301,302,303],"test_type":{"Sanity":{"ShouldRun":true,"E2M":false,"Threshold":false,"BulkAmount":0},"Bulk":{"ShouldRun":false,"E2M":false,"Threshold":false,"BulkAmount":0},"Isolated":{"Tests":null}}}
{"time":"2026-05-19T07:33:18.397564473Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2026-05-19T07:33:18.397652962Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2026-05-19T07:33:18.397675846Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2026-05-19T07:33:18.397693842Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":false}
{"time":"2026-05-19T07:33:18.397706082Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2026-05-19T07:33:18.397719339Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2026-05-19T07:33:18.397734733Z","level":"INFO","msg":"Sanity: config values","test":"sanity","cluster_owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","account_public_key":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","operator_ids":[300,301,302,303]}
{"time":"2026-05-19T07:33:18.397826854Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2026-05-19T07:33:18.441921517Z","level":"INFO","msg":"Available slots","test":"sanity","node":300,"slots":3000}
{"time":"2026-05-19T07:33:18.44756829Z","level":"INFO","msg":"Available slots","test":"sanity","node":301,"slots":1008}
{"time":"2026-05-19T07:33:18.453189143Z","level":"INFO","msg":"Available slots","test":"sanity","node":302,"slots":3000}
{"time":"2026-05-19T07:33:18.458513802Z","level":"INFO","msg":"Available slots","test":"sanity","node":303,"slots":1325}
{"time":"2026-05-19T07:33:46.408143056Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-19T07:33:46.408164801Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-19T07:33:50.262173497Z","level":"INFO","msg":"Cluster asset type before write","test":"sanity","asset_type":1,"asset_err":null,"asset_label":"ETH"}
{"time":"2026-05-19T07:33:54.300641268Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"registerValidator","gas_estimate_error":"gas estimation failed: execution reverted","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-19T07:33:54.312273644Z","level":"INFO","msg":"1️⃣ Registering validator","test":"sanity","module":"contract","tx_hash":"0xddac05bebd0ecbf7cd0000ab3d70de8e0fb1b81cdc3bfe0ec839150fff1b053a"}
{"time":"2026-05-19T07:33:54.32354073Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xddac05bebd0ecbf7cd0000ab3d70de8e0fb1b81cdc3bfe0ec839150fff1b053a"}
{"time":"2026-05-19T07:34:04.33351703Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xddac05bebd0ecbf7cd0000ab3d70de8e0fb1b81cdc3bfe0ec839150fff1b053a"}
{"time":"2026-05-19T07:34:04.333586322Z","level":"ERROR","msg":"❌ FAILURE: Contract Registration Failed - Transaction verification failed","test":"sanity","txHash":"0xddac05bebd0ecbf7cd0000ab3d70de8e0fb1b81cdc3bfe0ec839150fff1b053a"}
{"time":"2026-05-19T07:34:04.333629133Z","level":"ERROR","msg":"❌ FAILURE: Sanity Test Suite Failed","module":"main","error":"sanity register step failed: transaction verification failed"}
{"time":"2026-05-19T07:34:04.333636704Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2026-05-19T07:34:04.333710465Z","level":"ERROR","msg":"❌ FAILURE: Test Execution Completed with Failures","module":"main","passed_suites":null,"failed_suites":["sanity"],"skipped_suites":["isolated"]}
{"time":"2026-05-19T07:34:04.333721592Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"sanity","error":"sanity register step failed: transaction verification failed"}
{"time":"2026-05-19T07:34:04.33372667Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"0d85e898730b40cf93db5bd05b39d9ce"}
{"time":"2026-05-19T07:34:04.336430936Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"0d85e898730b40cf93db5bd05b39d9ce"}
{"time":"2026-05-19T07:34:04.35308093Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"0d85e898730b40cf93db5bd05b39d9ce"}

2026-05-19T07:36:03Z 🔴 Resources cleaned up — test run is finalised. You can rerun the test now.

@vaclav-ssvlabs
Copy link
Copy Markdown
Contributor Author

/test cancel

@ssv-test-bot
Copy link
Copy Markdown

ssv-test-bot Bot commented May 19, 2026

🛑 @vaclav-ssvlabs requested cancellation. Forced cleanup on 1 test run(s); resources will be released shortly.

@vaclav-ssvlabs
Copy link
Copy Markdown
Contributor Author

/test extraEnvs=VACLAV_ENV=12345asd#

@ssv-test-bot
Copy link
Copy Markdown

ssv-test-bot Bot commented May 19, 2026

🔴 Test run for PR #2856

Branch: vaclav-test-ci-123
Commit: a756fcf

Status

2026-05-19T07:57:57Z Starting test run
2026-05-19T07:57:58Z Test run submitted successfully for commit a756fcf
2026-05-19T07:57:59Z container image for commit a756fcf327c91aae98dc43078a9ddb499be5e307 is build and present in container registry
2026-05-19T07:58:00Z automatically allocated node range cluster=300,301,302,303 for test run on network=hoodi_stage
2026-05-19T07:58:02Z deploying commit a756fcf327c91aae98dc43078a9ddb499be5e307 to ssv nodes 300,301,302,303
2026-05-19T07:59:19Z commit a756fcf327c91aae98dc43078a9ddb499be5e307 is deployed to ssv nodes 300,301,302,303
2026-05-19T07:59:20Z started test executor with tests=(sanity,e2m),(bulk,e2m,5) and network=hoodi_stage

Click to expand executor logs
═══════════════════════════════════════════════════════════════════════════════════════════════════════
                🧪  WELCOME TO Executor - SSV's Distributed Test Automation Execution Layer  🧪            

                                  ___ __  __ ___  ___  _   _  _____  ___   ___ 
                                 | __|\\ \\/ /| __|/ __|| | | ||_   _|/ _ \\ | _ \\
                                 | _|  >  < | _|| (__ | |_| |  | | | (_) ||   /
                                 |___|/_/\\_\\|___|\\___| \\___/   |_|  \\___/ |_|_\\
                                        

═══════════════════════════════════════════════════════════════════════════════════════════════════════
{"time":"2026-05-19T07:59:57.31847021Z","level":"INFO","msg":"🐳 Docker mode: Using environment variables for config generation"}
{"time":"2026-05-19T07:59:57.323888265Z","level":"INFO","msg":"🌐 Orchestrator API Client initialized","endpoint":"http://aetheria-orchestrator.aetheria.svc:8080"}
{"time":"2026-05-19T07:59:57.323904694Z","level":"INFO","msg":"📝 Requesting configuration generation from orchestrator API","network":"hoodi_stage","clusters":[300,301,302,303],"tests":"(sanity,e2m),(bulk,e2m,5)"}
{"time":"2026-05-19T07:59:57.370846269Z","level":"INFO","msg":"✅ Configuration generated successfully via API","resource_usage_id":"fbceae1b7c1d743fa27508050da98bcb"}
{"time":"2026-05-19T07:59:57.371133969Z","level":"INFO","msg":"🚀 EXECUTOR Initialized Successfully"}
{"time":"2026-05-19T07:59:57.371145741Z","level":"INFO","msg":"⚙️ Test configuration loaded","network":"hoodi_stage","nodes":[300,301,302,303],"test_type":{"Sanity":{"ShouldRun":true,"E2M":true,"Threshold":false,"BulkAmount":0},"Bulk":{"ShouldRun":true,"E2M":true,"Threshold":false,"BulkAmount":5},"Isolated":{"Tests":null}}}
{"time":"2026-05-19T07:59:59.694846706Z","level":"INFO","msg":"Generated operator pubkeys file","module":"scanner","path":"/app/shared/data/operators/operator-pubkeys-hoodi_stage.json"}
{"time":"2026-05-19T07:59:59.694904657Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"consensus"}
{"time":"2026-05-19T07:59:59.694911928Z","level":"INFO","msg":"Registered parser","module":"main","component":"parser-client","parser_type":"flow-emits"}
{"time":"2026-05-19T07:59:59.694924002Z","level":"INFO","msg":"Validation client initialized","module":"main","loki_enabled":true,"e2m_enabled":true}
{"time":"2026-05-19T07:59:59.694932274Z","level":"INFO","msg":"🔑 KEY STEP: Starting Isolated Tests Concurrently if exists...","module":"main"}
{"time":"2026-05-19T07:59:59.694941919Z","level":"INFO","msg":"🔑 KEY STEP: Starting Sanity Test Suite","module":"main"}
{"time":"2026-05-19T07:59:59.694956799Z","level":"INFO","msg":"Sanity: config values","test":"sanity","cluster_owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","account_public_key":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","operator_ids":[300,301,302,303]}
{"time":"2026-05-19T07:59:59.694996511Z","level":"INFO","msg":"No isolated tests configured, skipping","module":"main"}
{"time":"2026-05-19T07:59:59.748041974Z","level":"INFO","msg":"Available slots","test":"sanity","node":300,"slots":3000}
{"time":"2026-05-19T07:59:59.755539338Z","level":"INFO","msg":"Available slots","test":"sanity","node":301,"slots":1008}
{"time":"2026-05-19T07:59:59.763948626Z","level":"INFO","msg":"Available slots","test":"sanity","node":302,"slots":3000}
{"time":"2026-05-19T07:59:59.770480842Z","level":"INFO","msg":"Available slots","test":"sanity","node":303,"slots":1325}
{"time":"2026-05-19T08:00:26.781605965Z","level":"INFO","msg":"Extracted keyshares path","test":"sanity","module":"keys","path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-19T08:00:26.781628431Z","level":"INFO","msg":"Key shares file generated successfully","test":"sanity","keyshare_path":"/app/shared/data/keyshares/0x83142d846a6b78759863175ff9ad7890bd47a1a534b63f3949080484082164adde97a829291e6dc0c5099d82657bf754.json"}
{"time":"2026-05-19T08:00:30.705720224Z","level":"INFO","msg":"Cluster asset type before write","test":"sanity","asset_type":1,"asset_err":null,"asset_label":"ETH"}
{"time":"2026-05-19T08:00:34.606516758Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"sanity","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"registerValidator","gas_estimate_error":"gas estimation failed: execution reverted: 0x95a0cf33","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-19T08:00:34.618818624Z","level":"INFO","msg":"1️⃣ Registering validator","test":"sanity","module":"contract","tx_hash":"0xf2721677e3945dccca9a4c4ad24813d3296a561dad512ee0e7835c9df529711a"}
{"time":"2026-05-19T08:00:34.628365252Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"sanity","module":"contract","attempt":1,"sleep":10,"tx_hash":"0xf2721677e3945dccca9a4c4ad24813d3296a561dad512ee0e7835c9df529711a"}
{"time":"2026-05-19T08:00:44.640704369Z","level":"INFO","msg":"Transaction receipt received","test":"sanity","module":"contract","tx_hash":"0xf2721677e3945dccca9a4c4ad24813d3296a561dad512ee0e7835c9df529711a"}
{"time":"2026-05-19T08:00:44.640745776Z","level":"ERROR","msg":"❌ FAILURE: Contract Registration Failed - Transaction verification failed","test":"sanity","txHash":"0xf2721677e3945dccca9a4c4ad24813d3296a561dad512ee0e7835c9df529711a"}
{"time":"2026-05-19T08:00:44.640776983Z","level":"ERROR","msg":"❌ FAILURE: Sanity Test Suite Failed","module":"main","error":"sanity register step failed: transaction verification failed"}
{"time":"2026-05-19T08:00:44.640784456Z","level":"INFO","msg":"🔑 KEY STEP: Starting Bulk Test Suite","module":"main"}
{"time":"2026-05-19T08:00:44.640803488Z","level":"INFO","msg":"Bulk: config values","test":"bulk","cluster_owner":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","account_public_key":"0x91E32eFb8139cd88caE0Df30d2Bf471294c6ed27","operator_ids":[300,301,302,303]}
{"time":"2026-05-19T08:00:44.670558805Z","level":"INFO","msg":"Available slots","test":"bulk","node":300,"slots":3000}
{"time":"2026-05-19T08:00:44.679051994Z","level":"INFO","msg":"Available slots","test":"bulk","node":301,"slots":1008}
{"time":"2026-05-19T08:00:44.685033625Z","level":"INFO","msg":"Available slots","test":"bulk","node":302,"slots":3000}
{"time":"2026-05-19T08:00:44.691458037Z","level":"INFO","msg":"Available slots","test":"bulk","node":303,"slots":1325}
{"time":"2026-05-19T08:01:59.783067545Z","level":"INFO","msg":"Extracted keyshares path for bulk keyshare","test":"bulk","module":"keys","path":"/app/shared/data/keyshares/keyshares-1779177719.json"}
{"time":"2026-05-19T08:02:03.576028331Z","level":"INFO","msg":"Cluster asset type before write","test":"bulk","asset_type":1,"asset_err":null,"asset_label":"ETH"}
{"time":"2026-05-19T08:02:10.808536409Z","level":"WARN","msg":"📝 tx gas limit set from fallback (estimate failed or zero; buffer applied)","test":"bulk","module":"contract","contract":"0xc07B3E9671f884FDa67E1e7D43d952E0e1369fd8","method":"bulkRegisterValidator","gas_estimate_error":"gas estimation failed: execution reverted: 0x95a0cf33","fallback_gas":350000,"gas_buffer_pct":1000,"gas_limit":3850000}
{"time":"2026-05-19T08:02:10.828294807Z","level":"INFO","msg":"1️⃣ Bulk Registering validators","test":"bulk","module":"contract","tx_hash":"0x3e7f699eb4be3487cdb27ae21497448000178daedcc535662cb13b211721409f"}
{"time":"2026-05-19T08:02:10.834294585Z","level":"INFO","msg":"Bulk registration transaction sent","test":"bulk","txHash":"0x3e7f699eb4be3487cdb27ae21497448000178daedcc535662cb13b211721409f"}
{"time":"2026-05-19T08:02:10.837370998Z","level":"INFO","msg":"Waiting for transaction receipt, retrying...","test":"bulk","module":"contract","attempt":1,"sleep":10,"tx_hash":"0x3e7f699eb4be3487cdb27ae21497448000178daedcc535662cb13b211721409f"}
{"time":"2026-05-19T08:02:20.842087704Z","level":"INFO","msg":"Transaction receipt received","test":"bulk","module":"contract","tx_hash":"0x3e7f699eb4be3487cdb27ae21497448000178daedcc535662cb13b211721409f"}
{"time":"2026-05-19T08:02:20.842141999Z","level":"ERROR","msg":"❌ FAILURE: Bulk Test Suite Failed","module":"main","error":"bulk register step failed: transaction verification failed"}
{"time":"2026-05-19T08:02:20.842148602Z","level":"INFO","msg":"⏳ PROGRESS: Waiting for all tests to finish","module":"main"}
{"time":"2026-05-19T08:02:20.842172203Z","level":"ERROR","msg":"❌ FAILURE: Test Execution Completed with Failures","module":"main","passed_suites":null,"failed_suites":["sanity","bulk"],"skipped_suites":["isolated"]}
{"time":"2026-05-19T08:02:20.842183088Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"sanity","error":"sanity register step failed: transaction verification failed"}
{"time":"2026-05-19T08:02:20.842189927Z","level":"ERROR","msg":"❌ FAILURE: Suite Failure Details","module":"main","suite":"bulk","error":"bulk register step failed: transaction verification failed"}
{"time":"2026-05-19T08:02:20.842195341Z","level":"INFO","msg":"🔑 KEY STEP: Starting resource teardown...","module":"main","usage_id":"fbceae1b7c1d743fa27508050da98bcb"}
{"time":"2026-05-19T08:02:20.846307381Z","level":"INFO","msg":"🧹 Requesting resource teardown","module":"main","usage_id":"fbceae1b7c1d743fa27508050da98bcb"}
{"time":"2026-05-19T08:02:20.873133982Z","level":"INFO","msg":"✅ SUCCESS: Completed resource teardown successfully","module":"main","usage_id":"fbceae1b7c1d743fa27508050da98bcb"}

2026-05-19T08:02:27Z 🟠 The test run failed. Resources are NOT deleted yet — held for 15 minutes for debugging (until 2026-05-19T08:17:26Z UTC). Please do NOT rerun this test until cleanup completes. To force immediate cleanup, comment /test cancel.

Result - 🔴 Failure

2026-05-19T08:17:32Z 🔴 Resources cleaned up — test run is finalised. You can rerun the test now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant