Investigate staging Dealbot startup failure

[silent-cipher]

Dealbot pods are failing to start due to a contract call revert during initialization. As a result, the worker pods are also unable to start successfully.

Failure Log:

{"level":"fatal","pid":14,"timestamp":1774938593839,"message":{"event":"bootstrap_failed","message":"Bootstrap failed","error":{"type":"error","name":"ContractFunctionExecutionError","message":"The contract function \"getClientDataSets\" reverted with the following reason:\nmessage execution failed (exit=[SysErrOutOfGas(7)], vm error=[unknown error (RetCode=7)])\n\nContract Call:\n  address:   0x537320bd004a7FDd3c1932ca64BD88268301322A\n  function:  getClientDataSets(address client)\n  args:                       (0xa5F90bc2AA73a2E0Bad4D7092a932644d5dD5d71)\n\nDocs: https://viem.sh/docs/contract/readContract\nDetails: message execution failed (exit=[SysErrOutOfGas(7)], vm error=[unknown error (RetCode=7)])\nVersion: viem@2.47.5","stack":"ContractFunctionExecutionError: The contract function \"getClientDataSets\" reverted with the following reason:\nmessage execution failed (exit=[SysErrOutOfGas(7)], vm error=[unknown error (RetCode=7)])\n\nContract Call:\n  address:   0x537320bd004a7FDd3c1932ca64BD88268301322A\n  function:  getClientDataSets(address client)\n  args:                       (0xa5F90bc2AA73a2E0Bad4D7092a932644d5dD5d71)\n\nDocs: https://viem.sh/docs/contract/readContract\nDetails: message execution failed (exit=[SysErrOutOfGas(7)], vm error=[unknown error (RetCode=7)])\nVersion: viem@2.47.5\n    at getContractError (file:///app/node_modules/.pnpm/viem@2.47.5_typescript@5.9.3_zod@4.3.6/node_modules/viem/_esm/utils/errors/getContractError.js:34:12)\n    at readContract (file:///app/node_modules/.pnpm/viem@2.47.5_typescript@5.9.3_zod@4.3.6/node_modules/viem/_esm/actions/public/readContract.js:58:15)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async getClientDataSets (file:///app/node_modules/.pnpm/@filoz+synapse-core@0.3.2_typescript@5.9.3_viem@2.47.5_typescript@5.9.3_zod@4.3.6_/node_modules/@filoz/synapse-core/dist/src/warm-storage/get-client-data-sets.js:4:18)\n    at async getPdpDataSets (file:///app/node_modules/.pnpm/@filoz+synapse-core@0.3.2_typescript@5.9.3_viem@2.47.5_typescript@5.9.3_zod@4.3.6_/node_modules/@filoz/synapse-core/dist/src/warm-storage/get-pdp-data-sets.js:4:18)\n    at async Promise.all (index 2)\n    at async fetchProviderSelectionInput (file:///app/node_modules/.pnpm/@filoz+synapse-core@0.3.2_typescript@5.9.3_viem@2.47.5_typescript@5.9.3_zod@4.3.6_/node_modules/@filoz/synapse-core/dist/src/warm-storage/fetch-provider-selection-input.js:5:51)\n    at async StorageContext.smartSelect (file:///app/node_modules/.pnpm/@filoz+synapse-sdk@0.40.1_typescript@5.9.3_viem@2.47.5_typescript@5.9.3_zod@4.3.6_/node_modules/@filoz/synapse-sdk/dist/src/storage/context.js:253:23)\n    at async StorageContext.createContexts (file:///app/node_modules/.pnpm/@filoz+synapse-sdk@0.40.1_typescript@5.9.3_viem@2.47.5_typescript@5.9.3_zod@4.3.6_/node_modules/@filoz/synapse-sdk/dist/src/storage/context.js:106:27)\n    at async StorageManager.createContexts (file:///app/node_modules/.pnpm/@filoz+synapse-sdk@0.40.1_typescript@5.9.3_viem@2.47.5_typescript@5.9.3_zod@4.3.6_/node_modules/@filoz/synapse-sdk/dist/src/storage/manager.js:509:26)"}},"context":"Main"}

[rjan90]

It is related to lack of USDFC funding on the wallet https://filfox.info/en/address/f410fux4qxqvkoorobowu24esvezgitk52xlrkuu5ttq

[rjan90]

I´m working on topping up the wallet now. But had to go and buy some USDC as I was lacking funds in my wallet

[silent-cipher]

Looks like it has enough funds on calibration

[rjan90]

Yeah, it might be a different cause on calibnet. At least on Mainnet that was one of the suspected causes: https://filecoinproject.slack.com/archives/C08TVNKJV7C/p1774898885414449?thread_ts=1774499389.477669&cid=C08TVNKJV7C

[silent-cipher]

The reason might be the number of datasets dealbot has on calibration - ~750.

[rjan90]

The reason might be the number of datasets dealbot has on calibration - ~750.

Yeah, that makes sense. Since we probably have been creating a lot of datasets due to: #402 (comment). And Calibnet has had a ton USDFC / FIL available so we did not drain the wallet similar to on Mainnet

[beck-8]

The problem may not be here

filecoin-pin data-set ls --view-address 0xa5F90bc2AA73a2E0Bad4D7092a932644d5dD5d71

There is no problem with this command at all, indicating that there is no problem with the number of data sets (I have a large number of data sets, and this command cannot be run).

I remember ravgg also said that the number of data sets should not be a problem.

[beck-8]

FilOzone/synapse-sdk#702

This may be the problem. So solving 429 will also restore it?

[silent-cipher]

This may be the problem. So solving 429 will also restore it?

I don’t think these are directly related. The failure we’re seeing is due to the getClientDataSets call on the FWSS state view contract reverting, likely because it’s hitting gas limits.
So this appears to be separate from the 429 issue.

[beck-8]

I tested it above, if the test method is correct. The gas limit is not reached

[silent-cipher]

This filecoin-pin command :

filecoin-pin data-set ls --view-address 0xa5F90bc2AA73a2E0Bad4D7092a932644d5dD5d71

This calls clientDataSets call on FWSS instead of getClientDataSets which runs out of gas.

[silent-cipher]

Found the origin of the failure here:

dealbot/apps/backend/src/wallet-sdk/wallet-sdk.service.ts

Line 304 in 59e5c93

const { costs, transaction } = await this.storageManager.prepare({

I’ll look into alternatives to ensureWalletAllowances.