Chaind downloading old blocks (ignoring blocks.start-slot parameter)

[got3nks]

My goal is to run chaind with only recent data (i.e. last 24/48 hours).

I started chaind sync with blocks.start-slot: 5790000. At first it worked, downloading only blocks after the specified slot.

I then removed blocks.start-slot from the configuration (as suggested, to avoid forced re-scans at future restart of the process), few moments later t_blocks table catched up with the chain tip.

{"level":"info","service":"blocks","impl":"standard","time":"2023-02-13T21:35:32+01:00","message":"Caught up"}

But now it is downloading old blocks data (and full history from slot 0 of t_beacon_committees which is still catching up with the chain tip, and t_proposer_duties which didn't take a very long time).

SELECT * FROM t_metadata 
"schema"	"{""version"": 11}"
"synccommittees.standard"	"{""latest_period"": 707}"
"proposerduties.standard"	"{""latest_epoch"": 181177}"
"finalizer.standard"	"{""latest_epoch"": 75471, ""latest_canonical_slot"": 2415072}"
"beaconcommittees.standard"	"{""latest_epoch"": 162866}"
"validators.standard"	"{""latest_epoch"": 181177, ""latest_balances_epoch"": 0}"
"blocks.standard"	"{""latest_slot"": 5797407}"

SELECT count(f_slot) FROM t_blocks WHERE f_slot < 5790000
2388518

Current configuration file chaind.yml:

log-level: debug
log-file: /home/chaind/chaind.log
chaindb:
  url: postgres://chain:xxx@localhost:5432
  max-connections: 16
eth2client:
  log-level: debug
  address: localhost:5052
eth1client:
  address: localhost:8545
blocks:
  enable: true
  address: localhost:5052
  # start-slot: 5790000
validators:
  enable: true
  balances:
    enable: false
beacon-committees:
  enable: true
proposer-duties:
  enable: true
finalizer:
  enable: true
eth1deposits:
  enable: false
summarizer:
  enable: false

[got3nks]

From my understading it's the finalizer module fetching old data. Is it possible to let it start from a specific slot (same as blocks.start-slot)?

I tried restarting the chaind process several times and querying the database I noticed this pattern:

• At first t_blocks table is updated with new blocks data, till the current beacon chain tip is reached.
• Then, t_blocks backlog gets fullfilled from where it left from the previous run. New blocks don't get added (neither they get finalized / f_canonical stays null), I guess until all old blocks (canonical) are added.

The log confirms the theory (slot 5799194 was mined just a moment ago):

{"level":"debug","service":"blocks","impl":"standard","slot":5799194,"block_root":"0x7022541e826b60ae1e912eaf832eb2a6c50c6a90a42ae9d06f39e8ca46927f74","time":"2023-02-14T23:39:12+01:00","message":"Another handler (either blocks or finalizer) running"}

[got3nks]

As a confirmation, I tried to run a new node with same configuration except disabling the finalizer and the old blocks don't get downloaded.

Both beaconcommittees.standard and proposerduties.standard still download all history from slot 0, but they don't take as much space as blocks and attestations.

Would be nice to have the possibility to enable the finalizer only on slots newer than blocks.start-slot.

[mcdee]

I have spent some time looking at this, however blocks.start-slot was never meant to provide a barrier for data gathering, and doing so would be a relatively large endeavor. I will leave this issue open to look at how this could be achieved, but it won't be any time soon I'm afraid.

[got3nks]

As a workaround, I played with the t_metadata table.

INSERT INTO t_metadata (f_key, f_value) VALUES ('finalizer.standard', '{"latest_epoch": 182345, "latest_canonical_slot": 5835040}');

Then I stopped chaind, enabled the finalizer module in the yml configuration and restarted chaind. At first I got some errors:

"message":"Failed to update attestations on finality; will retry next finality update"

But after updating manually the latest_epoch as soon as there were new finalized blocks in t_blocks and restarting chaind it seems to be working without downloading any old blocks data.

[mentheosis]

I have had similar issues, even when disabling finalizer service all services seem to start at genesis regardless of the start-slot or start-epoch params I passed. I also tried manually updating the metadata table as mentioned above and that seemed to work for some services, but blocks and summarizer now appear stuck per this other issue: #83

[got3nks]

The first time you start chaind the blocks.start-slot parameter is ignored because of DB schema creation (it's fixed in current master, but if you are using 0.7.0 binaries this is most likely the reason).
You should let it create the database (wait a few seconds), then kill the process and restart it.

As for the metadata table manual changes and issues thereafter I don't know. Luckily it worked for me (don't know if it can make any difference, but I waited for beaconcommittees and proposerduties to be fully sync'd before changing the metadata values). But for sure we are not supposed to change these value manually.