ceac: skip peer-cache rows outside the verdict lookback window#540
Open
catoneone wants to merge 1 commit into
Open
ceac: skip peer-cache rows outside the verdict lookback window#540catoneone wants to merge 1 commit into
catoneone wants to merge 1 commit into
Conversation
scores_index has no TTL, so the table grows unbounded as miners register/dereg/upload new revisions — observed: 653 rows / ~2.6 GB of R2 pulls on every refresh-service restart even though >70% are older than verdict_lookback_days and can never influence a verdict. _pick_origin already filters peers older than the same window during the verdict math, so the R2 work was pure overhead. Compute peer_fb_floor = min(pending.first_block) - lookback_blocks once per tick and skip rows whose first_block is below it before issuing the R2 fetch. Earliest-pending anchor keeps the floor valid for every pending candidate this tick will judge. lookback_days=0 disables the filter (matches the existing 'all rows always loaded' behaviour expected by the older fixtures). Logged at INFO: peer cache size + how many rows were skipped, so an operator can sanity-check the trim depth across a deploy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
`scores_index` has no TTL — it accumulates rows for every miner
(hotkey, revision) that ever ran through anti-copy, including
deregistered ones. Observed on the validator at the time of writing:
restart
`_pick_origin` already filters them out during the verdict math
10-15+ minutes on a large table, blocking the first batch of
verdicts behind it.
What
Compute a single floor up front:
```
peer_fb_floor = min(pending.first_block) - lookback_blocks
```
`min()` (not `max()`) anchors off the earliest pending candidate, so
the resulting floor is valid for every pending row this tick will
judge. Rows whose `first_block` is below `peer_fb_floor` are skipped
before the R2 fetch. The log line now reports both the cache size and
the skip count so trim depth is visible per tick:
```
[anticopy.verdict] peer cache built: 192 blobs (skipped 461 outside 7d lookback)
```
`verdict_lookback_days=0` keeps the original "load everything"
behaviour intact (validated by a new test alongside the affirmative
case).
Test plan
(5 existing + 2 new: skip-outside-window + lookback-zero-disables-filter).
(skipped Y outside Nd lookback)` log appears with X+Y == `list_all`
size, and that the tick completes proportionally faster than the
pre-PR 2.6 GB pull.
Notes
Doesn't touch the actual verdict math or row-level decision criteria —
strictly an I/O / memory optimisation for the cache-build step. The
verdict outcomes are identical (the rows we now skip were already
being filtered out by `_pick_origin`'s own lookback check).