tbc: store tx byte location in tx index, bump DB to v6#1052
Open
marcopeereboom wants to merge 1 commit into
Open
tbc: store tx byte location in tx index, bump DB to v6#1052marcopeereboom wants to merge 1 commit into
marcopeereboom wants to merge 1 commit into
Conversation
marcopeereboom
force-pushed
the
marco/lazy-block
branch
from
9f1a394 to
99d768f
Compare
marcopeereboom
force-pushed
the
marco/tx-offset
branch
from
461b684 to
4a29640
Compare
This was referenced May 29, 2026
joshuasing
reviewed
May 29, 2026
marcopeereboom
force-pushed
the
marco/lazy-block
branch
from
99d768f to
e103e96
Compare
marcopeereboom
force-pushed
the
marco/tx-offset
branch
2 times, most recently
from
1a04a83 to
a97fdbd
Compare
Contributor
Author
|
Don't merge yet! |
marcopeereboom
force-pushed
the
marco/lazy-block
branch
2 times, most recently
from
1dba573 to
ffd4579
Compare
marcopeereboom
force-pushed
the
marco/tx-offset
branch
2 times, most recently
from
720c095 to
4d15662
Compare
github-actions
Bot
added
area: hemictl
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
ClaytonNorthey92
requested changes
Jun 11, 2026
marcopeereboom
force-pushed
the
marco/tx-offset
branch
from
4d15662 to
c56d69f
Compare
ClaytonNorthey92
approved these changes
Jun 15, 2026
Store TxLoc (offset + length within raw block) in the t entry value instead of nil. This allows callers to jump directly to a tx's bytes in the raw block without scanning — O(1) instead of O(txs_in_block). BlockHashByTxId now returns (*chainhash.Hash, wire.TxLoc, error). All callers updated. No separate method needed — callers that only need the hash use bh, _, err := BlockHashByTxId(...). processTxs calls block.TxLoc() and stores the location via NewTxMappingWithLoc. Errors from TxLoc() are logged at Errorf and the indexer falls back to nil values (legacy format). BlockTxUpdate uses stack-allocated reusable buffers instead of slicing loop variables. The previous code sliced the range variable and passed the slice to leveldb.Batch.Put. appendRec copies immediately, but the interaction between range variable reuse, map deletion, and GC is not guaranteed safe. Stack buffers are zero-alloc and independent per iteration. DB version 5 -> 6. Upgrade path wipes the transactions index for rebuild with TxLoc values. The index is fully derived from block data. Ref: #1050
marcopeereboom
force-pushed
the
marco/tx-offset
branch
from
c56d69f to
ad5bf1d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Store tx byte location (
TxLoc: offset + length within raw block) in the tx index't'entry value, which was previously nil. This enables O(1) tx lookup by jumping directly to the tx's bytes in the raw block — no scanning, no SHA256 hashing, no full block deserialization.Depends on #1051 (lazy block reader).
Problem
Every tx lookup via
BlockHashByTxIdrequires a subsequent full block deserialization to find the tx. The't'entry value was nil — wasted space that could carry the byte offset. CPU profile shows 60% in SHA256 hashing fromFindTxscanning every tx in the block to find a match.Solution
BlockHashByTxIdsignature changed to return(*chainhash.Hash, wire.TxLoc, error)— all callers updatedprocessTxsnow callsblock.TxLoc()and stores offset+length viaNewTxMappingWithLocBlockTxUpdateuses stack-allocated reusable buffers instead of slicing loop variables (addresses potential data integrity issue documented in tbc: tx index intermittently loses entries during IBD #1050)TxById(RPC) — deserializes only the target txtxOutFromOutPoint(UTXO unwind) — deserializes only the target txhandleBlockHashByTxIdRequest(RPC) — hash only, ignores TxLochemictl— hash only, ignores TxLocblock.TxLoc()logged at Errorf, falls back to nil valuesTesting
TestDbUpgradeV6— seeds v5 DB, runs upgrade, verifies index wiped and version bumpedTestTxLocRoundTrip— stores TxLoc, reads back, verifies offset+length matchTestTxLocOffsetCorrectness— stores raw block, uses offset to extract tx bytes, deserializes, verifies txid and output values matchTestTxLocLegacyNilValue— nil-value entry returns zero TxLoc gracefullyImpact
With TxLoc, each cache miss costs: 1 LevelDB read (tx index) + 1 LevelDB/cache read (raw block) + parse ~200 bytes of one tx. No full block deserialization. No SHA256 scanning. Eliminates the need for parallel lookup strategies.
Related
Files changed
database/tbcd/database.go—BlockHashByTxIdreturns TxLoc,NewTxMappingWithLocdatabase/tbcd/level/level.go— implementation, stack buffers, DB v6database/tbcd/level/level_test.go— 4 new testsdatabase/tbcd/level/upgrade.go—v6()wipes tx indexservice/tbc/txindex.go— stores TxLocservice/tbc/tbc.go—TxByIduses TxLocservice/tbc/utxoindex.go—txOutFromOutPointuses TxLocservice/tbc/rpc.go— caller updatedservice/tbc/cpfp_test.go— stub updatedservice/tbc/tbc_test.go— version expectations updatedcmd/hemictl/hemictl.go— caller updated