Kernel api reindexer #28
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As a first step, implement the equivalent of what was implemented in the now deprecated libbitcoinconsensus header. Also add a test binary to exercise the header and library. Unlike the deprecated libbitcoinconsensus the kernel library can now use the hardware-accelerated sha256 implementations thanks for its statically-initialzed context. The functions kept around for backwards-compatibility in the libbitcoinconsensus header are not ported over. As a new header, it should not be burdened by previous implementations. Also add a new error code for handling invalid flag combinations, which would otherwise cause a crash. The macros used in the new C header were adapted from the libsecp256k1 header. To make use of the C header from C++ code, a C++ header is also introduced for wrapping the C header. This makes it safer and easier to use from C++ code. Co-authored-by: stickies-v <stickies-v@protonmail.com>
Exposing logging in the kernel library allows users to follow operations. Users of the C header can use `kernel_logging_connection_create(...)` to pass a callback function to Bitcoin Core's internal logger. Additionally the level and category can be globally configured. By default, the logger buffers messages until `kernel_loggin_connection_create(...)` is called. If the user does not want any logging messages, it is recommended that `kernel_disable_logging()` is called, which permanently disables the logging and any buffering of messages.
The context introduced here holds the objects that will be required for running validation tasks, such as the chosen chain parameters, callbacks for validation events, and interrupt handling. These will be used by the chainstate manager introduced in subsequent commits. This commit also introduces conventions for defining option objects. A common pattern throughout the C header will be: ``` options = object_option_create(); object = object_create(options); ``` This allows for more consistent usage of a "builder pattern" for objects where options can be configured independently from instantiation.
As a first option, add the chainparams. For now these can only be instantiated with default values. In future they may be expanded to take their own options for regtest and signet configurations. This commit also introduces a unique pattern for setting the option values when calling the `*_set(...)` function.
The notifications are used for notifying on connected blocks and on warning and fatal error conditions. The user of the C header may define callbacks that gets passed to the internal notification object in the `kernel_NotificationInterfaceCallbacks` struct. Each of the callbacks take a `user_data` argument that gets populated from the `user_data` value in the struct. It can be used to recreate the structure containing the callbacks on the user's side, or to give the callbacks additional contextual information.
This is the main driver class for anything validation related, so expose it here. Creating the chainstate manager options will currently also trigger the creation of their respectively configured directories. The chainstate manager and block manager options are consolidated into a single object. The kernel might eventually introduce a separate block manager object for the purposes of being a light-weight block store reader. The chainstate manager will associate with the context with which it was created for the duration of its lifetime. It is only valid if that context remains in memory too. The tests now also create dedicated temporary directories. This is similar to the behaviour in the existing unit test framework. Co-authored-by: stickies-v <stickies-v@protonmail.com>
Re-use the same pattern used for the context options. This allows users to set the number of threads used in the validation thread pool.
The library will now internally load the chainstate when a new ChainstateManager is instantiated. Options for controlling details of loading the chainstate will be added over the next few commits.
The added function allows the user process and validate a given block with the chainstate manager. The *_process_block(...) function does some preliminary checks on the block before passing it to `ProcessNewBlock(...)`. These are similar to the checks in the `submitblock()` rpc. Richer processing of the block validation result will be made available in the following commits through the validation interface. The commits also adds a utility for deserializing a `CBlock` (`kernel_block_create()`) that may then be passed to the library for processing. The tests exercise the function for both mainnet and regtest. The commit also adds the data of 206 regtest blocks (some blocks also contain transactions).
Adds options for wiping the chainstate and block tree indexes to the chainstate load options. In combination and once the `*_import_blocks(...)` function is added in a later commit, this triggers a reindex. For now, it just wipes the existing data.
This allows a user to run the kernel without creating on-disk files for the block tree and chainstate indexes. This is potentially useful in scenarios where the user needs to do some ephemeral validation operations. One specific use case is when linearizing the blocks on disk. The block files store blocks out of order, so a program may utilize the library and its header to read the blocks with one chainstate manager, and then write them back in order, and without orphans, with another chainstate maanger. To save disk resources and if the indexes are not required once done, it may be beneficial to keep the indexes in memory for the chainstate manager that writes the blocks back again.
Add `btck_import_blocks` to import block data and rebuild indexes. The function can either reindex all existing block files if the indexes were previously wiped through the chainstate load options, or import blocks from a single specified file path.
Calling interrupt can halt long-running functions associated with objects that were created through the passed-in context.
This adds the infrastructure required to process validation events. For
now the external validation interface only has support for the
`BlockChecked` , `NewPoWValidBlock`, `BlockConnected`, and
`BlockDisconnected` callback. Support for the other internal
validation interface methods can be added in the future.
The validation interface follows an architecture for defining its
callbacks and ownership that is similar to the notifications.
The task runner is created internally with a context, which itself
internally creates a unique ValidationSignals object. When the user
creates a new chainstate manager the validation signals are internally
passed to the chainstate manager through the context.
A validation interface can register for validation events with a
context. Internally the passed in validation interface is registerd with
the validation signals of a context.
The callbacks block any further validation execution when they are
called. It is up to the user to either multiplex them, or use them
otherwise in a multithreaded mechanism to make processing the validation
events non-blocking.
I.e. for a synchronous mechanism, the user executes instructions
directly at the end of the callback function:
```mermaid
sequenceDiagram
participant V as Validation
participant C as Callback
V->>C: Call callback
Note over C: Process event (blocks)
C-->>V: Return
Note over V: Validation resumes
```
To avoid blocking, the user can submit the data to e.g. a worker thread
or event manager, so processing happens asynchronously:
```mermaid
sequenceDiagram
participant V as Validation
participant C as Callback
participant W as Worker Thread
V->>C: Call callback
C->>W: Submit to worker thread
C-->>V: Return immediately
Note over V: Validation continues
Note over W: Process event async
```
These allow for the interpretation of the data in a `BlockChecked` validation interface callback. The validation state passed through `BlockChecked` is the source of truth for the validity of a block (the mode). It is also useful to get richer information in case a block failed to validate (the result).
This adds a function for streaming bytes into a user-owned data structure. Use it in the tests for verifying the implementation of the validation interface's `BlockChecked` method.
This adds functions for reading a block from disk with a retrieved block tree entry. External services that wish to build their own index, or analyze blocks can use this to retrieve block data. The block tree can now be traversed from the tip backwards. This is guaranteed to work, since the chainstate maintains an internal block tree index in memory and every block (besides the genesis) has an ancestor. The user can use this function to iterate through all blocks in the chain (starting from the tip). The tip is retrieved from a separate `Chain` object, which allows distinguishing whether entries are currently in the best chain. Once the block tree entry for the genesis block is reached a nullptr is returned if the user attempts to get the previous entry.
This adds functions for reading the undo data from disk with a retrieved block tree entry. The undo data of a block contains all the spent script pubkeys of all the transactions in a block. For ease of understanding the undo data is renamed to spent outputs with seperate data structures exposed for a block's and a transaction's spent outputs. In normal operations undo data is used during re-orgs. This data might also be useful for building external indexes, or to scan for silent payment transactions. Internally the block undo data contains a vector of transaction undo data which contains a vector of the coins consumed. The coins are all int the order of the transaction inputs of the consuming transactions. Each coin can be used to retrieve a transaction output and in turn a script pubkey and amount. This translates to the three-level hierarchy the api provides: Block spent outputs contain transaction spent outputs, which contain individual coins. Each coin includes the associated output, the height of the block is contained in, and whether it is from a coinbase transaction.
Introduce btck_BlockHash as a type-safe identifier for a block. Adds functions to retrieve block tree entries by hash or height, get block hashes and heights from entries. access the genesis block, and check if blocks are in the active chain.
This introduces the transaction outpoint, input and id types. This now allows a user to retrieve a transaction output from a prior transaction that a transaction outpoint is pointing to. This is exercised in the tests by verifying the script of every transaction in the test chain.
Re-write the bitcoin-chainstate utility by only using the kernel C++ API header instead of internal Bitcoin Core code.
Now that an API has been defined, remove the override for symbol visibility of the library. It is now possible to build the library with reduced exports.
And turn it on in the CI.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.