Upstream test deprecate rent exemption threshold#44
Open
deeprnd wants to merge 63 commits into
Open
Conversation
By popular demand, this modifies fd_alloc to support larger allocations before falling back to the slow locking underlying allocator of last resort (fd_wksp). This couldn't be done simply by updating the sizeclass config as the underlying fd_alloc memory layout couldn't support the required larger superblock footprints. Thus, fd_alloc was tweaked to be more like fd_vinyl_data (which already had a notion of sizeclasses large enough to hold account state as large as 10 MiB). Most important, fd_alloc_superblock_t is now where block sizeclass metadata is stored (previously it was specified in the fd_alloc_hdr_t of allocated blocks in the sizeclass). This supports many more sizeclasses (because there is more space in the fd_alloc_superblock_t to store it) and much larger sizeclass block sizes (because moving the sizeclass frees up the space in the fd_alloc_hdr_t needed to support larger superblock footprints). This also supports better allocator diagnostics (because it is now possible to find every live user allocation in an fd_alloc instance without user assistance). Additionally, like fd_vinyl_data, user allocations are treated distinctly from superblock allocations. This allows more compact superblock nesting and streamlined allocation recursion. Accordingly, fd_alloc_superblock_t was modified to include the block sizeclass and fd_alloc_superblock_t are blocks are now all guaranteed to have 8 byte alignment. Though it naively looks like it got 8 bytes larger, because of the previous alignment requirements and the implicitly preprended fd_alloc_hdr_t, it is practically comparable to smaller now even though it holds more info. Likewise, fd_alloc_hdr_t was modified to encode a 2-bit type (user small, user large, nested superblock or root superblock) and, for allocations contained in a superblock (user small or nested superblock), 6-bit block_idx and offset to the containing superblock (26-bits encoded in 24-bits). TL;DR This allows sizeclass blocks as large as ~32 MiB to be supported while keeping the allocator memory footprint overhead comparable. Various cleanups in support of the above: - Added gen_szc_cfg sizeclass configuration generation for all the above. This has been tuned to do reduce the number of sizeclasses (hence preallocations) for the relative amount of overallocation from rounding up to the nearest block size. - fd_alloc was turned into a quasi-opaque handle instead of a fully opaque handle. Alignment requirements were also made less stringent. This makes it easier to inline various operations, streamline various unit tests and use different sizeclass configurations. - Two compile time sizeclass configurations are provided: small (which carves small allocations out of 256 KiB wksp partitions where allocations less than ~37KiB in size are considered smalll ... similar to previous) and large (64 MiB partitions and ~10 MiB small allocations). Default is large. - The use of 16-byte wide atomics on x86-like platforms was eliminated. This makes the fd_alloc shared memory layout and so forth identical across target platform of the same endianness. This was made possible by limiting the maximum size workspace backing an fd_alloc to 1 PiB (which in turns allows an aligned workspace global address and a suitably wide lockfree ABA version number to be encoded in a ulong). - Accordingly, FD_ALLOC_MAGIC was updated (and is now independent of build target). - fd_alloc_delete allows more fine grained control over how aggressively to cleanup left over allocations in the underlying workspace. - The necessary iterations for fd_alloc_preferred_sizeclass needs is explicitly provided by the sizeclass configuration. - Added minor block_set optimizations including atomic operations slightly optimized to use op_then_fetch style (instead of fetch_then_op) and the fd_alloc_block_set_all API that returns the full block set for a given block count. - Compile-time FD_ALLOC_STYLE define to enable alignment pad clearing for more robust and faster discovery of user allocations. - fd_alloc_is_empty was simplified to use fd_wksp_tag functionality. - ASAN and MSAN support was cleaned up and fixed to work correctly under concurrent load. - fd_alloc_fprintf updated for all the above, with more extensive diagnostics, including a strong guarantee it will find all current allocations under the conditions of an idle fd_alloc with alignment padding clearing (FD_ALLOC_STYLE==1). - Updated test_alloc to match the above (including verifying sizeclassing and ability to do concurrent testing under ASAN and MSAN options). - Usual drive-by include and comment cleanups.
currently we would crash if NULL were to be passed to the function
By popular demand, updated the sizeclass configuration for fd_vinyl_data to be less aggressive about preeemptively allocating blocks. This uses superblocks that are 2 to 4 times smaller than previously and 4 times fewer sizeclasses, which implies there will be roughly an order of magnitude less preemptive allocation. The tradeoff is that superblocks hold 2 to 4 times fewer allocation blocks and a deeper nesting (which implies some increase in computational overhead). Likewise, this increases the worst case rounding up of allocations to sizeclass block sizes by roughly a a factor of 2 (which is a bit of a mixed bag ... a few percent extra overhead on fixed sized allocations but a cheaper ability to do incremental resizing). The volume footprint increased somewhat with this. But gen_szc_cfg can be easily modified to use the old volume footprint (or even a smaller or more power-of-2-ish volume footprint).
Use a static signing domain instead of a random key. This makes it easier to prove that there are no signing confusion bugs.
* fix(secp256r1): add one more point p256_scalarmulbase() is called with blocksize=6, so it scans, in constant time, 43 windows * 32 = 1376 affine points of the base point table. The table had only 1375 entries, so every secp256r1 signature verify read one point (64 bytes) past the end of fd_secp256r1_base_point_table. Add the missing 2^252*32*G entry and a static assert pinning the table size. * Update src/ballet/secp256r1/fd_secp256r1_s2n.c Co-authored-by: David Rubin <drubin+git@jumptrading.com> --------- Co-authored-by: David Rubin <drubin+git@jumptrading.com>
Allow remembering Make config using shell variables
Add support for subscribing to votes via the RPC tile. https://solana.com/docs/rpc/websocket/votesubscribe
sendmmsg was only ever sending to one dest, might as well use the simpler sendmsg syscall then
exec/s improve ~5x
no security/correctness impact, just a consistency nit
Move non-parametric tests to new scheduler unit test. Improves fuzzers exec/s by ~30%
This is algorithmically a fairly sloppy way to ingest /proc/interrupts, but it doesn't matter, this monitoring code barely uses any CPU.
find -exec returns 0 even if one of the commands failed. This would mask actual errors.
This commit adds two fuzzers: - `gossvf_tile` fuzzes the gossvf tile in isolation by mocking its in/out links in a 8 peer environment. Fast ~1050 exec/s - gossvf_gossip_pair: Tests the gossvf and the gossip tile together. Two peers are talking to each other and we use "fault injection" to test with invalid inputs. In addition we let the fuzzer do things like skip ahead in time to have fast fuzz-throughput as we are not bounded by a real network and want to test the time-based events.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Type of change
Related work
Risk & impact
How to test
Runtime / contract changes (if applicable)
Generated code / artifacts (if applicable)
make -C src/disco/metrics metrics)cd src/flamenco/features && make generate)make -C src/flamenco/runtime/tests protobufs)Build / config / docs changes (if applicable)
justfile/tooling updatedFiredancer scope (if applicable)
Firedancer notes
Checklist
Implementation
Tests
just tests-allcommand executed successfullyIf tests were not added, explain why
Observability / operations (if applicable)
Security & privacy (if applicable)
Release notes
Release note (if needed)