Sync with upstream floor-licker/polyfill-rs (32 commits)#1
Conversation
- Remove duplicate package key in Cargo.toml - Add local polymarket-rs-client dev-dependency for examples
Match upstream behavior: only retry derive-api-key when create-api-key fails with an HTTP status error.
There was a problem hiding this comment.
Pull request overview
Syncs this fork with upstream floor-licker/polyfill-rs, bringing in the new /prices-history helpers, updated WebSocket streaming message formats, and the new “book hot path” (decode + apply) processing pipeline with associated tests/benchmarks and CI changes.
Changes:
- Adds a simd-json tape-based WS
bookhot-path processor (WsBookUpdateProcessor) plus aWebSocketBookApplierstream that applies updates directly to anOrderBookManager. - Introduces
/prices-historyclient helpers and associated integration/unit tests. - Expands test/benchmark/CI tooling (no-alloc regression tests, WS integration tests, Criterion benchmark, updated CI workflow).
Reviewed changes
Copilot reviewed 22 out of 23 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
tests/ws_integration_tests.rs |
Adds ignored real WS integration tests (market/user channels). |
tests/simple_auth_test.rs |
Avoids logging secrets; improves auth test output. |
tests/prices_history_integration_tests.rs |
Adds ignored real API integration test for /prices-history. |
tests/no_alloc_hot_paths.rs |
Adds allocator-counting regression tests for no-alloc hot paths. |
tests/integration_tests.rs |
Tweaks authenticated order-flow test behavior and error handling. |
tests/common/mod.rs |
Adds TestConfig::from_env() helper. |
src/ws_hot_path.rs |
Implements simd-json tape-based WS book processing and apply stats. |
src/types.rs |
Updates WS message schema (event_type), adds /prices-history types, adjusts related structs. |
src/stream.rs |
Reworks WS buffering to bounded VecDeque; adds WebSocketBookApplier. |
src/lib.rs |
Re-exports new types and modules (ws_hot_path, prices history types, book applier). |
src/decode.rs |
Updates WS message parsing to official event_type shape + adds deserializers. |
src/client.rs |
Adds /prices-history helper methods + improves error handling/fallback behavior. |
src/book.rs |
Adds apply_book_update and hot-path-oriented WS book update helpers. |
scripts/run_integration_tests.sh |
Adds a script to run ignored real-API tests serially. |
rustfmt.toml |
Simplifies rustfmt config and notes stable compatibility. |
examples/snipe.rs |
Updates example to new StreamMessage variants and book snapshot handling. |
examples/demo.rs |
Updates example to new WS auth/message types and removes unused imports. |
docs/TESTING.md |
Updates testing instructions (ignored tests, no-alloc tests, CI notes). |
benches/ws_hot_path.rs |
Adds Criterion bench comparing tape hot path vs serde decode+apply. |
README.md |
Adds rationale and clarifies “zero-alloc” definition; includes WS hot-path benchmark numbers. |
Cargo.toml |
Adds ws_hot_path bench and polymarket-rs-client dev dependency. |
Cargo.lock |
Updates lockfile (dependency graph changes, RustSec-related updates). |
.github/workflows/ci.yml |
Splits no-alloc tests into a dedicated CI job; skips them in the main test job. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| At the time that this project was started, `polymarket-rs-client` was a Polymarket Rust Client with a few GitHub stars, but which seemed to be unmaintained. I took on the task of creating a Rust client which could beat the benchmarks quoted in the README.md of that project, with the added constraint of also maintaining zero alloc hot paths. | ||
|
|
||
| I also want to take a moment to clarify what zero-alloc means because I've now recieved double digit messages about this on twitter/x and telegram. In general, zero alloc means either zero alloc in hot paths (which can be a bit more arbitrary) or atlernatively it can mean zero alloc after init/warm-up, which is the objective of this repository. Succinctly that means that **the per-message handling loop never touches the heap**. |
There was a problem hiding this comment.
Spelling: "recieved" -> "received".
| I also want to take a moment to clarify what zero-alloc means because I've now recieved double digit messages about this on twitter/x and telegram. In general, zero alloc means either zero alloc in hot paths (which can be a bit more arbitrary) or atlernatively it can mean zero alloc after init/warm-up, which is the objective of this repository. Succinctly that means that **the per-message handling loop never touches the heap**. | |
| I also want to take a moment to clarify what zero-alloc means because I've now received double digit messages about this on twitter/x and telegram. In general, zero alloc means either zero alloc in hot paths (which can be a bit more arbitrary) or atlernatively it can mean zero alloc after init/warm-up, which is the objective of this repository. Succinctly that means that **the per-message handling loop never touches the heap**. |
|
|
||
| At the time that this project was started, `polymarket-rs-client` was a Polymarket Rust Client with a few GitHub stars, but which seemed to be unmaintained. I took on the task of creating a Rust client which could beat the benchmarks quoted in the README.md of that project, with the added constraint of also maintaining zero alloc hot paths. | ||
|
|
||
| I also want to take a moment to clarify what zero-alloc means because I've now recieved double digit messages about this on twitter/x and telegram. In general, zero alloc means either zero alloc in hot paths (which can be a bit more arbitrary) or atlernatively it can mean zero alloc after init/warm-up, which is the objective of this repository. Succinctly that means that **the per-message handling loop never touches the heap**. |
There was a problem hiding this comment.
Spelling: "atlernatively" -> "alternatively".
| I also want to take a moment to clarify what zero-alloc means because I've now recieved double digit messages about this on twitter/x and telegram. In general, zero alloc means either zero alloc in hot paths (which can be a bit more arbitrary) or atlernatively it can mean zero alloc after init/warm-up, which is the objective of this repository. Succinctly that means that **the per-message handling loop never touches the heap**. | |
| I also want to take a moment to clarify what zero-alloc means because I've now recieved double digit messages about this on twitter/x and telegram. In general, zero alloc means either zero alloc in hot paths (which can be a bit more arbitrary) or alternatively it can mean zero alloc after init/warm-up, which is the objective of this repository. Succinctly that means that **the per-message handling loop never touches the heap**. |
| pub use crate::stream::{MarketStream, StreamManager, WebSocketBookApplier, WebSocketStream}; | ||
| pub use crate::ws_hot_path::{WsBookApplyStats, WsBookUpdateProcessor}; |
There was a problem hiding this comment.
tokio-tungstenite is an optional dependency behind the stream feature, but lib.rs unconditionally re-exports WebSocketStream/WebSocketBookApplier and unconditionally declares pub mod stream;. As-is, building with --no-default-features (or without stream) will fail to compile because src/stream.rs references tokio_tungstenite types. Consider gating the stream module + related re-exports behind #[cfg(feature = "stream")] (and/or providing non-stream stubs) so the feature flag actually works.
| simd_json::fill_tape(bytes, &mut self.buffers, &mut tape).map_err(|e| { | ||
| PolyfillError::parse("Failed to parse WebSocket JSON", Some(Box::new(e))) | ||
| })?; | ||
|
|
||
| let root = tape.as_value(); | ||
| let stats = process_root_value(root, books)?; | ||
|
|
||
| // Reset the tape to detach lifetimes and keep capacity for reuse. | ||
| self.tape = Some(tape.reset()); | ||
| Ok(stats) |
There was a problem hiding this comment.
WsBookUpdateProcessor::process_bytes takes self.tape out of the struct, but on any early-return (fill_tape(...)? or process_root_value(...)?) the tape is not put back. That leaves self.tape as None and subsequent calls will panic at "tape must be present". Ensure the tape is always restored (e.g., store it back in a match/let result = ...; self.tape = Some(tape.reset()); result pattern, or use a scope guard) even on error paths.
| simd_json::fill_tape(bytes, &mut self.buffers, &mut tape).map_err(|e| { | |
| PolyfillError::parse("Failed to parse WebSocket JSON", Some(Box::new(e))) | |
| })?; | |
| let root = tape.as_value(); | |
| let stats = process_root_value(root, books)?; | |
| // Reset the tape to detach lifetimes and keep capacity for reuse. | |
| self.tape = Some(tape.reset()); | |
| Ok(stats) | |
| // Perform all fallible work in a closure so we can always restore the tape, | |
| // even if an error occurs and we return early. | |
| let result = (|| -> Result<WsBookApplyStats> { | |
| simd_json::fill_tape(bytes, &mut self.buffers, &mut tape).map_err(|e| { | |
| PolyfillError::parse("Failed to parse WebSocket JSON", Some(Box::new(e))) | |
| })?; | |
| let root = tape.as_value(); | |
| process_root_value(root, books) | |
| })(); | |
| // Reset the tape to detach lifetimes and keep capacity for reuse. | |
| self.tape = Some(tape.reset()); | |
| result |
Sync with Upstream
This PR syncs our fork with the latest changes from floor-licker/polyfill-rs.
Changes Include:
/prices-historyAPI helper methodsThis will have merge conflicts with our custom changes that need manual resolution.