Skip to content

Sync with upstream floor-licker/polyfill-rs (32 commits)#1

Open
Canvinus wants to merge 32 commits into
mainfrom
upstream-main
Open

Sync with upstream floor-licker/polyfill-rs (32 commits)#1
Canvinus wants to merge 32 commits into
mainfrom
upstream-main

Conversation

@Canvinus

Copy link
Copy Markdown

Sync with Upstream

This PR syncs our fork with the latest changes from floor-licker/polyfill-rs.

Changes Include:

  • /prices-history API helper methods
  • WS hot-path Criterion benchmarks
  • Book-applier stream with no-alloc hot paths
  • Security lockfile updates (RustSec advisories)
  • WebSocket integration tests
  • Bounded queues replacing unbounded channels
  • Better error handling for order endpoints

⚠️ Note

This will have merge conflicts with our custom changes that need manual resolution.

- Remove duplicate package key in Cargo.toml
- Add local polymarket-rs-client dev-dependency for examples
Match upstream behavior: only retry derive-api-key when create-api-key fails with an HTTP status error.
Copilot AI review requested due to automatic review settings January 31, 2026 19:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Syncs this fork with upstream floor-licker/polyfill-rs, bringing in the new /prices-history helpers, updated WebSocket streaming message formats, and the new “book hot path” (decode + apply) processing pipeline with associated tests/benchmarks and CI changes.

Changes:

  • Adds a simd-json tape-based WS book hot-path processor (WsBookUpdateProcessor) plus a WebSocketBookApplier stream that applies updates directly to an OrderBookManager.
  • Introduces /prices-history client helpers and associated integration/unit tests.
  • Expands test/benchmark/CI tooling (no-alloc regression tests, WS integration tests, Criterion benchmark, updated CI workflow).

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tests/ws_integration_tests.rs Adds ignored real WS integration tests (market/user channels).
tests/simple_auth_test.rs Avoids logging secrets; improves auth test output.
tests/prices_history_integration_tests.rs Adds ignored real API integration test for /prices-history.
tests/no_alloc_hot_paths.rs Adds allocator-counting regression tests for no-alloc hot paths.
tests/integration_tests.rs Tweaks authenticated order-flow test behavior and error handling.
tests/common/mod.rs Adds TestConfig::from_env() helper.
src/ws_hot_path.rs Implements simd-json tape-based WS book processing and apply stats.
src/types.rs Updates WS message schema (event_type), adds /prices-history types, adjusts related structs.
src/stream.rs Reworks WS buffering to bounded VecDeque; adds WebSocketBookApplier.
src/lib.rs Re-exports new types and modules (ws_hot_path, prices history types, book applier).
src/decode.rs Updates WS message parsing to official event_type shape + adds deserializers.
src/client.rs Adds /prices-history helper methods + improves error handling/fallback behavior.
src/book.rs Adds apply_book_update and hot-path-oriented WS book update helpers.
scripts/run_integration_tests.sh Adds a script to run ignored real-API tests serially.
rustfmt.toml Simplifies rustfmt config and notes stable compatibility.
examples/snipe.rs Updates example to new StreamMessage variants and book snapshot handling.
examples/demo.rs Updates example to new WS auth/message types and removes unused imports.
docs/TESTING.md Updates testing instructions (ignored tests, no-alloc tests, CI notes).
benches/ws_hot_path.rs Adds Criterion bench comparing tape hot path vs serde decode+apply.
README.md Adds rationale and clarifies “zero-alloc” definition; includes WS hot-path benchmark numbers.
Cargo.toml Adds ws_hot_path bench and polymarket-rs-client dev dependency.
Cargo.lock Updates lockfile (dependency graph changes, RustSec-related updates).
.github/workflows/ci.yml Splits no-alloc tests into a dedicated CI job; skips them in the main test job.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread README.md

At the time that this project was started, `polymarket-rs-client` was a Polymarket Rust Client with a few GitHub stars, but which seemed to be unmaintained. I took on the task of creating a Rust client which could beat the benchmarks quoted in the README.md of that project, with the added constraint of also maintaining zero alloc hot paths.

I also want to take a moment to clarify what zero-alloc means because I've now recieved double digit messages about this on twitter/x and telegram. In general, zero alloc means either zero alloc in hot paths (which can be a bit more arbitrary) or atlernatively it can mean zero alloc after init/warm-up, which is the objective of this repository. Succinctly that means that **the per-message handling loop never touches the heap**.

Copilot AI Jan 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spelling: "recieved" -> "received".

Suggested change
I also want to take a moment to clarify what zero-alloc means because I've now recieved double digit messages about this on twitter/x and telegram. In general, zero alloc means either zero alloc in hot paths (which can be a bit more arbitrary) or atlernatively it can mean zero alloc after init/warm-up, which is the objective of this repository. Succinctly that means that **the per-message handling loop never touches the heap**.
I also want to take a moment to clarify what zero-alloc means because I've now received double digit messages about this on twitter/x and telegram. In general, zero alloc means either zero alloc in hot paths (which can be a bit more arbitrary) or atlernatively it can mean zero alloc after init/warm-up, which is the objective of this repository. Succinctly that means that **the per-message handling loop never touches the heap**.

Copilot uses AI. Check for mistakes.
Comment thread README.md

At the time that this project was started, `polymarket-rs-client` was a Polymarket Rust Client with a few GitHub stars, but which seemed to be unmaintained. I took on the task of creating a Rust client which could beat the benchmarks quoted in the README.md of that project, with the added constraint of also maintaining zero alloc hot paths.

I also want to take a moment to clarify what zero-alloc means because I've now recieved double digit messages about this on twitter/x and telegram. In general, zero alloc means either zero alloc in hot paths (which can be a bit more arbitrary) or atlernatively it can mean zero alloc after init/warm-up, which is the objective of this repository. Succinctly that means that **the per-message handling loop never touches the heap**.

Copilot AI Jan 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spelling: "atlernatively" -> "alternatively".

Suggested change
I also want to take a moment to clarify what zero-alloc means because I've now recieved double digit messages about this on twitter/x and telegram. In general, zero alloc means either zero alloc in hot paths (which can be a bit more arbitrary) or atlernatively it can mean zero alloc after init/warm-up, which is the objective of this repository. Succinctly that means that **the per-message handling loop never touches the heap**.
I also want to take a moment to clarify what zero-alloc means because I've now recieved double digit messages about this on twitter/x and telegram. In general, zero alloc means either zero alloc in hot paths (which can be a bit more arbitrary) or alternatively it can mean zero alloc after init/warm-up, which is the objective of this repository. Succinctly that means that **the per-message handling loop never touches the heap**.

Copilot uses AI. Check for mistakes.
Comment thread src/lib.rs
Comment on lines +150 to +151
pub use crate::stream::{MarketStream, StreamManager, WebSocketBookApplier, WebSocketStream};
pub use crate::ws_hot_path::{WsBookApplyStats, WsBookUpdateProcessor};

Copilot AI Jan 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tokio-tungstenite is an optional dependency behind the stream feature, but lib.rs unconditionally re-exports WebSocketStream/WebSocketBookApplier and unconditionally declares pub mod stream;. As-is, building with --no-default-features (or without stream) will fail to compile because src/stream.rs references tokio_tungstenite types. Consider gating the stream module + related re-exports behind #[cfg(feature = "stream")] (and/or providing non-stream stubs) so the feature flag actually works.

Copilot uses AI. Check for mistakes.
Comment thread src/ws_hot_path.rs
Comment on lines +57 to +66
simd_json::fill_tape(bytes, &mut self.buffers, &mut tape).map_err(|e| {
PolyfillError::parse("Failed to parse WebSocket JSON", Some(Box::new(e)))
})?;

let root = tape.as_value();
let stats = process_root_value(root, books)?;

// Reset the tape to detach lifetimes and keep capacity for reuse.
self.tape = Some(tape.reset());
Ok(stats)

Copilot AI Jan 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WsBookUpdateProcessor::process_bytes takes self.tape out of the struct, but on any early-return (fill_tape(...)? or process_root_value(...)?) the tape is not put back. That leaves self.tape as None and subsequent calls will panic at "tape must be present". Ensure the tape is always restored (e.g., store it back in a match/let result = ...; self.tape = Some(tape.reset()); result pattern, or use a scope guard) even on error paths.

Suggested change
simd_json::fill_tape(bytes, &mut self.buffers, &mut tape).map_err(|e| {
PolyfillError::parse("Failed to parse WebSocket JSON", Some(Box::new(e)))
})?;
let root = tape.as_value();
let stats = process_root_value(root, books)?;
// Reset the tape to detach lifetimes and keep capacity for reuse.
self.tape = Some(tape.reset());
Ok(stats)
// Perform all fallible work in a closure so we can always restore the tape,
// even if an error occurs and we return early.
let result = (|| -> Result<WsBookApplyStats> {
simd_json::fill_tape(bytes, &mut self.buffers, &mut tape).map_err(|e| {
PolyfillError::parse("Failed to parse WebSocket JSON", Some(Box::new(e)))
})?;
let root = tape.as_value();
process_root_value(root, books)
})();
// Reset the tape to detach lifetimes and keep capacity for reuse.
self.tape = Some(tape.reset());
result

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants