feat: adopt sketch-core runtime modules into existing sketches/ layout by zzylol · Pull Request #36 · ProjectASAP/asap_sketchlib

zzylol · 2026-05-01T18:44:03Z

Summary

Phase 0.5 of the edge-framework design doc (#213) — retire the standalone sketch-core crate and consolidate ASAP's runtime sketches into asap_sketchlib's existing src/sketches/ layout (no separate asap/ namespace).

For each sketch that already had a home in src/sketches/, the wire-format-aligned variant (with public bucket-array fields, sparse multi-cell delta types, apply_delta, msgpack ser/de, from_raw) is appended to the existing file under a clear section header. The existing in-process variants (DDSketch, CountMin<S, Mode, H>, Count, HyperLogLog, KLL<T>, CMSHeap) are unchanged.

Changes

Single-file homes (existing + appended wire-format types):

Existing file	New types added
`src/sketches/ddsketch.rs`	`DdSketch`, `DdSketchDelta`
`src/sketches/countmin.rs`	`CountMinSketch`, `CountMinDelta`
`src/sketches/count.rs`	`CountSketch`, `CountSketchDelta`
`src/sketches/hll.rs`	`HllSketch`, `HllSketchDelta`, `HllVariant`
`src/sketches/kll.rs`	`KllSketch`, `KllSketchData`, `KllBackend`
`src/sketches/cms_heap.rs`	`CountMinSketchWithHeap`, `CmsHeapItem`

New sibling files (no existing home):

hydra_kll.rs, set_aggregator.rs, delta_set_aggregator.rs
count_min_sketchlib.rs, count_min_with_heap_sketchlib.rs, kll_sketchlib.rs (FFI helpers)

Top-level:

src/asap_runtime.rs — ImplMode (Legacy | Sketchlib) selector for the runtime CountMin / KLL / CountMinWithHeap dispatch.

Naming-conflict resolutions

octo_delta::HllDelta already exists (single-register, octo path), so the wire-format multi-register delta is renamed to HllSketchDelta.
common::input::HeapItem already exists, so the wire-format CMSHeap item is renamed to CmsHeapItem.

Tests

```
cargo test --lib → 376 passed, 0 failed
```
(308 existing + 68 ported wire-format tests; no regressions.)

Followups

ProjectASAP/ASAPQuery-backend#73: switch consumer imports from `sketch_core::` to `asap_sketchlib::sketches::`; drop the `asap-common/sketch-core` workspace member.
ProjectASAP/ASAPQuery#307: same migration for the legacy fork.

Update PR #307 against the reorganized asap_sketchlib (PR #36) — runtime sketches now live in the existing src/sketches/ layout (single home per sketch concept), not under a separate `asap::` namespace. Path swaps in asap-query-engine (legacy fork; smaller surface than ASAPQuery-backend — no DDSketch / CountSketch / HllSketch accumulators): - asap_sketchlib::asap::count_min::* → ::sketches::countmin::* - asap_sketchlib::asap::kll::* → ::sketches::kll::* - asap_sketchlib::asap::count_min_with_heap::* → ::sketches::cms_heap::* - asap_sketchlib::asap::hydra_kll::* → ::sketches::hydra_kll::* - asap_sketchlib::asap::set_aggregator::* → ::sketches::set_aggregator::* - asap_sketchlib::asap::delta_set_aggregator::* → ::sketches::delta_set_aggregator::* - asap_sketchlib::asap::config::* → ::asap_runtime::* Rename carried through: - HeapItem → CmsHeapItem (avoids common::input::HeapItem collision) main.rs aliases asap_runtime as `config` so existing clap-derive references keep working. Tests: - cargo check -p query_engine_rust → clean - cargo test -p query_engine_rust --lib precompute_operators → 87 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed e473ccc). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Phase 0.5 of the edge-framework design doc (#213) — retire the standalone `sketch-core` crate and consolidate ASAP's runtime sketches into asap_sketchlib's existing `src/sketches/` layout (no separate `asap/` namespace). For each sketch type that already had a home in `src/sketches/`, the wire-format-aligned variant (with public bucket-array fields, sparse multi-cell delta types, `apply_delta`, msgpack ser/de, and `from_raw` constructors — the surface needed for proto decode + delta-transmission reconstitution; paper §6.2 B3 / B4) is appended to the existing file under a clearly-marked section header. The existing in-process variants (`DDSketch`, `CountMin<S, Mode, H>`, `Count`, `HyperLogLog`, `KLL<T>`, `CMSHeap`) are unchanged. Single-file homes (existing → +appended wire-format types): - `src/sketches/ddsketch.rs` +DdSketch + DdSketchDelta - `src/sketches/countmin.rs` +CountMinSketch + CountMinDelta - `src/sketches/count.rs` +CountSketch + CountSketchDelta - `src/sketches/hll.rs` +HllSketch + HllSketchDelta + HllVariant - `src/sketches/kll.rs` +KllSketch + KllSketchData + KllBackend - `src/sketches/cms_heap.rs` +CountMinSketchWithHeap + CmsHeapItem New sibling files (no existing home — follow the one-file-per-sketch convention): - `src/sketches/hydra_kll.rs` (HydraKllSketch) - `src/sketches/set_aggregator.rs` (SetAggregator) - `src/sketches/delta_set_aggregator.rs` - `src/sketches/count_min_sketchlib.rs` (FFI helper) - `src/sketches/count_min_with_heap_sketchlib.rs` (FFI helper) - `src/sketches/kll_sketchlib.rs` (FFI helper) Top-level: - `src/asap_runtime.rs` — `ImplMode` (Legacy | Sketchlib) backend selector, used by the runtime CountMin / KLL / CountMinWithHeap dispatch. Naming-conflict resolutions: - `octo_delta::HllDelta` already exists (single-register, octo path), so the wire-format multi-register delta is renamed to `HllSketchDelta`. - `common::input::HeapItem` already exists, so the wire-format CMSHeap item is renamed to `CmsHeapItem`. Other notes: - `SketchInput` references in the wrappers (sketch-core was pinned against an older asap_sketchlib) renamed to current `DataInput`; KLL update path switched from `&DataInput::F64(v)` to `&v` to match the typed `KLL<T>` generic API. - `clap::ValueEnum` derive on `ImplMode` is gated behind a new `asap-cli` feature so non-CLI consumers don't pay for `clap`. The query engine enables this feature. - New deps added to Cargo.toml: `xxhash-rust` (xxh32), `dsrs` (datasketches-rs), `clap` (optional, behind `asap-cli`), `ctor` (dev-dep for the legacy-mode test initializer that previously lived in sketch-core's lib.rs). Tests: cargo test --lib → 376 passed, 0 failed (308 existing + 68 ported wire-format tests). No regressions. Followup: ASAPQuery-backend PR #73 and ASAPQuery PR #307 will switch consumer imports from `sketch_core::*` and the older `asap_sketchlib::asap::*` to the new `asap_sketchlib::sketches::*` paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: retire sketch-core mirror * refactor: switch consumer imports to asap_sketchlib::sketches::* Update PR #307 against the reorganized asap_sketchlib (PR #36) — runtime sketches now live in the existing src/sketches/ layout (single home per sketch concept), not under a separate `asap::` namespace. Path swaps in asap-query-engine (legacy fork; smaller surface than ASAPQuery-backend — no DDSketch / CountSketch / HllSketch accumulators): - asap_sketchlib::asap::count_min::* → ::sketches::countmin::* - asap_sketchlib::asap::kll::* → ::sketches::kll::* - asap_sketchlib::asap::count_min_with_heap::* → ::sketches::cms_heap::* - asap_sketchlib::asap::hydra_kll::* → ::sketches::hydra_kll::* - asap_sketchlib::asap::set_aggregator::* → ::sketches::set_aggregator::* - asap_sketchlib::asap::delta_set_aggregator::* → ::sketches::delta_set_aggregator::* - asap_sketchlib::asap::config::* → ::asap_runtime::* Rename carried through: - HeapItem → CmsHeapItem (avoids common::input::HeapItem collision) main.rs aliases asap_runtime as `config` so existing clap-derive references keep working. Tests: - cargo check -p query_engine_rust → clean - cargo test -p query_engine_rust --lib precompute_operators → 87 passed, 0 failed Depends on ProjectASAP/asap_sketchlib#36 (force-pushed e473ccc). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

PR #36 silently reverted PR #37's hh_keys field on CountSketchDelta when its merge landed. Restore on top of the post-#39 module names (src/sketches/countsketch.rs). Mirrors sketchlib-go's Delta.HHKeys / apply-delta heavy-hitter rebuild path. Unblocks ASAPQuery-backend's PR-#74 consumer to pin asap_sketchlib to main, which in turn unblocks ASAPCollector Phase 3 step 3 (backend consumes asap-precompute-rs). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…dule and legacy msgpack shims (#46) * refactor: introduce wrapper + message_pack_format top-level modules Move PR #36–#45 wire-format-aligned variants out of src/sketches/ into a new src/wrapper/ module so the in-process algorithms and the sketchlib-go-aligned producer types live in separate trees. Add a src/message_pack_format.rs placeholder noting that asap_sketchlib (Rust) and sketchlib-go each maintain their own MessagePack representation, byte-compatible at the envelope level. Files moved: - src/wrapper/{ddsketch,countminsketch,countsketch,hll,kll,countminsketch_topk}.rs carry the wire-format types appended after the per-file banner in PR #36+. - src/wrapper/{hydra_kll,set_aggregator,delta_set_aggregator}.rs are whole-file moves (they were entirely wire-format). CountL2HH stays in src/sketches/countsketch_topk.rs — it's an in-process top-k algorithm used by sketch_framework, not a wire-format type. No semantic changes; cargo test --lib --all-features still 447 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: clean unused imports, apply fmt + clippy in wrappers Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(hashspec): drop broken intra-doc links in module-level docs The `[`CANONICAL_HASH_SEED_TABLE`]` and `[`CANONICAL_HASH_SEED`]` shortcut links in the module-level `//!` docs failed to resolve under `cargo doc --all-features`, breaking the Rust Docs CI workflow with `-D warnings`. Both items are still defined in this module; the doc text simply uses inline-code formatting now instead of intra-doc links so docgen succeeds. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(message_pack_format): make module concrete with DTOs + codec trait Move all wire-format DTOs (WireFormat, CmsData, CountMinSketchWithHeapSerialized, HydraKllSketchData, KllSketchData, StringSet, DeltaResult) out of individual wrappers and into message_pack_format::dto as the single source of truth. Add a unified Error enum and a MessagePackCodec trait (static-dispatch, zero cost) that every wire-format-aligned wrapper implements. The legacy inherent serialize_msgpack / deserialize_msgpack methods are preserved with their original signatures as thin shims over the trait, so callers and FFI bindings need no changes. KllSketchData and DeltaResult remain pub via re-exports from their wrapper modules. Add tests/msgpack_compat.rs with round-trip and DTO-shape coverage for every type, plus #[ignore]-d slots reserved for cross-language golden-bytes fixtures from sketchlib-go. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(message_pack_format): drop broken intra-doc links in module docs `[`Error`]`, `[`MessagePackCodec`]`, `[`dto`]` shortcut links inside the module-level `//!` block failed to resolve under `cargo doc --all-features` even though the items are defined in the same module — likely a rustdoc quirk around `pub use` shadowing the shortcut-link namespace. Switch to inline-code formatting so docgen succeeds; the items are still discoverable via the usual rustdoc sidebar. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * update file structure: clear separation of wrapper and msgpack_pack_format * docs(message_pack_format): drop broken intra-doc links reintroduced in restructure The module restructure in 746e335 re-added `[`Error`]` / `[`MessagePackCodec`]` shortcut links in the `//!` block that fail to resolve under `cargo doc --all-features` (same rustdoc quirk fixed earlier in 64b6989). Switch back to inline-code formatting so docgen passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: add wrapper and message_pack_format pages The docs predate the wrapper / message_pack_format split and didn't cover either module. Add dedicated pages for each and link them from index, project_overview, and library_map. Entry pages stay lean — details live on the dedicated pages so readers who don't need them aren't forced through them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(message_pack_format): split into native/ and portable/ Reorganize the ser/de module along an audience boundary so reviewers can tell at a glance whether a change touches the cross-language wire contract or only Rust-internal persistence. - portable/: cross-language wire format shared with sketchlib-go. The 9 existing per-algorithm files move here unchanged (only super:: imports rewritten to crate::message_pack_format::). Touching this directory is a protocol change. - native/: thin MessagePackCodec trait shims over the existing serialize_to_bytes / deserialize_from_bytes methods on the pure-Rust generic sketches in src/sketches/. Byte format is Rust-internal; Go never reads it. sketches/ and wrapper/ logic is unchanged. Existing public API methods (CountMin::serialize_to_bytes, CountMinSketch::serialize_msgpack, etc.) are preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(message_pack_format): unbreak CI and refresh docs for native/portable split Drop the outer `///` doc on `pub mod message_pack_format;` in lib.rs — it was merged with mod.rs's inner `//!` doc and forced intra-doc link resolution into the crate root, breaking `[`native`]`, `[`portable`]`, and `[`Error`]`. Also fix a stale `hydra_kll` path under `portable::` and pick up `cargo fmt` reorderings. Rewrite docs/message_pack_format.md to cover the new native/ vs portable/ layout. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(wrapper): drop serialize_msgpack/deserialize_msgpack shims The wrapper sketch types each carried inherent serialize_msgpack / deserialize_msgpack methods that just forwarded to MessagePackCodec:: to_msgpack / from_msgpack with extra error-type adaptation. Now that MessagePackCodec is the canonical entry point, the shims add a second name for the same operation and fragment the error surface (RmpEncodeError vs Box<dyn Error> vs MsgPackError) for no benefit. Removes the inherent shims from CountMinSketch, CountMinSketchWithHeap, CountSketch, DdSketch, HllSketch, KllSketch, HydraKllSketch, and SetAggregator. Internal callers (aggregate_* helpers, tests) now use to_msgpack / from_msgpack directly. The delta_set_aggregator free functions are unchanged — they are not type-method shims and keep their pre-trait signature. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(wrapper): drop hashspec bypass; reuse sketches FastPath math `wrapper::CountSketch::update`/`estimate` and `wrapper::CountMinSketch:: update`/`estimate` were each open-coding a Go-compatible hash → (col, sign) derivation via `common::hashspec` self-functions. The same packed bit-slicing math already lives in `MatrixHashType::Packed64` / `sketches::CountMin<.., FastPath>::insert_many`, and produces bit-identical envelope bytes against the existing Go golden fixtures. Changes: - `wrapper::CountSketch::{update,estimate}` use `MatrixHashType::Packed64 + DefaultXxHasher::hash64_seeded(0,..)` directly. No behavior change. - `wrapper::CountMinSketch` switches its backend from `RegularPath` to `FastPath` and routes update/estimate through `backend.insert_many` / `backend.estimate` — the wrapper no longer bypasses the backend. New parity probe in `tests/sketches_go_parity_probe.rs` locks in that `sketches::Count<Vector2D<i64>, FastPath, DefaultXxHasher>` and `sketches::CountMin<Vector2D<f64>, FastPath, DefaultXxHasher>` emit byte-identical SketchEnvelope encodings to the captured sketchlib-go goldens (1577-byte CountSketch, 8275-byte CMS Frequency-Only). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: delete mod wrapper; wire DTOs live in message_pack_format The `wrapper/` module was a parallel set of "wire-format-aligned" sketch types added when the runtime sketches in `sketches/` were believed not to produce Go-byte-compatible output. The parity probes added in this branch (`tests/sketches_go_parity_probe.rs`) confirm `sketches::Count<Vector2D<i64>, FastPath>`, `sketches::CountMin<Vector2D<f64>, FastPath>`, `sketches::HyperLogLogImpl<ErtlMLE, P14>`, and `sketches::DDSketch` all emit byte-identical envelopes to `sketchlib-go` for the same goldens, so the parallel layer is redundant. Changes: - Move every type formerly under `src/wrapper/*.rs` into the corresponding `src/message_pack_format/portable/*.rs` file, keeping the wire DTO and runtime ops together. Internal tests come along. - Drop the `crate::wrapper` module entirely. - Re-export the runtime + wire DTO types at the crate root from their new home in `message_pack_format::portable::*`. - Add small public accessors to `sketches::DDSketch` (`alpha`/`sum`/`store_counts`/`store_offset`) and to `sketches::HyperLogLogImpl::registers_as_slice` so the parity probe can build wire envelopes directly from `sketches::*` without reaching into private fields. - Extend `tests/sketches_go_parity_probe.rs` with HLL + DDSketch probes (CountSketch + CountMin already there); all 4 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * sketches: expose KLL::k() accessor The compactor capacity `k` is private inside `sketches::KLL<T>` but needed by downstream consumers that build the Go-compatible `KllSketchData { k, sketch_bytes }` wire envelope around a directly- held `sketches::KLL<f64>`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ci): rustfmt + rustdoc; refresh docs for deleted wrapper module CI was red on fmt and cargo doc: - `cargo fmt --all -- --check` flagged unrelated formatting drift across lib.rs, kll.rs, countminsketch.rs, countsketch.rs, and msgpack_compat.rs — applied rustfmt. - `cargo doc -D warnings` failed on broken intra-doc links to `crate::wrapper` (the module was deleted in bfe8f37) and an unqualified `MatrixHashType::Packed64` link. Repointed the surviving rustdoc to `crate::message_pack_format::portable` / `crate::MatrixHashType`. Also removed the now-dead `Error::into_encode` helper that clippy/rustc warned about (it was the last caller of the serialize_msgpack shim deleted in f847b18). Doc refresh under docs/: - wrapper.md now describes the wire-format-aligned types in their new home under message_pack_format/portable/ instead of the removed src/wrapper/ tree. - index.md, library_map.md, project_overview.md, and message_pack_format.md updated to drop src/wrapper/ references and reflect that DTO + runtime ops are co-located per-algorithm. - Test file headers (msgpack_compat.rs, sketches_go_parity_probe.rs) no longer reference the deleted wrapper module. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: drop wrapper.md; fold useful bits into message_pack_format.md The wrapper.md page documented a module that no longer exists. The only content not already in message_pack_format.md was the in-process (generic sketches) vs wire-aligned (portable) selection guidance and the per-file type listing — both folded into the `portable/` section of message_pack_format.md. Updated index.md, project_overview.md, and library_map.md to drop their wrapper.md links. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

zzylol mentioned this pull request May 1, 2026

refactor: retire sketch-core mirror ProjectASAP/ASAPQuery#307

Merged

zzylol force-pushed the refactor/adopt-sketch-core-modules branch from 63051a8 to e473ccc Compare May 1, 2026 19:18

zzylol changed the title ~~feat: adopt sketch-core runtime modules~~ feat: adopt sketch-core runtime modules into existing sketches/ layout May 1, 2026

zzylol force-pushed the refactor/adopt-sketch-core-modules branch 4 times, most recently from 8db5e7f to 943817e Compare May 1, 2026 20:34

zzylol force-pushed the refactor/adopt-sketch-core-modules branch 2 times, most recently from d5015ed to 0338b3b Compare May 1, 2026 20:53

zzylol force-pushed the refactor/adopt-sketch-core-modules branch from 0338b3b to d84ff15 Compare May 1, 2026 21:16

zzylol force-pushed the refactor/adopt-sketch-core-modules branch from d84ff15 to 5f990ae Compare May 1, 2026 21:43

zzylol merged commit d22a9ab into main May 2, 2026
2 checks passed

This was referenced May 2, 2026

refactor: sketch-core retirement + branch-pin cleanup + elastic DSL test fix ProjectASAP/ASAPQuery#309

Merged

fix(countsketch): restore hh_keys field + apply_delta topk-rebuild #42

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: adopt sketch-core runtime modules into existing sketches/ layout#36

feat: adopt sketch-core runtime modules into existing sketches/ layout#36
zzylol merged 1 commit into
mainfrom
refactor/adopt-sketch-core-modules

zzylol commented May 1, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zzylol commented May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Naming-conflict resolutions

Tests

Followups

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

zzylol commented May 1, 2026 •

edited

Loading