Skip to content

Commit 768324c

Browse files
milindsrivastava1997milindsrivastava1997
committed
Merge remote-tracking branch 'origin/main' into 258-add-capability-based-matching-to-asap-query-engine-instead-of-matching-incoming-queries-against-pre-configured-queries-in-inference_config
2 parents 0244096 + a3894c1 commit 768324c

62 files changed

Lines changed: 5878 additions & 906 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/accuracy_performance.yml‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
name: PR Evaluation
2+
3+
# NOTE: GitHub-hosted runners are noisy. Latency numbers are indicative only.
4+
# For precise benchmarks, register a self-hosted runner once asap-tools infra
5+
# is decoupled from Cloudlab. See PDF eval guide Phase 3.
6+
7+
on:
8+
pull_request:
9+
branches:
10+
- main
11+
paths:
12+
- 'asap-query-engine/**'
13+
- 'asap-planner-rs/**'
14+
- 'asap-summary-ingest/**'
15+
- 'asap-quickstart/**'
16+
- '.github/workflows/accuracy_performance.yml'
17+
- 'benchmarks/**'
18+
workflow_dispatch:
19+
20+
permissions:
21+
contents: read
22+
packages: write
23+
pull-requests: write
24+
25+
jobs:
26+
# ---------------------------------------------------------------------------
27+
# Job 1: build images once from branch code and push with a SHA-based tag.
28+
# All downstream jobs pull these images instead of rebuilding.
29+
# ---------------------------------------------------------------------------
30+
build:
31+
name: Build CI images
32+
runs-on: ubuntu-latest
33+
outputs:
34+
image-tag: ${{ steps.tag.outputs.value }}
35+
36+
steps:
37+
- name: Checkout repository
38+
uses: actions/checkout@v4
39+
40+
- name: Set up Docker Buildx
41+
uses: docker/setup-buildx-action@v3
42+
43+
- name: Log in to GHCR
44+
uses: docker/login-action@v3
45+
with:
46+
registry: ghcr.io
47+
username: ${{ github.repository_owner }}
48+
password: ${{ secrets.GITHUB_TOKEN }}
49+
50+
- name: Compute image tag
51+
id: tag
52+
run: echo "value=sha-$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT
53+
54+
- name: Build and push asap-planner-rs
55+
uses: docker/build-push-action@v6
56+
with:
57+
context: .
58+
file: asap-planner-rs/Dockerfile
59+
push: true
60+
tags: ghcr.io/projectasap/asap-planner-rs:${{ steps.tag.outputs.value }}
61+
cache-from: type=registry,ref=ghcr.io/projectasap/asap-planner-rs:buildcache
62+
cache-to: type=registry,ref=ghcr.io/projectasap/asap-planner-rs:buildcache,mode=max
63+
64+
- name: Build and push asap-summary-ingest
65+
uses: docker/build-push-action@v6
66+
with:
67+
context: asap-summary-ingest
68+
file: asap-summary-ingest/Dockerfile
69+
push: true
70+
tags: ghcr.io/projectasap/asap-summary-ingest:${{ steps.tag.outputs.value }}
71+
build-args: BASE_IMAGE=ghcr.io/projectasap/asap-base:latest
72+
73+
- name: Build and push asap-query-engine
74+
uses: docker/build-push-action@v6
75+
with:
76+
context: .
77+
file: asap-query-engine/Dockerfile
78+
push: true
79+
tags: ghcr.io/projectasap/asap-query-engine:${{ steps.tag.outputs.value }}
80+
cache-from: type=registry,ref=ghcr.io/projectasap/asap-query-engine:buildcache
81+
cache-to: type=registry,ref=ghcr.io/projectasap/asap-query-engine:buildcache,mode=max
82+
83+
# ---------------------------------------------------------------------------
84+
# Job 2: pull the images built above, deploy the full stack, and evaluate.
85+
# ---------------------------------------------------------------------------
86+
eval:
87+
name: Full-stack PR evaluation
88+
needs: build
89+
runs-on: ubuntu-latest
90+
timeout-minutes: 60
91+
env:
92+
ASAP_IMAGE_TAG: ${{ needs.build.outputs.image-tag }}
93+
94+
steps:
95+
- name: Checkout repository
96+
uses: actions/checkout@v4
97+
98+
- name: Log in to GHCR
99+
uses: docker/login-action@v3
100+
with:
101+
registry: ghcr.io
102+
username: ${{ github.repository_owner }}
103+
password: ${{ secrets.GITHUB_TOKEN }}
104+
105+
- name: Pull and start full stack
106+
run: |
107+
docker compose \
108+
-f asap-quickstart/docker-compose.yml \
109+
-f benchmarks/docker-compose.yml \
110+
up -d
111+
112+
- name: Show running containers
113+
run: |
114+
docker compose \
115+
-f asap-quickstart/docker-compose.yml \
116+
-f benchmarks/docker-compose.yml \
117+
ps
118+
119+
- name: Wait for all services to be healthy
120+
run: bash benchmarks/scripts/wait_for_stack.sh
121+
122+
- name: Wait for pipeline and data ingestion
123+
run: bash benchmarks/scripts/ingest_wait.sh
124+
125+
- name: Set up Python 3.11
126+
uses: actions/setup-python@v5
127+
with:
128+
python-version: '3.11'
129+
130+
- name: Install Python dependencies
131+
run: pip install requests
132+
133+
- name: Run baseline queries (Prometheus)
134+
run: python benchmarks/scripts/run_baseline.py
135+
136+
- name: Run ASAP queries (query engine)
137+
run: python benchmarks/scripts/run_asap.py
138+
139+
- name: Compare results and evaluate
140+
run: python benchmarks/scripts/compare.py
141+
142+
- name: Upload evaluation reports
143+
if: always()
144+
uses: actions/upload-artifact@v4
145+
with:
146+
name: eval-reports-${{ github.run_id }}
147+
path: benchmarks/reports/
148+
149+
- name: Print docker logs on failure
150+
if: failure()
151+
run: |
152+
docker compose \
153+
-f asap-quickstart/docker-compose.yml \
154+
-f benchmarks/docker-compose.yml \
155+
logs --no-color
156+
157+
- name: Teardown stack
158+
if: always()
159+
run: |
160+
docker compose \
161+
-f asap-quickstart/docker-compose.yml \
162+
-f benchmarks/docker-compose.yml \
163+
down -v

‎README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
2+
[![Build](https://github.com/ProjectASAP/ASAPQuery/actions/workflows/rust.yml/badge.svg)](https://github.com/ProjectASAP/ASAPQuery/actions/workflows/rust.yml)
3+
14
# ASAPQuery
25

36
**ASAPQuery** is a drop-in query accelerator for queries in various languages like PromQL and SQL. ASAPQuery delivers:

‎asap-common/sketch-core/report.md‎

Lines changed: 62 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,18 @@
1-
# Sketchlib Fidelity Report
1+
# Report
22

3-
Compares the **legacy** Count-Min Sketch implementation in `sketch-core` vs the new **sketchlib-rust** backend.
3+
Compares the **legacy** sketch implementations in `sketch-core` vs the **sketchlib-rust** backends (Count-Min Sketch, Count-Min-With-Heap, KLL, HydraKLL).
44

55
## Fidelity harness
66

7-
The fidelity binary selects backends via CLI flags.
7+
The fidelity binary selects backends via CLI flags (`--cms-impl`, `--kll-impl`, `--cmwh-impl`).
88

9-
| Goal | Command |
10-
|-------------|---------------------------------------------------------------|
11-
| CMS sketchlib | `cargo run -p sketch-core --bin sketchlib_fidelity -- --cms-impl sketchlib` |
12-
| CMS legacy | `cargo run -p sketch-core --bin sketchlib_fidelity -- --cms-impl legacy` |
9+
| Goal | Command |
10+
|--------------------------|--------------------------------------------------------------------------------------------------------------|
11+
| Default (all sketchlib) | `cargo run -p sketch-core --bin sketchlib_fidelity` |
12+
| All legacy | `cargo run -p sketch-core --bin sketchlib_fidelity -- --cms-impl legacy --kll-impl legacy --cmwh-impl legacy` |
13+
| Legacy KLL only | `cargo run -p sketch-core --bin sketchlib_fidelity -- --cms-impl sketchlib --kll-impl legacy --cmwh-impl sketchlib` |
14+
| CMS sketchlib only | `cargo run -p sketch-core --bin sketchlib_fidelity -- --cms-impl sketchlib` |
15+
| CMS legacy only | `cargo run -p sketch-core --bin sketchlib_fidelity -- --cms-impl legacy` |
1316

1417
## Unit tests
1518

@@ -68,3 +71,55 @@ The heap is maintained by local updates; recall is measured against the **true**
6871
| 2048 | 200000 | 2000 | 20 | sketchlib-rust | 1.00 | 0.9982 | 0.021 | 0.067 |
6972
| 2048 | 200000 | 2000 | 50 | Legacy | 0.40 | 0.9999983 | 5.60 | 16.49 |
7073
| 2048 | 200000 | 2000 | 50 | sketchlib-rust | 0.48 | 0.9999990 | 3.90 | 12.95 |
74+
75+
---
76+
77+
### KllSketch (quantiles, absolute rank error)
78+
79+
For each quantile \(q\), we compute the sketch estimate `est_value`, then:
80+
`abs_rank_error = |rank_fraction(exact_sorted_values, est_value) - q|`.
81+
82+
#### k=20
83+
84+
| n_updates | Mode | q=0.5 | q=0.9 | q=0.99 |
85+
|-----------|----------------|---------|---------|---------|
86+
| 200000 | Legacy | 0.0104 | 0.0145 | 0.0028 |
87+
| 200000 | sketchlib-rust | 0.0275 | 0.0470 | 0.0061 |
88+
| 50000 | Legacy | 0.0131 | 0.0091 | 0.0054 |
89+
| 50000 | sketchlib-rust | 0.0110 | 0.0116 | 0.0031 |
90+
91+
#### k=50
92+
93+
| n_updates | Mode | q=0.5 | q=0.9 | q=0.99 |
94+
|-----------|----------------|---------|---------|---------|
95+
| 200000 | Legacy | 0.0013 | 0.0021 | 0.0012 |
96+
| 200000 | sketchlib-rust | 0.0101 | 0.0044 | 0.0074 |
97+
98+
#### k=200
99+
100+
| n_updates | Mode | q=0.5 | q=0.9 | q=0.99 |
101+
|-----------|----------------|---------|---------|---------|
102+
| 200000 | Legacy | 0.0021 | 0.0036 | 0.0000 |
103+
| 200000 | sketchlib-rust | 0.0015 | 0.0001 | 0.0002 |
104+
105+
---
106+
107+
### HydraKllSketch (per-key quantiles, mean/max absolute rank error across 50 keys)
108+
109+
#### rows=2, cols=64
110+
111+
| k | n | domain | Mode | q=0.5 (mean / max) | q=0.9 (mean / max) |
112+
|-----|--------|--------|----------------|--------------------|--------------------|
113+
| 20 | 200000 | 200 | Legacy | 0.0170 / 0.0546 | 0.0165 / 0.0452 |
114+
| 20 | 200000 | 200 | sketchlib-rust | 0.0254 / 0.0629 | 0.0546 / 0.0942 |
115+
116+
#### rows=3, cols=128
117+
118+
| k | n | domain | Mode | q=0.5 (mean / max) | q=0.9 (mean / max) |
119+
|-----|--------|--------|----------------|--------------------|--------------------|
120+
| 20 | 200000 | 200 | Legacy | 0.0166 / 0.0591 | 0.0114 / 0.0304 |
121+
| 20 | 200000 | 200 | sketchlib-rust | 0.0216 / 0.0534 | 0.0238 / 0.1087 |
122+
| 50 | 200000 | 200 | Legacy | 0.0099 / 0.0352 | 0.0087 / 0.0330 |
123+
| 50 | 200000 | 200 | sketchlib-rust | 0.0119 / 0.0458 | 0.0119 / 0.0296 |
124+
| 20 | 100000 | 100 | Legacy | 0.0141 / 0.0574 | 0.0149 / 0.0471 |
125+
| 20 | 100000 | 100 | sketchlib-rust | 0.0202 / 0.0621 | 0.0287 / 0.0779 |

0 commit comments

Comments
 (0)