feat: ClickHouse-Grafana drop-in (clickhouse-dropin/)

[milindsrivastava1997]

Tracking issue for adding a clickhouse-dropin/ — a self-contained docker-compose that transparently accelerates an existing ClickHouse + Grafana stack, analogous to asap-dropin/ for Prometheus.

Design: ASAPQuery sits in front of ClickHouse, impersonating its HTTP interface. Grafana's ClickHouse datasource plugin points to ASAP instead of ClickHouse directly. Unsupported queries are forwarded to the real ClickHouse.

---

TODOs

• Protocol / adapter gaps (blockers)

  • Fix endpoint URL mismatch — adapter exposes /clickhouse/query but Grafana plugins expect root /
  • Auth passthrough — forward X-ClickHouse-User / X-ClickHouse-Key headers (official plugin) and Basic Auth (Altinity plugin)
  • Thread database URL param through the adapter
  • Strip (or handle) FORMAT clauses appended by Grafana plugins before sketch execution
  • Set correct Content-Type: text/plain; charset=UTF-8 on responses
  • Respect time range from query (adapter currently always uses SystemTime::now())
  • Make adapter selectable from engine_config.yaml without editing main.rs

• Data ingest (open design question — decision needed before implementation)

  • Option A: polling ingest — ASAP periodically SELECTs new rows from ClickHouse (no extra infra, needs new ingest driver)
  • Option B: Kafka tap — reuse existing Kafka ingest driver, but adds infra complexity for users
  • Option C: query-driven ingest — fetch historical data from ClickHouse lazily on first query observation

• Query tracker for ClickHouse

  • Implement build_schema_from_clickhouse() using system.columns / information_schema.columns
  • Extend planner to parse SQL structure (identify time column, dimension columns, aggregated values)
  • Remove/relax the hard-coded Prometheus-only restriction in engine_config.rs

• Deployment scaffold

  • Create clickhouse-dropin/ with docker-compose.yml, docker-compose.dev.yml, docker-compose.override.yml
  • .env (CLICKHOUSE_URL, CLICKHOUSE_DATABASE, CLICKHOUSE_USER, CLICKHOUSE_PASSWORD, ASAP port)
  • config/engine_config.yaml with backend.type: clickhouse
  • Grafana provisioning config for ClickHouse datasource

• README

  • Prerequisites, networking modes, .env config
  • How to redirect Grafana datasource URL to ASAP
  • Verification steps
  • Differences from Prometheus dropin (SQL not PromQL, ingest path, query tracker limitations)

• Testing

  • Verify adapter compatibility with official ClickHouse Grafana plugin (endpoint, auth, FORMAT, Content-Type)
  • Verify adapter compatibility with Altinity plugin
  • Integration test for forwarding path (unsupported query → real ClickHouse → response)

---

Suggested order: fix protocol gaps → resolve ingest design → deployment scaffold + README → query tracker extension

[milindsrivastava1997]

feat: ClickHouse-Grafana drop-in (clickhouse-dropin/)

Goal: A self-contained docker-compose setup that transparently accelerates an existing ClickHouse-Grafana stack — same user experience as asap-dropin/ for Prometheus.

Design: ASAPQuery sits in front of ClickHouse, impersonating the ClickHouse HTTP interface so Grafana's datasource plugin points to ASAP instead. Unsupported queries are forwarded to the real ClickHouse. Users update one Grafana datasource URL; nothing else changes.

---

Protocol / adapter gaps (blockers)

1. Endpoint URL mismatch — The biggest compatibility blocker. Grafana's ClickHouse plugins (both official and Altinity) send queries to the root / endpoint (http://ch:8123/?query=...). ASAPQuery currently exposes /clickhouse/query. The adapter's get_query_endpoint() must return / (or a configurable path) to be compatible without requiring users to change anything in their plugin config.

2. Authentication passthrough — The adapter does not inspect or forward X-ClickHouse-User / X-ClickHouse-Key headers that the official plugin sets, nor Basic Auth that the Altinity plugin uses. Currently no-ops; forwarded queries to real ClickHouse would fail auth.

3. database URL parameter — Both plugins encode the database as a URL query param (?database=default). The adapter doesn't read or thread it through.

4. FORMAT clause stripping — Grafana plugins append FORMAT TabSeparated (or FORMAT JSONCompact) to every query. The adapter passes the query through unmodified — the forwarding path to real ClickHouse is fine, but the sketch execution path would need to strip (or handle) the FORMAT clause before executing.

5. Response Content-Type header — The real ClickHouse sets Content-Type: text/plain; charset=UTF-8 for TabSeparated responses. Some plugins check this; the adapter doesn't set it explicitly.

6. Time range is ignored — The adapter always uses SystemTime::now() and ignores any time bounds sent by the client. For Grafana dashboard panels querying historical ranges, this will return wrong data. ClickHouse SQL queries encode time via WHERE timestamp BETWEEN ? AND ? in the query string itself — ASAP needs to parse or at least respect this.

7. Hardcoded adapter initialization — docs/03-how-to-guides/operations/manual-stack-run-clickhouse.md acknowledges that using the ClickHouse adapter currently requires editing main.rs. The adapter must be selectable purely from engine_config.yaml.

---

Data ingest (open design question)

For Prometheus: Prometheus pushes data via remote_write. ClickHouse has no equivalent push mechanism. The dropin needs to define an ingest path. Options to evaluate:

• Option A — Polling ingest: ASAPQuery periodically SELECTs new rows from a ClickHouse table. No infrastructure changes for user. Needs a new ingest driver.
• Option B — Kafka tap: User sets up a Kafka connector from ClickHouse; ASAP reads the same Kafka topic the benchmark pipeline uses. Reuses existing Kafka ingest driver but adds infrastructure complexity.
• Option C — Query-driven ingest: On first query observation, ASAP fetches historical data from ClickHouse for the relevant time window and streams it into sketches. Lazier but avoids a background poller.

Decision needed before the dropin can be implemented. Option A seems most analogous to the Prometheus case (pull-based, no extra infra).

---

Query tracker for ClickHouse (in scope)

Currently hard-gated on Prometheus in engine_config.rs:36-38 because LocalPlannerClient::plan() calls build_schema_from_prometheus() to discover metric labels. To support ClickHouse:

8. Implement build_schema_from_clickhouse() — Query information_schema.columns (or system.columns) to discover column names and types for tables being queried. ClickHouse exposes this cleanly.

9. Extend planner to understand SQL query structure — Currently the planner works with PromQL metrics. For SQL it needs to identify: which table, which column is the time axis, which columns are dimensions (GROUP BY), which are aggregated values. May need a SQL parser or structured query hints.

10. Remove/relax the hard-coded Prometheus-only restriction once the schema discovery and planner extension are in place.

---

Deployment / docs

11. Create clickhouse-dropin/ directory with:

    • docker-compose.yml, docker-compose.dev.yml, docker-compose.override.yml
    • .env (CLICKHOUSE_URL, CLICKHOUSE_DATABASE, CLICKHOUSE_USER, CLICKHOUSE_PASSWORD, ASAP port)
    • config/engine_config.yaml with backend.type: clickhouse

12. README covering:

    • Prerequisites (existing ClickHouse instance + Grafana with ClickHouse plugin)
    • Networking modes (bare metal, Docker host network, Docker bridge)
    • .env configuration
    • Grafana datasource: change URL from http://clickhouse:8123 → http://asap:8088 (or whatever port)
    • Verification: how to confirm queries are hitting ASAP vs forwarded to ClickHouse
    • Difference from Prometheus dropin (SQL not PromQL; ingest path differs; query tracker needs manual config until Automatically generate ArroyoSketch UDAFs wtih Rust macros #9 is done)

13. Grafana provisioning configs — equivalent of asap-quickstart/config/grafana/provisioning/datasources/datasources.yml but for the ClickHouse plugin

---

Testing

14. Verify adapter compatibility with official ClickHouse Grafana plugin (endpoint, auth, FORMAT, Content-Type)
15. Verify adapter compatibility with Altinity plugin (uses different auth scheme)
16. Integration test for the forwarding path (unsupported query → real ClickHouse → response)

---

Suggested order

(14) audit/fix protocol gaps → resolve ingest design question → (11–12) dropin scaffold → (8–10) query tracker extension