Skip to content

dotnet: PostgresLogStore — first per-domain analytics impl on PG (HOL-29)#109

Merged
BrewingCoder merged 1 commit into
mainfrom
issue-29-pg-logstore
May 9, 2026
Merged

dotnet: PostgresLogStore — first per-domain analytics impl on PG (HOL-29)#109
BrewingCoder merged 1 commit into
mainfrom
issue-29-pg-logstore

Conversation

@BrewingCoder
Copy link
Copy Markdown
Owner

Summary

First Postgres-backed implementation of HoldFast.Analytics.ILogStore. Mirrors ClickHouseService's six log methods with backend-equivalent semantics — same cursor format, same query-filter shape, same pagination pattern. Sets the implementation pattern for HOL-30..33.

What this PR adds

  • analytics.logs table (migration 0010) — TimescaleDB hypertable with daily chunks + 30-day drop_chunks retention policy. Conditional via pg_extension lookup so it degrades to a regular table if TimescaleDB isn't installed.
  • analytics.log_keys / analytics.log_key_values catalogs (migration 0011) — inline-upserted from PostgresLogStore.WriteLogsAsync rather than maintained by a materialized view (CH used SummingMergeTree + MV). Immediate consistency, no refresh policy. Future PR can swap to TimescaleDB continuous aggregates if write volume warrants.
  • PostgresLogStore.cs — direct Npgsql, no EF Core on the high-volume insert path. Bulk inserts via NpgsqlBinaryImporter (binary COPY).
  • Per-domain DI swap in Program.cs: Storage:Analytics:LogStore = postgres binds ILogStore to PostgresLogStore; default stays ClickHouse. Per-store granularity (rather than the single-knob Storage:Analytics planned for HOL-34) lets each impl land + be verified independently across HOL-29..33.

Behavior parity with CH impl

  • Cursor encoding/decoding via HoldFast.Analytics.CursorHelper — cursors round-trip across backend switches
  • CountLogsAsync returns 0L (matches CH stub; both backends will get a real impl together)
  • ReadLogsHistogramAsync buckets by severity_text over N=48 equal-width time slices
  • query filter: body ILIKE %q% OR log_attributes->>'service_name' ILIKE %q% (CH does the same map-key lookup; PG uses JSONB ->>)

Test plan

  • dotnet build -c Release — 0 errors / 0 new warnings
  • dotnet test3,079 pass (was 3,061; +18 new)
    • 14 unit tests for ClampLimit + DateOnlyOf edge cases (negative limits, overflow, mixed DateTimeKind, default sentinels)
    • 4 live integration tests against localhost:5432, gated [Trait("Category","PgIntegration")] + Skip.IfNot so CI without a DB sidecar passes:
      • WriteLogs + ReadLogs round-trip with JSONB attributes — body, severity, service info, attribute dictionary all survive the binary-COPY → query path
      • log_keys / log_key_values populated correctly — distinct values per key match expected
      • Empty/null attributes don't crash inline upsert — defensive null-handling on LogAttributes
      • Counts aggregate across batches via ON CONFLICT DO UPDATE — verified 3+2+1=6 across two WriteLogs calls
  • Migrations applied cleanly to the running timescale/timescaledb-ha:pg16 container — analytics.logs is a hypertable per timescaledb_information.hypertables

Out of scope

  • The DI swap is per-domain; HOL-34 will consolidate to Storage:Analytics = ClickHouse | Postgres once HOL-30..33 land
  • Smoke ingest with Storage:Analytics:LogStore=postgres set on the running backend image — the integration tests prove the impl works against live PG; full end-to-end via the OTLP ingest pipeline waits until HOL-34's image-level switch

Stats

7 files changed, +1023 / -5.

Closes HOL-29. Second story of the Postgres-backend EPIC (HOL-28).

🤖 Generated with Claude Code

…-29)

First Postgres-backed implementation of HoldFast.Analytics.ILogStore.
Mirrors ClickHouseService's six log methods with backend-equivalent
semantics: same cursor format (RFC3339+UUID base64), same query-filter
shape (body ILIKE + service_name attribute lookup), same pagination
pattern (over-fetch by 1 to detect HasNextPage).

What this PR adds:

- analytics.logs table (PG migration 0010) — TimescaleDB hypertable
  with daily chunks, 30-day retention via add_retention_policy.
  Conditional `CREATE EXTENSION` already lives in 0003 — if TS isn't
  installed the table degrades to a regular PG relation and retention
  falls back to in-app DELETE (handled by DataRetentionWorker later).
- analytics.log_keys + analytics.log_key_values catalogs (migration
  0011) — inline-upserted from PostgresLogStore.WriteLogsAsync rather
  than maintained by a materialized view (CH used SummingMergeTree+MV).
  Trade-off: more work per insert, immediate consistency, no refresh
  policy. Hobby scale handles this fine; future PR can swap to
  TimescaleDB continuous aggregates if write volume warrants.
- PostgresLogStore.cs — direct Npgsql, no EF Core (kept ILogStore on
  the high-volume insert path). Bulk inserts via NpgsqlBinaryImporter
  (binary COPY) — significantly faster than per-row INSERT for
  batches > ~50 rows.
- Per-domain DI swap in Program.cs: Storage:Analytics:LogStore =
  postgres binds ILogStore to PostgresLogStore; default stays CH.
  This is the per-store version of HOL-34's planned single-knob
  Storage:Analytics switch — keeping it per-domain for HOL-29..33 lets
  each impl land + verify independently.
- 14 unit tests for ClampLimit + DateOnlyOf edge cases (negative,
  zero, overflow, mixed DateTimeKind, default sentinel).
- 4 live integration tests for the read/write paths against a real PG
  on localhost:5432, gated by [Trait("Category","PgIntegration")] +
  Skip.IfNot so CI without a DB sidecar still passes:
    - WriteLogs + ReadLogs round-trip with JSONB attributes
    - log_keys / log_key_values populated correctly
    - Empty/null attributes don't crash inline upsert
    - Counts aggregate across batches via ON CONFLICT DO UPDATE

Behavior parity with the CH impl:
- CountLogsAsync returns 0L (matches CH stub)
- ReadLogsHistogramAsync buckets by SeverityText
- Cursor decoding/encoding goes through HoldFast.Analytics.CursorHelper
  (same library both backends use, so cursors round-trip across switches)

Tests: 3,079 pass (was 3,061; +18 = 14 unit + 4 integration).

Refs HOL-28 / HOL-29.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@BrewingCoder BrewingCoder merged commit 41b3bf1 into main May 9, 2026
3 of 4 checks passed
@BrewingCoder BrewingCoder deleted the issue-29-pg-logstore branch May 9, 2026 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant