dotnet: PostgresLogStore — first per-domain analytics impl on PG (HOL-29)#109
Merged
Conversation
…-29)
First Postgres-backed implementation of HoldFast.Analytics.ILogStore.
Mirrors ClickHouseService's six log methods with backend-equivalent
semantics: same cursor format (RFC3339+UUID base64), same query-filter
shape (body ILIKE + service_name attribute lookup), same pagination
pattern (over-fetch by 1 to detect HasNextPage).
What this PR adds:
- analytics.logs table (PG migration 0010) — TimescaleDB hypertable
with daily chunks, 30-day retention via add_retention_policy.
Conditional `CREATE EXTENSION` already lives in 0003 — if TS isn't
installed the table degrades to a regular PG relation and retention
falls back to in-app DELETE (handled by DataRetentionWorker later).
- analytics.log_keys + analytics.log_key_values catalogs (migration
0011) — inline-upserted from PostgresLogStore.WriteLogsAsync rather
than maintained by a materialized view (CH used SummingMergeTree+MV).
Trade-off: more work per insert, immediate consistency, no refresh
policy. Hobby scale handles this fine; future PR can swap to
TimescaleDB continuous aggregates if write volume warrants.
- PostgresLogStore.cs — direct Npgsql, no EF Core (kept ILogStore on
the high-volume insert path). Bulk inserts via NpgsqlBinaryImporter
(binary COPY) — significantly faster than per-row INSERT for
batches > ~50 rows.
- Per-domain DI swap in Program.cs: Storage:Analytics:LogStore =
postgres binds ILogStore to PostgresLogStore; default stays CH.
This is the per-store version of HOL-34's planned single-knob
Storage:Analytics switch — keeping it per-domain for HOL-29..33 lets
each impl land + verify independently.
- 14 unit tests for ClampLimit + DateOnlyOf edge cases (negative,
zero, overflow, mixed DateTimeKind, default sentinel).
- 4 live integration tests for the read/write paths against a real PG
on localhost:5432, gated by [Trait("Category","PgIntegration")] +
Skip.IfNot so CI without a DB sidecar still passes:
- WriteLogs + ReadLogs round-trip with JSONB attributes
- log_keys / log_key_values populated correctly
- Empty/null attributes don't crash inline upsert
- Counts aggregate across batches via ON CONFLICT DO UPDATE
Behavior parity with the CH impl:
- CountLogsAsync returns 0L (matches CH stub)
- ReadLogsHistogramAsync buckets by SeverityText
- Cursor decoding/encoding goes through HoldFast.Analytics.CursorHelper
(same library both backends use, so cursors round-trip across switches)
Tests: 3,079 pass (was 3,061; +18 = 14 unit + 4 integration).
Refs HOL-28 / HOL-29.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First Postgres-backed implementation of
HoldFast.Analytics.ILogStore. MirrorsClickHouseService's six log methods with backend-equivalent semantics — same cursor format, same query-filter shape, same pagination pattern. Sets the implementation pattern for HOL-30..33.What this PR adds
analytics.logstable (migration 0010) — TimescaleDB hypertable with daily chunks + 30-daydrop_chunksretention policy. Conditional viapg_extensionlookup so it degrades to a regular table if TimescaleDB isn't installed.analytics.log_keys/analytics.log_key_valuescatalogs (migration 0011) — inline-upserted fromPostgresLogStore.WriteLogsAsyncrather than maintained by a materialized view (CH usedSummingMergeTree + MV). Immediate consistency, no refresh policy. Future PR can swap to TimescaleDB continuous aggregates if write volume warrants.PostgresLogStore.cs— direct Npgsql, no EF Core on the high-volume insert path. Bulk inserts viaNpgsqlBinaryImporter(binary COPY).Program.cs:Storage:Analytics:LogStore = postgresbindsILogStoretoPostgresLogStore; default stays ClickHouse. Per-store granularity (rather than the single-knobStorage:Analyticsplanned for HOL-34) lets each impl land + be verified independently across HOL-29..33.Behavior parity with CH impl
HoldFast.Analytics.CursorHelper— cursors round-trip across backend switchesCountLogsAsyncreturns0L(matches CH stub; both backends will get a real impl together)ReadLogsHistogramAsyncbuckets byseverity_textover N=48 equal-width time slicesqueryfilter:body ILIKE %q%ORlog_attributes->>'service_name' ILIKE %q%(CH does the same map-key lookup; PG uses JSONB->>)Test plan
dotnet build -c Release— 0 errors / 0 new warningsdotnet test— 3,079 pass (was 3,061; +18 new)ClampLimit+DateOnlyOfedge cases (negative limits, overflow, mixedDateTimeKind, default sentinels)localhost:5432, gated[Trait("Category","PgIntegration")]+Skip.IfNotso CI without a DB sidecar passes:log_keys/log_key_valuespopulated correctly — distinct values per key match expectedLogAttributesON CONFLICT DO UPDATE— verified 3+2+1=6 across twoWriteLogscallstimescale/timescaledb-ha:pg16container —analytics.logsis a hypertable pertimescaledb_information.hypertablesOut of scope
Storage:Analytics = ClickHouse | Postgresonce HOL-30..33 landStorage:Analytics:LogStore=postgresset on the running backend image — the integration tests prove the impl works against live PG; full end-to-end via the OTLP ingest pipeline waits until HOL-34's image-level switchStats
7 files changed, +1023 / -5.
Closes HOL-29. Second story of the Postgres-backend EPIC (HOL-28).
🤖 Generated with Claude Code