Tracking: v0.3.0 — hardening, rate limiting, observability

[haydenk]

Tracking issue for the v0.3.0 milestone.

Goal: Production hardening, rate limiting, observability, integration tests.

Turns v0.2.0 into something that can be exposed publicly and operated with confidence: security middleware, concurrency correctness, performance at full data volume, and metrics.

Ordered in rough working sequence. Phase 1 is deliberately a single issue because it can invalidate several items in Phase 2.

Phase 1 — Make the architectural call before touching the ingest hot path

• Investigate DuckDB for CSV ingestion in the ETL #43 — Investigate DuckDB for CSV ingestion in the ETL (prototype first — may close CSV parser flexible mode silently discards malformed rows #29, CSV header layout is hardcoded and never validated against actual file headers #32, Generated normalized_address expression recalculated on every row update #35 and reshape Refactor block_in_place and block_on pattern in ETL loader #18, Fix FUL deactivation memory scaling #25, Document lifecycle state transitions are not atomic (no transaction wrapping) #31; revisit the backlog before starting any of the items below)

Phase 2 — Ingest performance & correctness

Scope of each depends on the outcome of #43. Re-evaluate before starting.

• Fix FUL deactivation memory scaling #25 — Fix FUL deactivation memory scaling
• Generated normalized_address expression recalculated on every row update #35 — Generated normalized_address expression recalculated on every row update
• Concurrent ETL workers can process files out of publish order #34 — Concurrent ETL workers can process files out of publish order
• Inconsistent batch completion logging between downloader and ETL binaries #33 — Inconsistent batch completion logging between downloader and ETL binaries

Phase 3 — API hardening

Independent of the ingest work — can run in parallel with Phase 2.

• Make CORS policy configurable instead of hardcoded permissive #24 — Make CORS policy configurable instead of hardcoded permissive
• Add rate limiting middleware to the API #23 — Add rate limiting middleware to the API

Phase 4 — Observability capstone

Do last so the metrics surface reflects the final pipeline shape.

• No pipeline observability endpoint or metrics export #36 — No pipeline observability endpoint or metrics export

---

Auto-closes when every linked issue is closed. Progress is tracked by GitHub via the task list above.