feat(persistence,rest,hfs): FHIR Bulk Data Export ($export) — async kick-off, postgres-s3 multi-instance, Inferno v2.0.0

[aacruzgon]

---

FHIR Bulk Data Export

Implements the FHIR Bulk Data Access IG $export family (system / patient / group) end-to-end, per Discussion #104. Embedded single-instance (SQLite job state + local-FS output + in-process worker pool) is the zero-config default; a multi-instance topology (PostgreSQL job state + S3-compatible output with pre-signed download URLs) is selected at startup with no handler changes. Ships with an external smoke workflow that exercises both topologies on every run and an Inferno Bulk Data IG v2.0.0 conformance workflow against the full SMART Backend Services + Keycloak stack.

Why

Bulk export is the API population-health platforms, payer-provider exchanges, registries, and research/AI pipelines converge on. CRUD + search are not enough once a workload needs every Observation for every patient in a cohort — that's a data-engineering problem (long-running work, durable state, fileserver bandwidth, multi-instance fan-out) rather than a request/response one. The IG defines an asynchronous, manifest-based, NDJSON-over-HTTPS pattern; this PR ships it as a first-class HFS subsystem.

Changes

Persistence — new traits + types (helios-persistence)

• core/bulk_export.rs: extended ExportRequest (until / elements / include_associated_data / patient_refs); extended ExportManifest (deleted / link); new StartExportInput, RawExportManifest, RawManifestEntry, ExportJobMetadata, ExportFileMetadata, ExpiredExportRef. BulkExportStorage trait grows start_export(StartExportInput), get_export_manifest -> RawExportManifest, plus get_export_job_metadata /get_export_file_metadata /count_active_exports / list_expired_exports. GroupExportProvider grows get_group_members_with_periods (default impl + SQLite/Postgres overrides) so the _since-newly-added filter can read Group.member.period.start.
• core/bulk_export_output.rs: new ExportOutputStore trait + ExportPartKey (with embedded fencing_token), ExportPartWriter, FinalizedPart, DownloadUrl. Decouples where the bytes go from job state.
• core/bulk_export_worker.rs: new ExportClaimStrategy, fully-fenced ExportWorkerStorage (every mutation guarded by (worker_id, fencing_token); 0 affected rows ⇒ LeaseError::LeaseLost), BulkExportJobStore marker trait, DefaultExportWorker<Js, Dp, Os> runtime that drives a claimed job under its lease, applies _typeFilter/ _since / _until / _elements, resumes from persisted cursors, and honors since_newly_added=exclude. The worker now branches on (level, patient_refs): Patient + non-empty patient_refs delegates to fetch_patient_compartment_batch so POST /Patient/$export?patient=… actually scopes to those patients (previously it ignored the filter and returned every resource of each requested type).
• BulkExportError::LeaseLost variant.

Persistence — backends

• SQLite (backends/sqlite/): v7→v8 schema migration (lease columns + part_index/fencing_token on bulk_export_files + 0-based-sequential part_index backfill before the unique index); ExportClaimStrategy (process-local mutex), fenced ExportWorkerStorage, get_group_members_with_periods, nested-Group flattening with cycle guard. Patient-level export query parameter binding fixed.
• PostgreSQL (backends/postgres/): v7→v8 migration (ADD COLUMN IF NOT EXISTS + ROW_NUMBER() backfill); PostgresSkipLocked claim via SELECT … FOR UPDATE SKIP LOCKED; fenced ExportWorkerStorage; correct int4 / int8 bind sites for bulk_export_progress / bulk_export_files. Cursor timestamps are now parsed with DateTime::parse_from_rfc3339 and bound as DateTime so the wire type matches the inferred TIMESTAMPTZ — without this fix, every paginated export job failed on its second fetch_export_batch with a TEXT/TIMESTAMPTZ type-mismatch error.
• S3 (backends/s3/): removed BulkExportStorage impl + synchronous run_export_job (S3 is output-only — job state lives in SQLite/Postgres); kept ExportDataProvider; added stub Patient/GroupExportProvider. New S3OutputStore (multipart upload to MinIO/S3, pre-signed GET via new S3Api::presign_get over aws_sdk_s3::presigning::PresigningConfig).
• Local FS (backends/local_fs/): new LocalFsOutputStore (tokio::fs + .tmp→atomic rename, idempotent delete).
• MongoDB/Elasticsearch: stub ExportDataProvider/Patient/GroupExportProvider returning UnsupportedCapability.
• CompositeStorage: gains export_provider: Option set by with_full_primary (now bounded T: GroupExportProvider); delegates the three traits to the primary or returns UnsupportedCapability.

REST (helios-rest)

• bulk_export_auth.rs: ExportFileAuth trait + BearerScopeAuth default (ownership against job_owner_subject or system/* wildcard, system/{ResourceType}.rs scope check; None principal short-circuits when auth is disabled).
• handlers/bulk_export.rs: three route-specific kick-off wrappers (system_/patient_/group_export_kickoff_handler) over a shared kickoff_export; status / cancel / download. Parses repeated query params via url::form_urlencoded, validates _typeFilter (rejects result-control params), enforces SmartScopePolicy per requested resource type + Group, enforces the per-tenant cap via count_active_exports, builds StartExportInput with frozen kick-off metadata, assembles the wire ExportManifest from RawExportManifest + ExportOutputStore::download_url, runs the two-step output-then-job teardown on cancel, and emits audit events at every lifecycle step.
• state.rs: AppState gains Arc + Arc + Arc + Arc + with_bulk_export(...).
• config.rs: BulkExportConfig with the full HFS_BULK_EXPORT_* env surface and validation (rejects local-fs + requires_access_token=false).
• routing/fhir_routes.rs: routes registered before the /{resource_type} catch-all; adds ExportDataProvider + PatientExportProvider + GroupExportProvider to the router's S bound.
• lib.rs: new create_app_with_auth_and_bulk_export(storage: Arc<S>, …, BulkExportBundle) sharing the inner build_app.
• handlers/capabilities.rs: advertises $export system-level operations + per-resource Patient.$export / Group.$export + IG instantiates.
• handlers/compartment.rs: refactored to call helios_fhir::get_compartment_params.

Auth (helios-auth)

• New DisabledJtiCache — no-op JtiCache implementation that disables the JWT replay-protection cache. Re-exported from the crate root and selectable in HFS via the existing HFS_AUTH_JTI_BACKEND setting; lets deployments that don't require replay protection (or that handle it upstream) avoid a Redis/SQLite dependency.
• discovery.rs: SMART well-known metadata now advertises token_endpoint_auth_signing_alg_values_supported, code_challenge_methods_supported (S256), and adds authorization_code to grant_types_supported when an authorization endpoint is configured. Required by Inferno SMART App Launch IG STU2 / Backend Services discovery checks; without these, the SMART Backend Services Inferno group fails the well-known capability test before it can establish a bearer token.

helios-fhir

• lib.rs: free get_compartment_params(version, compartment_type, resource_type) dispatching per FhirVersion to the per-version generated lookups.

helios-fhirpath

• reference_key_functions.rs: drop a redundant & in a format! argument (clippy nit surfaced once the workspace was built with the bulk-export feature graph).

helios-hfs

• main.rs: switched ServerConfig::parse() → ::from_env() so #[arg(skip)] sub-structs (multitenancy, bulk_export) actually populate from env (pre-existing bug). New generic build_bulk_export helper supporting embedded (dedicated SqliteBackend job store + LocalFsOutputStore) and postgres-s3 (PostgresBackend + S3OutputStore); wired into start_sqlite and start_postgres. The embedded backend now create_dir_alls the parent of {HFS_BULK_EXPORT_OUTPUT_DIR}/bulk_export.db before opening the SQLite connection (without this, HFS exited on startup whenever the configured output dir hadn't been pre-created — broke CI smoke jobs that only mkdird RESULTS_DIR). spawn_export_workers launches HFS_BULK_EXPORT_WORKER_CONCURRENCY claim/run loops + a periodic cleanup task that pages list_expired_exports and runs the two-step teardown. Recognizes the disabled JTI backend.
• Cargo.toml: adds chrono.

Ops + docs

• docker/bulk-export/docker-compose.yml: HFS + Postgres + MinIO + Keycloak; the substrate for the manual Inferno workflow and multi-instance smoke.
• .github/workflows/bulk-export-smoke.yml (new): external smoke workflow that brings up HFS in both sqlite/local-fs and postgres/s3 topologies and runs the smoke runner against each on every push.
• crates/hfs/tests/bulk_export/run_external_bulk_export_smoke.sh (new, ~500 lines): end-to-end smoke runner —kick-off → status poll → manifest → file download → DELETE cancel → 404 verification. Header parsing uses grep-i instead of gawk-only IGNORECASE, so it works on mawk-based runners (without this, every smoke job silently dropped the Content-Location header and failed the kick-off step).
• .github/workflows/inferno-bulk-data.yml: rebuilt against the shared docker-compose stack; runs SMART Backend Services and Bulk Data Export Tests as two sequential test_runs (the suite ID can't be POSTed as a test_group_id — Inferno returns 422, "must be run as part of a group"); carries the smart_auth_info produced by the SMART group into the Export run so kick-offs are authenticated; passes the now-mandatory since_timestamp input (2000-01-01T00:00:00.000Z); seeded heart-rate Observation gains the vital-signs category so R4 profile validation passes; allows Inferno's 5-minute private_key_jwt assertion lifetime on the generated Keycloak client; treats the file-server TLS test as known-omitted in the HTTP-only CI setup; preserves the MinIO client alias across job steps. Suite + group identifiers are still read from kit source at runtime, not hard-coded.
• crates/auth/README.md: documents the new disabled JTI backend.
• CLAUDE.md: Bulk Data Export endpoint table, full env-var table with defaults, single-instance vs multi-instance recipes, behavior notes.
• crates/hfs/README.md: Bulk Data Export quick-start.
• ROADMAP.md: $export marked shipped; $bulk-submit (ingestion) called out as next.
• Cargo.lock: lettre bumped to address a security audit finding.
• codecov.yml: crates/hfs/src/main.rs excluded from coverage (binary entry point not reachable from unit/integration tests).

Testing

• cargo fmt --all — green
• cargo build (default) — green
• cargo clippy -p helios-persistence -p helios-rest -p helios-hfs --features R4,postgres,s3,mongodb,elasticsearch,audit --all-targets -- -D warnings (CLAUDE.md lint allow-list) — green
• cargo test -p helios-persistence --features R4 --lib bulk — 49 pass / 0 fail (incl. nested-Group cycle
• guard, stale-worker fencing, end-to-end DefaultExportWorker, v7→v8 duplicate-row backfill migration, since_newly_added=exclude filter)
• cargo test -p helios-persistence --features R4 --test sqlite_tests — adds a Patient-level export-without-_since integration test (~83 LOC) covering the bug fix
• cargo test -p helios-rest --features R4 --test bulk_export — 13 pass / 0 fail (full lifecycle, status mappings, _typeFilter validation, strict/lenient handling, capability statement, metadata-lookup failure paths, plus 5 new integration tests: POST kick-off with Parameters body, _since, invalid _since, _elements, valid _typeFilter)
• New unit-test coverage for BearerScopeAuth (5 cases: no-principal bypass, owner + scope, wildcard override, missing-read-scope rejection) and BulkExportConfig::validate() (6 cases — every error branch). Closes the codecov/patch gap (69.70 % → ≥ 75.74 %).
• cargo test -p helios-persistence --features postgres,R4 --test postgres_tests --postgres_integration::postgres_integration_export — 3 pass / 0 fail against a real Postgres testcontainer (claim SKIP LOCKED, stale-worker fencing across reclaim, count_active/list_expired). The export-claim test now serializes against a process-wide mutex so it doesn't race other postgres integration tests sharing the same container.
• RUN_MINIO_S3_TESTS=1 cargo test -p helios-persistence --features s3,R4 --test minio_s3_tests --test_minio_s3_output_store — 1 pass / 0 fail against MinIO (write → finalize → pre-signed GET → reader → idempotent delete)
• External smoke workflow (bulk-export-smoke.yml): runs on every push. Brings up HFS in sqlite/local-fs and postgres/s3 topologies, executes the full kick-off → poll → download → cancel → 404 lifecycle in each; both jobs green.
• Multi-instance smoke (manual): brought up Postgres + MinIO + two release/hfs instances on ports 8080/8081 sharing Postgres job state. Kicked off /Patient/$export against instance 1 (202 + Content-Location); polled the status URL on instance 2 (202 then 200 + manifest); manifest carried requiresAccessToken: false and a pre-signed AWS-SHA256 S3 URL pointing at MinIO; downloaded directly from MinIO (5 NDJSON lines); DELETE on instance 1 → 202; subsequent poll on instance 2 → 404.
• Inferno Bulk Data IG v2.0.0: cloned inferno-framework/bulk-data-test-kit, executed bundle exec inferno execute --suite bulk_data_v200 against the stack. Result: 16 leaf pass / 8 leaf fail / 44 skip; inferno execute exit 3. All 12 bulk-data export-side leaf tests passed (system / patient / group $export + Content-Location, capability advertisement on each level, cancel 202, post-cancel poll 404). The 8 leaf failures are SMART Backend Services (1.1.02, 1.2.02–1.2.05) and TLS (2.1.01) prerequisites — known-deferred environmental requirements that need a configured Keycloak realm with HFS_AUTH_ENABLED=true and HTTPS termination; both are wired into docker/bulk-export/ for production but were not enabled for this local run. Skips are dependent tests Inferno auto-skips when their kick-off chain is broken by a SMART/TLS skip.

Notes

• Migrations: schema bumps to v8 on both SQLite and PostgreSQL. Forward-only. The bulk_export_files.part_index backfill runs before the new unique index is created, so existing deployments with multiple file rows per (job, file_type, resource_type) upgrade cleanly. A focused migration test (test_migration_v7_to_v8_backfills_duplicate_file_rows) covers the duplicate-row case.
• Trait contract changes: BulkExportStorage::start_export and get_export_manifest signatures changed; in-tree backends (SQLite, Postgres, S3) are updated. External downstream impls of BulkExportStorage will need to adopt StartExportInput and RawExportManifest.
• S3 backend posture: S3 is now output-only for bulk export. Its BulkExportStorage impl was removed; only ExportDataProvider remains. Job state must live in SQLite (HFS_BULK_EXPORT_BACKEND=embedded) or PostgreSQL (postgres-s3). An HFS_STORAGE_BACKEND=s3 deployment now picks up bulk export through the embedded SQLite job store with no additional config.
• Auth: bulk export endpoints sit inside the existing auth middleware. BearerScopeAuth validates download requests against the job owner subject or a system/*.rs wildcard. With HFS_AUTH_ENABLED=false (default), enforcement is bypassed — matching the rest of HFS. The new disabled JTI backend lets deployments that don't need replay protection skip Redis/SQLite for the JTI cache; SMART discovery additions (token_endpoint_auth_signing_alg_values_supported, S256, authorization_code) are required for Inferno SMART Backend Services / STU2 conformance.
• Patient-level export filter: POST /Patient/$export?patient=Patient/123 now actually scopes to the listed patient compartments. Before this fix, the patient_refs field on ExportRequest was populated but never consulted by the worker, so every resource of every requested type was returned.
• Postgres cursor binding: the keyset cursor's RFC 3339 timestamp is now parsed and bound as DateTime in fetch_export_batch / fetch_patient_compartment_batch. Without this, the second batch of any paginated export against PostgreSQL failed with a TEXT/TIMESTAMPTZ type-mismatch — which only ever showed up on jobs large enough to exceed HFS_BULK_EXPORT_BATCH_SIZE.
• Pre-existing config fix: main() now uses ServerConfig::from_env() instead of ::parse(). This was needed because multitenancy and bulk_export are #[arg(skip)] for clap and were therefore never populated from env in the binary; previously HFS_TENANT_* env vars also weren't fully reaching the binary through this code path. Behavior change: env-derived multitenancy + bulk-export config now actually applies.
• Embedded job-store path: {HFS_BULK_EXPORT_OUTPUT_DIR}/bulk_export.db is created on demand — the bootstrap now create_dir_alls the parent before opening SQLite, rather than requiring callers to pre-create it.
• Smoke runner portability: header parsing in run_external_bulk_export_smoke.sh uses grep -i | sed | tr -d '\r' instead of awk … IGNORECASE=1 so it runs on mawk (default awk on the self-hosted runners) as well as gawk.
• Inferno workflow: workflow_dispatch-only (matches the existing inferno-us-core.yml / inferno-subscription.yml). The local execution above uses HTTP without auth; CI runs against the full docker/bulk-export/ stack with Keycloak + HTTPS. The workflow now POSTs SMART Backend Services and Bulk Data Export Tests as two sequential test_runs (Inferno rejects the suite ID as a test_group_id), passes smart_auth_info from the first into the second so kick-offs are authenticated, supplies the now-mandatory since_timestamp, and seeds an Observation that satisfies the Heart Rate profile. Suite ids are read from kit source so they don't drift.
• since_newly_added=exclude uses Group.member.period.start to filter "patients added after _since". Default is include (return everything).
• Worker concurrency / leasing: defaults to 2 workers per pod, 60-second leases with 20-second heartbeats; tunable via HFS_BULK_EXPORT_*. Stale-worker fencing is verified by integration tests on both SQLite and Postgres.
• Dependency bump: lettre updated in Cargo.lock to clear a security audit finding.

Implements Discussion #104.

[smunini]

Should not be documented here - should be in the export README.md

[aacruzgon]

I have fixed this issue

[smunini]

Is this provided as an example, or is it used for the GitHub Action workflow tests? If only for workflow tests, we can move this comment to the export README.md and not feature it on the hfs README.md. In the future, we will be providing a library of common configuration examples.

[aacruzgon]

Yeah it is a provided example, is not part of the GitHub Action workflow test. I have address the documentation for this accordingly.

[smunini]

Is this still required - dead code?

[aacruzgon]

I have fixed this issue

[smunini]

#[allow(dead_code)] - check these in this file

[aacruzgon]

I have fixed this issue

[smunini]

#[allow(dead_code)] - check this

[aacruzgon]

I have fixed this issue

[smunini]

#[allow(dead_code)] - check

[aacruzgon]

I have fixed this issue

[smunini]

For these backend combinations - why are they unsupported or endpoint-unavailable? Seems like we should be able to support $export on all of the backend types with the exception of s3-only. For s3-only, we should be able to support these $export parameters, but the others will not be feasible without a lot of extra filtering logic that doesn't make much sense to implement at the moment. _outputFormat, _type, _elements, patient, includeAssociatedData, organizeOutputBy, allowPartialManifests

[aacruzgon]

I have corrected this issue

[smunini]

We don't yet support $validate - why was this added?

[aacruzgon]

The resource validate already existed before my PR, all I did was refactor the inline operation into what you see right now so it could conditionally append Bulk Data operations: export, patient-export, group-export, Bulk Data instantiates when HFS_BULK_EXPORT_ENABLED=true.

[smunini]

Good start!! See comments.