ci: run @cipherstash/bench EXPLAIN-shape tests in CI to catch index-engagement regressions

[coderdan]

Summary

@cipherstash/bench (introduced in #424) is currently dev-run only — its scripts are named test:local / bench:local so the repo's default pnpm test (turbo over all packages) skips it. That keeps the existing CI green but means we lose regression coverage for the very class of bug the bench was created to catch.

We should add a dedicated CI job that exercises the EXPLAIN-shape assertions on every PR.

Why a separate job

The existing Run Tests job runs all package tests in one step. The bench needs:

• A Postgres instance with EQL installed (and the eql_v2.encrypted_operator_class operator family — which the existing remote DATABASE_URL secret may or may not have, and which the test suite's other packages don't currently need).
• The CipherStash workspace credentials (already in repo secrets).
• A 10k-row encrypted seed (~30–60s on first run; idempotent on repeat).

Slotting this into the existing Run Tests step would slow every CI run for everyone. Cleaner as a parallel job.

Proposed shape

bench-regression:
  name: Bench regression (index engagement)
  runs-on: blacksmith-4vcpu-ubuntu-2404
  services:
    postgres:
      image: ghcr.io/cipherstash/postgres-with-eql:latest   # or build locally each run
      ports: ['5432:5432']
      env:
        POSTGRES_DB: cipherstash
        POSTGRES_USER: cipherstash
        POSTGRES_PASSWORD: password
      options: >-
        --health-cmd pg_isready
        --health-interval 10s
        --health-timeout 5s
        --health-retries 5
  env:
    CS_WORKSPACE_CRN: \${{ secrets.CS_WORKSPACE_CRN }}
    CS_CLIENT_ID: \${{ secrets.CS_CLIENT_ID }}
    CS_CLIENT_KEY: \${{ secrets.CS_CLIENT_KEY }}
    CS_CLIENT_ACCESS_KEY: \${{ secrets.CS_CLIENT_ACCESS_KEY }}
    DATABASE_URL: postgres://cipherstash:password@localhost:5432/cipherstash
    BENCH_ROWS: 1000   # smaller for CI speed; planner picks the same plan shape
  steps:
    - uses: actions/checkout@v6
    - uses: pnpm/action-setup@v6.0.3
    - uses: actions/setup-node@v6
      with: { node-version: 22, cache: pnpm }
    - run: pnpm install --frozen-lockfile
    - run: pnpm -F @cipherstash/bench db:setup
    - run: pnpm -F @cipherstash/bench test:local

Open design questions

1. EQL image: do we publish cipherstash/postgres-with-eql to GHCR, or build the existing local/Dockerfile per-run in CI? Per-run build is simpler but adds ~30–60s per CI run. A published image is one more thing to maintain.
2. Fork PRs: secrets aren't available to PRs from forks by default. The bench job should skip cleanly if secrets.CS_WORKSPACE_CRN is empty (rather than fail). Match the pattern in e2e-tests.
3. Row scale: 1k rows is plenty for plan-shape assertions (the planner picks the index based on stats, which ANALYZE populates regardless of size). Keep wall-clock benches (__benches__/) out of CI.
4. Caching: 10k seed is slow; could cache the encrypted fixture as a Postgres dump in CI to speed up. Out of scope for v1.

Related

• perf(bench): add @cipherstash/bench for index-engagement validation #424 — introduced the bench (currently dev-run only).
• perf(stack/drizzle): wrap eq/ne/inArray/notInArray in eql_v2.hmac_256(...) #425 — first regression-prone fix the bench was designed to catch.