Skip to content

Fix slow health summary endpoint#369

Merged
Wiesenwischer merged 3 commits intomainfrom
bugfix/health-summary-performance
May 5, 2026
Merged

Fix slow health summary endpoint#369
Wiesenwischer merged 3 commits intomainfrom
bugfix/health-summary-performance

Conversation

@Wiesenwischer
Copy link
Copy Markdown
Owner

Summary

The /api/health/{environmentId} endpoint became progressively slower as the HealthSnapshots table grew on long-running instances (observed on test-ux-dokker: skeleton loaders never resolved).

Root cause: the raw SQL in GetLatestForEnvironment wrapped EnvironmentId in UPPER(), which prevents SQLite from using the index and forces a full table scan on every request. Compounded by the absence of a composite index covering the (env, deployment, latest) access pattern, and a 30-day default retention that lets the table grow large.

Changes

  • Remove UPPER() / ToUpperInvariant() from GetLatestForEnvironment, RemoveForDeployment, and GetTransitions. Guid parameters are passed via FromSqlInterpolated so EF Core's standard Guid-to-TEXT conversion is used on both sides of the comparison.
  • Replace the single-column EnvironmentId index with a composite (EnvironmentId, DeploymentId, CapturedAtUtc) index that directly covers the "latest snapshot per deployment in environment" query.
  • New EF Core migration AddHealthSnapshotsCompositeIndex performs the index swap.
  • Lower default snapshot retention from 30 to 7 days. At a 30s collection interval this still yields ~20k snapshots per deployment, while the UI history view requests only the latest 50.

Test plan

  • dotnet build clean (0 errors, 0 new warnings)
  • Unit + integration tests for health snapshots / environment summary pass (84 tests)
  • Verify on test-ux-dokker that /health page loads quickly after deploy
  • Confirm migration applies cleanly to existing instances

…and adding composite index

The /api/health/{environmentId} endpoint became progressively slower as the
HealthSnapshots table grew on long-running instances. Root cause: the raw SQL
in GetLatestForEnvironment wrapped EnvironmentId in UPPER(), which prevented
SQLite from using the index and forced a full table scan on every request.

Changes:
- Remove UPPER() / ToUpperInvariant() from GetLatestForEnvironment,
  RemoveForDeployment, and GetTransitions; pass Guid parameters via
  FromSqlInterpolated so EF Core's standard Guid-to-TEXT conversion is used
  on both sides of the comparison.
- Replace the single-column EnvironmentId index with a composite
  (EnvironmentId, DeploymentId, CapturedAtUtc) index that covers the
  "latest snapshot per deployment in environment" query directly.
- Lower the default snapshot retention from 30 to 7 days. With a 30s
  collection interval this still yields ~20k snapshots per deployment,
  while the UI history view requests only the latest 50.
@github-actions github-actions Bot added bug Something isn't working documentation Improvements or additions to documentation labels May 5, 2026
@Wiesenwischer Wiesenwischer enabled auto-merge (squash) May 5, 2026 10:04
The CI run for the previous commit failed because:
- The migration used MigrationBuilder.DropIndex/CreateIndex, which emit plain
  DROP/CREATE INDEX without IF [NOT] EXISTS guards. On a baseline-skipped legacy
  database where the InitialCreate index was never materialised, this throws.
- The legacy-DB test fixture only created StackSources, so it was not a realistic
  stand-in for an EnsureCreated()-era database (which would contain the full
  baseline schema).

Changes:
- Switch the index swap to raw SQL with DROP INDEX IF EXISTS / CREATE INDEX
  IF NOT EXISTS, so the migration is safe regardless of whether the legacy
  baseline materialised the old single-column index.
- Extend CreateLegacyStackSourcesTable to also create a minimal HealthSnapshots
  table with the InitialCreate-era indexes, so MigrationBaselineTests exercise
  a realistic legacy schema.
@Wiesenwischer Wiesenwischer merged commit 8934d4c into main May 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant