Skip to content

feat(quality): OpenAPI spec path coverage across both transports#222

Merged
vriesdemichael merged 1 commit into
mainfrom
claude/rest-api-coverage-analysis-cfp0g
Jun 4, 2026
Merged

feat(quality): OpenAPI spec path coverage across both transports#222
vriesdemichael merged 1 commit into
mainfrom
claude/rest-api-coverage-analysis-cfp0g

Conversation

@vriesdemichael
Copy link
Copy Markdown
Owner

Summary

Adds an automated OpenAPI spec path coverage report so unimplemented API surface is visible and tracked in CI.

The existing generated_contracts metric only counts generated-client calls of the form x.client.OpWithResponse(...). It is blind to services built on the raw internal/transport/httpclient — most notably the entire pullrequest service — so it under-reports real API coverage (it implies PR endpoints are ~0% covered when create/merge/decline/reopen/approve/auto-merge/participants are all implemented).

What this does

A new standalone -spec-coverage mode in tools/quality-report measures coverage at the (HTTP method, path) level, joining the Bitbucket 9.4 OpenAPI spec against operations reached through both transports:

  1. The generated typed client (internal/openapi/generated), restricted to the operations actually called from internal/services.
  2. The hand-rolled httpclient (GetJSON/PostJSON/...), with request paths resolved statically from the services source — handling string literals, fmt.Sprintf assembly, single-return path helpers, local path variables, and parameter-literal substitution (so the shared transition("merge"|"decline"|"reopen") helper resolves to its real endpoints).

Resolution is undercount-safe: unresolvable dynamic segments collapse to a non-matching placeholder rather than wildcard-matching, so the number is a floor, never inflated.

Deliverables

  • docs/quality/spec-coverage.json — committed, deterministic artifact: 112/546 (20.5%) with per-tag breakdown and the full list of unimplemented gaps (method, path, tag, summary). Does not depend on coverage profiles or live tests.
  • Task targets: quality:spec-coverage, quality:spec-coverage:update, quality:spec-coverage:verify.
  • CI: artifact verification wired into the CI-safe unit-tests job.
  • Unit tests for normalization, verb substitution, raw-path resolution (incl. parameter substitution), and generated-client WithBody parsing.
  • Docs: docs/quality/README.md and AGENTS.md updated with the regeneration workflow.

Verification

  • go test ./tools/quality-report/ — passing; gofmt/go vet clean.
  • Artifact round-trips: task quality:spec-coverage:verify passes against the committed file.
  • Spot-checked classification: raw PR endpoints (create/merge/decline/reopen/approve/auto-merge/assign-participant) count as covered; genuine gaps (delete PR, change participant status, merge-base, search participants) remain listed.

Resolves #208

https://claude.ai/code/session_012Ax4aEmr2XgmGi52Grzhk3


Generated by Claude Code

The existing generated_contracts metric only counts generated-client calls
(x.client.OpWithResponse(...)), so it is blind to services built on the raw
httpclient (notably the entire pullrequest service), under-reporting real API
coverage.

Add a standalone `-spec-coverage` mode to tools/quality-report that measures
coverage at the (HTTP method, path) level, joining the Bitbucket OpenAPI spec
against operations reached through:
  - the generated typed client (restricted to the used set), and
  - the hand-rolled httpclient (paths resolved statically from services source,
    including helper functions, fmt.Sprintf assembly, local path vars, and
    parameter-literal substitution for shared action helpers).

Writes a deterministic, committed docs/quality/spec-coverage.json with per-tag
coverage and the full list of unimplemented gaps. The artifact does not depend
on coverage profiles or live tests, so it is generated/verified independently
and gated in CI via the CI-safe unit-tests job.

Resolves #208
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 55.16304% with 165 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.31%. Comparing base (c213f55) to head (6fad3d1).

Files with missing lines Patch % Lines
tools/quality-report/speccoverage.go 56.38% 139 Missing and 18 partials ⚠️
tools/quality-report/main.go 0.00% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #222      +/-   ##
==========================================
- Coverage   83.90%   83.31%   -0.60%     
==========================================
  Files          80       81       +1     
  Lines       16624    17000     +376     
==========================================
+ Hits        13949    14164     +215     
- Misses       1933     2080     +147     
- Partials      742      756      +14     
Flag Coverage Δ
combined_scoped 83.31% <55.16%> (-0.60%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vriesdemichael vriesdemichael merged commit 357b22b into main Jun 4, 2026
6 checks passed
@vriesdemichael vriesdemichael deleted the claude/rest-api-coverage-analysis-cfp0g branch June 4, 2026 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

quality-report: measure path-level spec coverage across both transports

3 participants