Skip to content

feat(ci): add release-time bench workflow on self-hosted runner #94

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
be0x74a merged 1 commit into from
May 8, 2026
Merged

feat(ci): add release-time bench workflow on self-hosted runner #94

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 112 additions & 0 deletions .github/scripts/bench-history-summary.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
#!/usr/bin/env bash
# bench-history-summary.sh — emits the markdown body of the release-bench
# step summary (see `.github/workflows/bench.yml`). One big table per event
# shape (source-update, self-heal, ns-flip), keyed by profile, with one row
# per profile whether or not it actually exercised that shape (em-dash for
# missing).
#
# Args:
# $1: path to bench.json — a JSON array of per-profile reports (from
# `jq -s` over the harness's stdout).
# $2: label string used in the run header and the bench-history link.
#
# Output: markdown to stdout.
#
# Notes:
# - The script reads bench.json once into a single jq pass per metric. This
# avoids the O(profiles * metrics) re-reads that would happen with
# per-row jq invocations.
# - LC_ALL=C pins printf's number formatting so "1.5ms" doesn't render as
# "1,5ms" on a comma-decimal locale (lesson from bench-comment.sh).
# - Em-dash represents missing/zero values uniformly: a profile that
# doesn't exercise a shape, or a shape that recorded zero samples.

set -euo pipefail
export LC_ALL=C

if [ "$#" -lt 2 ]; then
echo "usage: $0 <bench.json> <label>" >&2
exit 2
fi

BENCH_JSON="$1"
LABEL="$2"

if [ ! -s "$BENCH_JSON" ]; then
echo "bench-history-summary: missing or empty $BENCH_JSON" >&2
exit 1
fi
if ! jq -e 'type == "array"' "$BENCH_JSON" >/dev/null 2>&1; then
echo "bench-history-summary: $BENCH_JSON is not a JSON array" >&2
exit 1
fi

# ms <ns-value> → "<X>ms" (one decimal) or "—" for zero/empty/null.
ms() {
local ns="${1:-}"
if [ -z "$ns" ] || [ "$ns" = "null" ] || [ "$ns" = "0" ]; then
echo "—"
else
awk -v ns="$ns" 'BEGIN { printf "%.1fms\n", ns / 1000000 }'
fi
}

# Cache profile names + total wall-clock once.
mapfile -t PROFILES < <(jq -r '.[].profile.Name' "$BENCH_JSON")
WALL_TOTAL=$(jq -r '[.[].duration_seconds] | add // 0 | floor' "$BENCH_JSON")
NUM_PROFILES=${#PROFILES[@]}

# field <profile-name> <jq-path> emits the raw value (or empty string if the
# profile doesn't exist or the field is missing).
field() {
local name="$1" path="$2"
jq -r --arg n "$name" --arg p "$path" '
.[] | select(.profile.Name == $n)
| (getpath($p | split(".")) // "") | tostring
' "$BENCH_JSON"
}

printf '# Bench: %s\n\n' "$LABEL"
printf 'Wall total: %ss • Profiles: %s\n\n' "$WALL_TOTAL" "$NUM_PROFILES"

# ── Source-update p99 ─────────────────────────────────────────────────────
printf '## Source-update p99 latency by profile and path\n\n'
printf '| Profile | NP | CP-sel slowest | CP-list slowest |\n'
printf '|---|---|---|---|\n'
for p in "${PROFILES[@]}"; do
np=$(ms "$(field "$p" measurements.e2e_np_source_update_p99_ns)")
sel=$(ms "$(field "$p" measurements.e2e_cp_sel_source_update_slowest_p99_ns)")
lst=$(ms "$(field "$p" measurements.e2e_cp_list_source_update_slowest_p99_ns)")
printf '| %s | %s | %s | %s |\n' "$p" "$np" "$sel" "$lst"
done
printf '\n'

# ── Self-heal p99 ─────────────────────────────────────────────────────────
printf '## Self-heal p99 latency by profile and path\n\n'
printf '| Profile | NP | CP-sel | CP-list |\n'
printf '|---|---|---|---|\n'
for p in "${PROFILES[@]}"; do
np=$(ms "$(field "$p" measurements.e2e_np_self_heal_p99_ns)")
sel=$(ms "$(field "$p" measurements.e2e_cp_sel_self_heal_p99_ns)")
lst=$(ms "$(field "$p" measurements.e2e_cp_list_self_heal_p99_ns)")
printf '| %s | %s | %s | %s |\n' "$p" "$np" "$sel" "$lst"
done
printf '\n'

# ── ns-flip p99 (CP-selector only) ────────────────────────────────────────
printf '## ns-flip p99 latency (CP-selector only)\n\n'
printf '| Profile | cleanup | add |\n'
printf '|---|---|---|\n'
for p in "${PROFILES[@]}"; do
cleanup=$(ms "$(field "$p" measurements.e2e_cp_sel_ns_flip_cleanup_p99_ns)")
add=$(ms "$(field "$p" measurements.e2e_cp_sel_ns_flip_add_p99_ns)")
printf '| %s | %s | %s |\n' "$p" "$cleanup" "$add"
done
printf '\n'

# Footer link points at the file the workflow's push step will create. Use
# GITHUB_REPOSITORY when available (CI) and fall back to the project repo
# for local dry-runs.
REPO="${GITHUB_REPOSITORY:-projection-operator/projection}"
printf 'Full distributions in the [bench-history JSON](https://github.com/%s/blob/bench-history/bench-history/%s.json) and the workflow artifact.\n' \
"$REPO" "$LABEL"
188 changes: 188 additions & 0 deletions .github/workflows/bench.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
name: Bench (release)

# Release-time bench: deploys the operator on a self-hosted Proxmox VM
# (label `bench-runner`), runs the full 8-profile matrix, and persists the
# resulting JSON to the `bench-history` orphan branch under
# `bench-history/<label>.json`. Manual trigger only — `pull_request` is
# deliberately omitted because self-hosted runners on public repos are
# attackable from fork-PR code (untrusted commits running on our infra).
# Operators trigger this from the Actions tab against a release tag, branch,
# or SHA.

on:
workflow_dispatch:
inputs:
ref:
description: 'Ref to bench (release tag like v0.4.0, branch, or SHA).'
required: true
label:
description: 'Override bench-history filename (default: derived from ref — vX.Y.Z for tags, sanitized-ref-sha7-ts otherwise).'
required: false
default: ''

# Top-level read-only; the bench job below adds `contents: write` for the
# bench-history orphan-branch push (least privilege).
permissions:
contents: read

jobs:
bench:
name: Bench full
runs-on: [self-hosted, bench-runner]
timeout-minutes: 240
permissions:
contents: write # bench-history orphan branch push
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: Checkout target ref
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
ref: ${{ inputs.ref }}
path: src

- name: Setup Go
uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
with:
go-version-file: src/go.mod
cache: true
cache-dependency-path: src/go.sum

- name: Create Kind cluster
uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc # v1.14.0
with:
cluster_name: bench-release
node_image: kindest/node:v1.32.0

- name: Build operator image
working-directory: src
run: make docker-build IMG=projection:bench

- name: Load image into Kind
run: kind load docker-image projection:bench --name bench-release

- name: Install CRDs
working-directory: src
run: make install

- name: Deploy operator
working-directory: src
run: make deploy IMG=projection:bench

- name: Wait for operator Deployment to be Ready
run: |
kubectl -n projection-system rollout status \
deploy/projection-controller-manager --timeout=180s

- name: Export Kind kubeconfig
id: kubeconfig
run: |
# The bench harness refuses ~/.kube/config for safety. Write to a
# dedicated path so the safety gate is satisfied.
KUBECONFIG_BENCH="$RUNNER_TEMP/bench-kubeconfig"
kind get kubeconfig --name bench-release > "$KUBECONFIG_BENCH"
echo "path=$KUBECONFIG_BENCH" >> "$GITHUB_OUTPUT"

- name: Run bench (full matrix)
id: bench
working-directory: src
env:
KUBECONFIG_BENCH: ${{ steps.kubeconfig.outputs.path }}
run: |
# The harness emits one JSON object per profile concatenated to
# stdout. Slurp into a single JSON array so downstream consumers
# (history files, summary script) can iterate uniformly. Doing this
# in the workflow avoids touching the bench harness.
go run ./test/bench \
--profile=full \
--kubeconfig="$KUBECONFIG_BENCH" \
--output=json \
> /tmp/bench-raw.json
jq -s '.' /tmp/bench-raw.json > /tmp/bench.json

- name: Derive history label
id: label
env:
REF: ${{ inputs.ref }}
OVERRIDE: ${{ inputs.label }}
run: |
# Three cases:
# - explicit label override → use as-is
# - looks like a release tag (v<major>.<minor>.<patch>[...]) → ref
# as-is so v0.4.0 / v1.0.0-rc1 file as themselves
# - branch or SHA → sanitize + sha7 + utc timestamp so two runs
# against the same branch don't clobber each other
if [ -n "$OVERRIDE" ]; then
label="$OVERRIDE"
elif [[ "$REF" =~ ^v[0-9]+\.[0-9]+\.[0-9]+ ]]; then
label="$REF"
else
sanitized=$(printf '%s' "$REF" | tr '/' '-' | tr -cd '[:alnum:]-')
sha7=$(git -C src rev-parse --short=7 HEAD)
ts=$(date -u +%Y-%m-%dT%H-%M-%SZ)
label="${sanitized}-${sha7}-${ts}"
fi
echo "label=$label" >> "$GITHUB_OUTPUT"

- name: Render step summary
env:
LABEL: ${{ steps.label.outputs.label }}
run: |
src/.github/scripts/bench-history-summary.sh /tmp/bench.json "$LABEL" \
>> "$GITHUB_STEP_SUMMARY"

- name: Push to bench-history orphan branch
env:
LABEL: ${{ steps.label.outputs.label }}
REPO: ${{ github.repository }}
run: |
# Self-bootstrap: if `bench-history` doesn't exist on origin, init
# an orphan branch with a README; otherwise fetch and append. Using
# a fresh temp checkout keeps the worktree at `src/` clean and
# avoids accidentally touching tracked files on the source branch.
set -euo pipefail
WORK=$(mktemp -d)
cd "$WORK"
git init -q
git remote add origin "https://github.com/${REPO}.git"
gh auth setup-git

if git ls-remote --exit-code origin bench-history >/dev/null 2>&1; then
git fetch -q origin bench-history
git checkout -q FETCH_HEAD
else
git checkout --orphan bench-history
git rm -rf . 2>/dev/null || true
cat > README.md <<'EOF'
# Bench history

Machine-generated by `.github/workflows/bench.yml`. One JSON file
per release run, named by the `label` input (defaults to the
release tag). Each file is a JSON array of per-profile bench
reports.
EOF
git add README.md
git -c user.name="github-actions[bot]" \
-c user.email="41898282+github-actions[bot]@users.noreply.github.com" \
commit -q -m "chore: initialize bench-history"
fi

mkdir -p bench-history
cp /tmp/bench.json "bench-history/${LABEL}.json"
git add "bench-history/${LABEL}.json"
git -c user.name="github-actions[bot]" \
-c user.email="41898282+github-actions[bot]@users.noreply.github.com" \
commit -q -m "bench: ${LABEL}"
git push origin HEAD:bench-history

- name: Upload bench artifact
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: bench-full-${{ steps.label.outputs.label || 'unlabeled' }}
path: /tmp/bench.json
if-no-files-found: ignore

- name: Cleanup Kind cluster
if: always()
run: kind delete cluster --name bench-release || true
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **Release-time bench workflow** (`.github/workflows/bench.yml`). Manually triggered (`workflow_dispatch`) against a release tag, branch, or SHA; runs the full 8-profile bench matrix on a self-hosted runner (label `bench-runner`) and persists the resulting JSON to the `bench-history` orphan branch under `bench-history/<label>.json`. The orphan branch is self-bootstrapped on first run. A markdown table of source-update / self-heal / ns-flip p99 latencies by profile is rendered into the workflow's step summary, and the bench JSON is uploaded as an artifact. `pull_request` triggers are deliberately omitted — self-hosted runners on public repos are exposed to fork-PR malicious code, so this workflow is operator-driven only. Per-PR shape-break smoke coverage continues to run on `ubuntu-22.04` via `bench-smoke.yml`.

### Fixed

- A `Projection` or `ClusterProjection` whose source object was never created now reports `SourceResolved=False, reason=SourceNotFound, message="source X/Y not found"`. Previously every source-NotFound case was bucketed as `reason=SourceDeleted` with the message `"source X/Y has been deleted"` — accurate when the source had previously existed and was deleted, but a lie when the source never existed in the first place. The two cases are now distinguished by `status.destinationName`: empty (never resolved) → `SourceNotFound`; populated (we previously projected it) → `SourceDeleted`. The `SourceDeleted` reason value is unchanged for the genuine deletion case; alerts on `Ready=False` continue to fire for both reasons.
Expand Down