Skip to content

GITHUB_ACTIONS.md

Latest commit

 

History

History
760 lines (573 loc) · 25.1 KB

GITHUB_ACTIONS.md

File metadata and controls

760 lines (573 loc) · 25.1 KB

GitHub Actions Reference

Last Updated: February 22, 2026 - Added SBOM generation (syft), GitHub attestation, package changelog & build history

This guide covers the automated workflows and actions used for container management in our production-ready CI/CD system.

🎯 Architecture Status: ✅ OPTIMIZED

  • 8 Workflows: All active and properly integrated
  • 7 Actions: All used across workflows
  • Core Scripts: Fully integrated with automation
  • Make Script: All functions utilized by workflows
  • Recent: SBOM generation (syft) + GitHub attestation, package changelog & build history, recreate-manifests workflow

Architecture Overview

Our GitHub Actions system uses a hybrid approach combining direct workflow calls for upstream changes with GitHub's native path filtering for code changes:

graph TD
    A[upstream-monitor.yaml<br/>📅 Schedule: 6AM UTC daily] -->|version changes| B[Matrix Strategy<br/>🔄 One call per container]
    B -->|workflow_call| C[auto-build.yaml<br/>🏗️ Build & Push]
    C -->|workflow_call| D[update-dashboard.yaml<br/>📊 GitHub Pages]
    
    E[Code Changes<br/>📝 Push/PR] -->|path filtering| C
    F[validate-version-scripts.yaml<br/>🧪 Quality Assurance] --> G[Standalone Testing]
    
    A --> H[check-upstream-versions<br/>📋 Version Detection]
    A --> I[update-version<br/>📝 File Updates]
    A --> J[close-duplicate-prs<br/>🔄 PR Management]
    
    C --> K[detect-containers<br/>🔍 Change Detection]
    C --> L[build-container<br/>🏗️ Build & Push]
    C --> M[setup-github-cli<br/>⚙️ CLI Config]
    
    F --> H
    
    style A fill:#e1f5fe
    style C fill:#f3e5f5
    style D fill:#e8f5e8
    style F fill:#fff3e0
Loading

Key Principles:

  • 🔄 Single Source of Truth: upstream-monitor is the only scheduled workflow
  • ⚡ Optimized Flow: Removed redundant version checking in build-container
  • 🎯 Hybrid Triggers: workflow_call for upstream changes, path filtering for code changes
  • 📋 Matrix Strategy: upstream-monitor calls auto-build once per updated container
  • 🛡️ Clean Separation: Each workflow has distinct, focused responsibilities
  • 🚀 Force Rebuild Trust: build-container trusts upstream-monitor decisions
  • Force Rebuild: Upstream changes use force_rebuild to ensure containers are updated

🔄 Workflows (All Active)

1. Upstream Monitor (upstream-monitor.yaml) - 📅 Primary Orchestrator

The core scheduler that monitors upstream sources and initiates the entire automation pipeline.

Status: ✅ Recently Fixed - JSON parsing errors resolved, matrix strategy optimized

Triggers:

  • 🕕 Schedule: 6 AM UTC daily (ONLY workflow with schedule triggers)
  • ⚙️ Manual: gh workflow run upstream-monitor.yaml

Responsibilities:

  • 🔍 Monitor upstream software versions using check-upstream-versions action
  • 📝 Create PRs for version updates using update-version action
  • 🔄 Manage duplicate PR cleanup using close-duplicate-prs action
  • 🚀 Trigger auto-build directly using optimized matrix strategy (one call per container)
  • 🔀 Auto-merge minor updates using gh pr checks --watch to poll CI status before merging with standard GITHUB_TOKEN (no elevated permissions needed — enforce_admins: false on master)

Key Inputs:

# Check specific container with debug
gh workflow run upstream-monitor.yaml \
  --field container=wordpress \
  --field debug=true

Workflow Chain:

graph LR
    A[Schedule/Manual] --> B[check-upstream-versions]
    B --> C[Matrix: create-update-prs]
    C --> D[update-version + close-duplicate-prs] 
    D --> E[Create PRs + Auto-merge minor]
    E --> F[Matrix: trigger-auto-build]
    F --> G[auto-build.yaml]
Loading
  1. Closes duplicate PRs using close-duplicate-prs action
  2. Triggers auto-build using matrix strategy (one call per container)

2. Auto Build (auto-build.yaml) - Build Orchestrator

Builds and pushes containers when triggered by upstream monitor or code changes.

Triggers:

  • workflow_call with matrix: From upstream-monitor (one call per updated container)
  • Push/PR Events: GitHub's native path filtering for container file changes
  • Manual: gh workflow run auto-build.yaml

Pipeline stages:

  1. detect-containers — Smart change detection via git diff or force input
  2. cache-base-images — Cache Docker Hub base images to GHCR (avoids rate limits)
  3. build-extensions — Build PostgreSQL extension images (if postgres detected)
  4. build-and-push — Multi-platform builds (native amd64 + arm64 runners) + SBOM generation + GitHub attestation
  5. create-manifest — Multi-arch manifest lists (GHCR primary, Docker Hub via cross-registry from GHCR sources)
  6. cache-lineage — Cache .build-lineage/ JSON artifacts + process SBOMs (changelog, build history)
  7. update-dashboard — Trigger dashboard regeneration

Features:

  • Native multi-architecture builds (separate amd64/arm64 runners, no QEMU)
  • Hybrid detection: Matrix for upstream changes, path filtering for code changes
  • Registry push automation via build-container action
  • Docker Hub manifests created using GHCR images as cross-registry sources
  • Build lineage tracking (base image digest, build args, timestamps)
  • SBOM generation via syft (SPDX JSON format, amd64 only — packages identical across arches)
  • GitHub attestation via actions/attest-sbom (Sigstore-signed supply chain compliance)
  • Package changelog — inter-version package diffs (added/removed/updated)
  • Build history — last 10 builds per variant with package counts

Scoped Build Inputs (optional):

  • scope_versions: Comma-separated major versions (e.g., "18"). Empty = all.
  • scope_flavors: Comma-separated flavors (e.g., "distributed,full"). Empty = all.
  • scope_extensions: Comma-separated extensions (e.g., "citus"). Empty = all.

When set, only matching builds are included in the matrix. Used automatically by upstream-monitor to avoid unnecessary rebuilds.

Usage:

# Build all containers manually
gh workflow run auto-build.yaml

# Force rebuild specific container
gh workflow run auto-build.yaml \
  --field container=wordpress \
  --field force_rebuild=true

# Rebuild containers without recompiling extensions (faster)
gh workflow run auto-build.yaml \
  --field container=postgres \
  --field force_rebuild=true \
  --field skip_extensions=true

# Scoped build: only PG 18 vector+full variants
gh workflow run auto-build.yaml \
  --field container=postgres \
  --field force_rebuild=true \
  --field scope_versions=18 \
  --field scope_flavors=vector,full

# Scoped build: rebuild only citus-affected flavors
gh workflow run auto-build.yaml \
  --field container=postgres \
  --field force_rebuild=true \
  --field scope_flavors=distributed,full \
  --field scope_extensions=citus

2b. Recreate Manifests (recreate-manifests.yaml) - Manifest-Only

Recreates multi-arch manifest lists without rebuilding containers. Use when manifests need to be regenerated (e.g., after a manifest fix, or to sync Docker Hub with GHCR).

Triggers:

  • Manual only: gh workflow run recreate-manifests.yaml

Inputs:

  • container (optional): Specific container, or all if empty
  • registry: both (default), ghcr, or dockerhub

Usage:

# Recreate Docker Hub manifests for all containers
gh workflow run recreate-manifests.yaml -f registry=dockerhub

# Recreate all manifests for postgres only
gh workflow run recreate-manifests.yaml -f container=postgres

# Recreate both GHCR + Docker Hub for all
gh workflow run recreate-manifests.yaml

3. Update Dashboard (update-dashboard.yaml) - Documentation Sync

Updates the repository dashboard and GitHub Pages after successful builds.

Triggers:

  • workflow_call: From auto-build.yaml (primary trigger)
  • Manual: For dashboard-only updates

Responsibilities:

  • Generate container status dashboard
  • Deploy updates to GitHub Pages
  • Maintain build history

4. Version Validation (validate-version-scripts.yaml) - Quality Assurance

Validates all version.sh scripts for functionality and standards compliance.

Triggers:

  • Push/PR: Changes to version.sh files
  • Manual: For testing and validation

Local Testing:

./validate-version-scripts.sh

Outputs:

  • Container update summary
  • Pull requests for version updates
  • Automatic build triggers

2. Auto Build (auto-build.yaml)

Builds and pushes containers when changes are detected.

Triggers:

  • Push to main/master (affecting container files)
  • Pull requests
  • workflow_call (from upstream-monitor)
  • Manual dispatch

Features:

  • Multi-architecture builds (amd64, arm64)
  • Smart change detection
  • Registry push automation
  • Build retry logic

Usage:

# Build all containers
gh workflow run auto-build.yaml

# Force rebuild specific container  
gh workflow run auto-build.yaml \
  --field container=wordpress \
  --field force_rebuild=true

3. Version Validation (validate-version-scripts.yaml)

Validates all version.sh scripts for functionality and standards compliance.

Triggers:

  • Changes to version.sh files
  • Manual dispatch

Local Testing:

./validate-version-scripts.sh

Dependency Matrix

Workflow → Action Dependencies

Workflow Actions Used Usage Count Trigger Type
upstream-monitor.yaml check-upstream-versions
check-dependency-versions
update-version
close-duplicate-prs		 4 actions Schedule (6 AM UTC daily)
Manual dispatch
auto-build.yaml detect-containers
build-container
docker-login		 3 actions workflow_call (from upstream-monitor)
Push/PR events
Manual dispatch
recreate-manifests.yaml detect-containers
docker-login		 2 actions Manual dispatch only
update-dashboard.yaml None (uses scripts directly) 0 actions workflow_call (from auto-build)
Manual dispatch
validate-version-scripts.yaml check-upstream-versions 1 action Push/PR events
Manual dispatch

Workflow → Workflow Dependencies

Source Workflow Triggers Target Workflow Method
upstream-monitor.yaml → auto-build.yaml workflow_call
auto-build.yaml → update-dashboard.yaml workflow_call

Action Usage Analysis

Action Used By Total Usage Purpose
check-upstream-versions upstream-monitor.yaml
validate-version-scripts.yaml		 2x Version detection and validation
build-container auto-build.yaml 1x Container building and pushing
detect-containers auto-build.yaml 1x Smart container change detection
check-dependency-versions upstream-monitor.yaml 1x 3rd party dependency version monitoring
close-duplicate-prs upstream-monitor.yaml 1x PR management and cleanup
update-version upstream-monitor.yaml 1x Version file updates
setup-github-cli auto-build.yaml 1x GitHub CLI configuration

Status: ✅ All 7 actions are actively used across workflows

Reusable Actions

Check Upstream Versions (.github/actions/check-upstream-versions)

Used by: upstream-monitor.yaml, validate-version-scripts.yaml

Checks for upstream version updates across containers using individual version.sh scripts.

Inputs:

  • container (optional): Specific container to check

Outputs:

  • containers_with_updates: JSON array of containers needing updates
  • update_count: Number of containers with updates
  • version_info: Detailed version information

Key Features:

  • Supports both single container and bulk checking
  • Integrates with shared helper functions
  • Handles version script failures gracefully

Check Dependency Versions (.github/actions/check-dependency-versions)

Used by: upstream-monitor.yaml

Checks 3rd party dependency versions against upstream releases (GitHub Releases API, PyPI).

Inputs:

  • container (optional): Specific container to check

Outputs:

  • containers_with_dep_updates: JSON array of container names with updates
  • dep_update_count: Total number of dependency updates found
  • dep_version_info: Full JSON output from orchestrator

Key Features:

  • Config-driven: reads dependency_sources section from each container's config.yaml
  • Supports GitHub releases, GitHub tags, and PyPI as sources
  • Postgres special case: delegates to version-extension.sh
  • Pre-flight validation warns about unregistered build_args (INV-05)
  • Writes GITHUB_STEP_SUMMARY with update table

Build Container (.github/actions/build-container)

Used by: auto-build.yaml

Builds a specific container using the universal make script with enhanced error handling and security scanning.

Inputs:

  • container: Container name to build
  • force_rebuild: Force rebuild even if up-to-date
  • dockerhub_username, dockerhub_token, github_token: Registry credentials
  • scan_vulnerabilities: Enable Trivy security scanning (default: true)
  • vulnerability_severity: Minimum severity to fail build - CRITICAL, HIGH, MEDIUM, LOW (default: CRITICAL)

Key Features:

  • Delegates to universal make script with --bare mode
  • Multi-registry support (Docker Hub + GHCR)
  • Security scanning with Trivy (CVE detection)
  • SARIF reports uploaded to GitHub Security tab
  • Proper exit code handling
  • Build status reporting

Security Scanning:

  • Scans for OS and library vulnerabilities after build
  • Blocks push on CRITICAL vulnerabilities by default
  • Generates SARIF reports for GitHub Security integration
  • PR builds show results without blocking

Build Caching (Registry-based):

  • Uses GHCR as persistent cache backend (ghcr.io/<owner>/<container>:buildcache)
  • Cache is read during build step (faster rebuilds)
  • Cache is written during GHCR push (updated after each build)
  • Docker Hub push uses read-only cache from GHCR
  • Significantly reduces build times for unchanged layers

Detect Containers (.github/actions/detect-containers)

Used by: auto-build.yaml

Intelligently detects which containers need building based on file changes.

Outputs:

  • containers: JSON array of containers to build
  • count: Number of containers detected

Logic:

  • Analyzes git changes for container-specific files
  • Supports forced rebuilds via input parameters
  • Excludes non-container directories

Close Duplicate PRs (.github/actions/close-duplicate-prs)

Used by: upstream-monitor.yaml

Manages PR lifecycle by closing outdated or duplicate version update PRs.

Features:

  • Prevents PR spam from multiple automation runs
  • Maintains clean PR history
  • Respects manual PRs and other automation

Update Version (.github/actions/update-version)

Used by: upstream-monitor.yaml

Updates container version files and creates commit for PR creation.

Features:

  • Updates version.sh or metadata files
  • Creates descriptive commit messages
  • Integrates with PR creation workflow

Setup GitHub CLI (.github/actions/setup-github-cli)

Used by: auto-build.yaml

Configures GitHub CLI for workflow operations.

Features:

  • Installs and configures gh CLI
  • Sets up authentication tokens
  • Provides consistent CLI environment

Check Upstream Versions (.github/actions/check-upstream-versions)

Checks for upstream version updates across containers.

Inputs:

  • container (optional): Specific container to check

Outputs:

  • containers_with_updates: JSON array of containers needing updates
  • update_count: Number of containers with updates
  • version_info: Detailed version information

Check Dependency Versions (.github/actions/check-dependency-versions)

Checks 3rd party dependency versions against upstream releases.

Inputs:

  • container (optional): Specific container to check

Outputs:

  • containers_with_dep_updates: JSON array of container names with updates
  • dep_update_count: Total dependency updates found
  • dep_version_info: Full JSON output

Build Container (.github/actions/build-container)

Builds a specific container with optimizations and error handling.

Inputs:

  • container: Container name to build
  • force_rebuild: Force rebuild even if up-to-date
  • dockerhub_username, dockerhub_token, github_token: Registry credentials

Features:

  • Multi-architecture support
  • Build caching
  • Registry push automation
  • Retry logic on failures

Detect Containers (.github/actions/detect-containers)

Intelligently detects which containers need building based on changes.

Outputs:

  • containers: JSON array of containers to build
  • count: Number of containers detected

Usage Examples

Manual Workflow Triggers

# PRIMARY: Monitor all containers for updates (triggers full pipeline)
gh workflow run upstream-monitor.yaml

# Check specific container with debug output
gh workflow run upstream-monitor.yaml \
  --field container=ansible \
  --field debug=true \
  --field create_pr=false

# SECONDARY: Force rebuild all containers (manual builds only)
gh workflow run auto-build.yaml \
  --field force_rebuild=true

# Build specific container manually
gh workflow run auto-build.yaml \
  --field container=wordpress \
  --field force_rebuild=true

# MANIFEST-ONLY: Recreate Docker Hub manifests without rebuilding
gh workflow run recreate-manifests.yaml -f registry=dockerhub

# Recreate manifests for specific container
gh workflow run recreate-manifests.yaml -f container=postgres

# UTILITY: Update dashboard only
gh workflow run update-dashboard.yaml

# TESTING: Validate version scripts
gh workflow run validate-version-scripts.yaml

Workflow Integration Pattern

Our optimized architecture follows this pattern for reliable automation:

# upstream-monitor.yaml (Primary Entry Point)
name: Upstream Monitor
on:
  schedule:
    - cron: '0 6 * * *'  # Only workflow with schedule (6 AM UTC daily)
  workflow_dispatch:
    inputs:
      container: { type: string }
      debug: { type: boolean, default: false }
      create_pr: { type: boolean, default: true }

jobs:
  # ... version checking jobs ...
  
  trigger-builds:
    if: success() && steps.check.outputs.update_count > 0
    uses: ./.github/workflows/auto-build.yaml  # workflow_call trigger
    with:
      container: ${{ inputs.container }}
    secrets: inherit

---

# auto-build.yaml (Secondary - Triggered by Primary)
name: Auto Build
on:
  workflow_call:    # PRIMARY trigger from upstream-monitor
    inputs:
      container: { type: string }
  push:            # SECONDARY trigger for code changes
    paths: ['*/Dockerfile', '*/version.sh']
  workflow_dispatch: # MANUAL trigger

jobs:
  # ... build jobs ...
  
  update-dashboard:
    if: success()
    uses: ./.github/workflows/update-dashboard.yaml  # workflow_call trigger
    secrets: inherit

Using Actions in Custom Workflows

name: Custom Container Workflow
on: workflow_dispatch

jobs:
  check-and-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Check for updates
        id: check
        uses: ./.github/actions/check-upstream-versions
        with:
          container: wordpress
      
      - name: Build if updated
        if: steps.check.outputs.update_count > 0
        uses: ./.github/actions/build-container
        with:
          container: wordpress
          force_rebuild: false
          github_token: ${{ secrets.GITHUB_TOKEN }}

Required Permissions

Workflows require specific GitHub token permissions based on their responsibilities:

Upstream Monitor Workflow

permissions:
  contents: write      # Update version files and create commits
  pull-requests: write # Create and manage version update PRs
  actions: write       # Trigger auto-build workflow via workflow_call

Auto Build Workflow

permissions:
  contents: read       # Read repository files for building
  packages: write      # Push to GHCR (ghcr.io registry)
  actions: write       # Trigger update-dashboard workflow via workflow_call
  attestations: write  # GitHub SBOM attestation via Sigstore
  id-token: write      # OIDC token for Sigstore signing

Update Dashboard Workflow

permissions:
  contents: write      # Update dashboard files and commit to gh-pages
  pages: write         # Deploy to GitHub Pages

Registry Credentials

# Via GitHub Secrets (for Docker Hub):
secrets:
  DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
  DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
  
# GitHub Container Registry uses GITHUB_TOKEN automatically

Environment Configuration

Global Workflow Environment

env:
  # Build Optimization
  DOCKER_BUILDKIT: 1                    # Enable BuildKit for faster builds
  BUILDX_NO_DEFAULT_ATTESTATIONS: 1     # Disable attestations for speed
  
  # Automation Limits
  MAX_OPEN_PRS_PER_CONTAINER: 2         # Limit concurrent PRs per container
  PR_AUTO_CLOSE_DAYS: 7                 # Auto-close stale PRs after 7 days
  
  # Registry Configuration  
  REGISTRY_URL: ghcr.io                 # Primary registry (GHCR)
  DOCKERHUB_REGISTRY: docker.io         # Secondary registry (Docker Hub)

Container-Specific Environment

# Set in Dockerfiles with flexible version handling
ARG VERSION=latest                      # Default to latest, override in builds
ENV CONTAINER_VERSION=${VERSION}        # Runtime version visibility

Troubleshooting

Workflow Architecture Issues

Scheduling Conflicts (RESOLVED):

  • ✅ Fixed: Only upstream-monitor.yaml has schedule triggers
  • ✅ Fixed: auto-build.yaml uses workflow_call instead of schedule
  • ✅ Fixed: Clean trigger chain eliminates race conditions

Workflow Not Triggering:

  • Check if upstream-monitor.yaml is the entry point (it should be)
  • Verify workflow_call triggers are properly configured
  • Ensure branch protection rules allow workflow_call
  • Validate GitHub token permissions for cross-workflow triggers

Version Script Failures

Local Testing:

# Test version script directly
cd container-name
./version.sh          # Check current version  
./version.sh latest   # Check upstream version

# Test with debug output
DEBUG=1 ./version.sh latest

Common Issues:

  • API rate limits (use authenticated requests where possible)
  • Network timeouts (check API response times)
  • JSON parsing errors (validate with jq locally)
  • Invalid version formats (check regex patterns)

Build Failures

Debug Build Process:

# Test build locally using make script
./make build container-name

# Enable verbose output
DEBUG=1 ./make build container-name

# Test specific architecture
./make build container-name --platform linux/amd64

Common Issues:

  • Docker daemon connectivity
  • Registry authentication (check DOCKERHUB_TOKEN)
  • Build context size (optimize .dockerignore)
  • Multi-architecture compatibility

Action Integration Issues

Action Path Resolution:

  • All actions use relative paths: uses: ./.github/actions/action-name
  • Verify action directories exist and contain action.yml
  • Check action inputs/outputs match workflow usage

Workflow Call Issues:

# Correct workflow_call syntax:
uses: ./.github/workflows/target-workflow.yaml
with:
  input_name: value
secrets: inherit  # Required for cross-workflow secrets

Performance Optimization

Workflow Hierarchy Benefits

Before Optimization:

  • Multiple workflows with overlapping schedules
  • Race conditions between concurrent runs
  • Redundant container detection logic

After Optimization:

  • Single scheduled entry point (upstream-monitor)
  • Sequential workflow execution via workflow_call
  • Shared container detection and build logic
  • Reduced GitHub Actions minutes usage

Build Performance

Multi-Architecture Strategy:

# Optimized build matrix
strategy:
  matrix:
    platform: 
      - linux/amd64
      - linux/arm64
  fail-fast: false  # Continue building other architectures on failure

Build Caching:

# Leverage BuildKit cache mounts
RUN --mount=type=cache,target=/var/cache/apt \
    apt-get update && apt-get install -y package

Best Practices

Workflow Design Principles

  1. Single Source of Truth: One workflow triggers the automation chain
  2. Proper Separation: Each workflow has distinct responsibilities
  3. Fail-Safe Operation: Workflows continue on non-critical errors
  4. Comprehensive Logging: All steps include descriptive names and summaries

Action Development

  1. Reusability: Actions should work across multiple workflows
  2. Error Handling: Graceful failure with descriptive error messages
  3. Input Validation: Validate all inputs before processing
  4. Output Consistency: Standardized output formats (JSON for complex data)

Security & Maintenance

  1. Minimal Permissions: Use least-privilege principle for workflow permissions
  2. Secret Management: Store credentials in GitHub Secrets, never in code
  3. Version Pinning: Use specific action versions (e.g., @v4) not @latest
  4. Regular Audits: Review workflow logs and update patterns quarterly