From 2e8b8a6871770479d1b7de81e29820a8afa63df0 Mon Sep 17 00:00:00 2001 From: Ivan Buttinoni Date: Wed, 11 Mar 2026 10:01:11 +0100 Subject: [PATCH 1/5] split ci.md into files, one per chapter --- Lessons/01-foundations.md | 531 +++++++ Lessons/02-core-ci-skills.md | 803 ++++++++++ Lessons/03-intermediate-ci.md | 425 +++++ Lessons/04-advanced-ci.md | 521 +++++++ Lessons/05-mastery.md | 239 +++ Lessons/99-reference.md | 206 +++ Lessons/ci.md | 2746 --------------------------------- Lessons/index.md | 17 + 8 files changed, 2742 insertions(+), 2746 deletions(-) create mode 100644 Lessons/01-foundations.md create mode 100644 Lessons/02-core-ci-skills.md create mode 100644 Lessons/03-intermediate-ci.md create mode 100644 Lessons/04-advanced-ci.md create mode 100644 Lessons/05-mastery.md create mode 100644 Lessons/99-reference.md delete mode 100644 Lessons/ci.md create mode 100644 Lessons/index.md diff --git a/Lessons/01-foundations.md b/Lessons/01-foundations.md new file mode 100644 index 0000000..8513188 --- /dev/null +++ b/Lessons/01-foundations.md @@ -0,0 +1,531 @@ +# Level 1 — Foundations + +## 1.1 The SDLC and Where CI Fits + +### The Traditional Problem + +Before CI, teams worked in isolation for days or weeks, then attempted to merge everything at once — a painful process known as **"integration hell"**. + +```text +Developer A (2 weeks of work) ──┐ +Developer B (2 weeks of work) ──┼──► MERGE ──► 💥 Conflicts everywhere +Developer C (2 weeks of work) ──┘ +``` + +### The CI Solution + +Integrate continuously — every change triggers an automated pipeline: + +```text +Developer A (1 commit) ──► Pipeline ──► ✅ Merged +Developer B (1 commit) ──► Pipeline ──► ✅ Merged +Developer C (1 commit) ──► Pipeline ──► ❌ Tests fail → Fix immediately +``` + +### The SDLC with CI + +```text +┌─────────┐ ┌──────┐ ┌───────┐ ┌──────┐ ┌─────────┐ ┌────────┐ +│ PLAN │──►│ CODE │──►│ BUILD │──►│ TEST │──►│ RELEASE │──►│ DEPLOY │ +└─────────┘ └──────┘ └───────┘ └──────┘ └─────────┘ └────────┘ + │ ▲ + └───── CI ─────┘ (automated feedback on every commit) +``` + +| Stage | Without CI | With CI | +|---------|-----------------------------|-------------------------------------------| +| Code | Manual reviews | Automated linting, type checking | +| Build | "Works on my machine" | Reproducible builds in clean environments | +| Test | Run manually, often skipped | Mandatory, automated on every commit | +| Release | Manual, error-prone | Automated versioning and changelogs | + +--- + +## 1.2 Git Mastery for CI + +### Essential Git Configuration + +```bash +# Identity +git config --global user.name "Your Name" +git config --global user.email "you@example.com" + +# Default branch +git config --global init.defaultBranch main + +# Useful aliases +git config --global alias.st "status --short" +git config --global alias.lg "log --oneline --graph --decorate --all" +git config --global alias.recent "branch --sort=-committerdate" +``` + +### Branching Strategies Compared + +#### Git Flow — scheduled releases + +```text +main ──────────────────────────────────────────────► (production tags) + └── develop ─────────────────────────────────────► (integration) + ├── feature/login ──────────► merge to develop + ├── feature/checkout ───────► merge to develop + └── release/1.2.0 ──────────► merge to main + develop + └── hotfix/critical ──► merge to main + develop +``` + +#### GitHub Flow — continuous web delivery + +```text +main ─────────────────────────────────────────────► (always deployable) + ├── feature/new-api ──► PR ──► merge ──► deploy + └── fix/auth-bug ─────► PR ──► merge ──► deploy +``` + +#### Trunk-Based Development — recommended for CI + +The model used by Google, Meta, and most high-performing engineering teams. Everyone integrates into `main` continuously, keeping the trunk always in a deployable state. + +```text + Day 1 Day 2 Day 3 + │ │ │ +main ───────┼─────────────────────┼─────────────────────┼──────► (always green) + │ ╭─ feat/login ─╮ │ ╭─ fix/auth ─╮ │ + │ │ (< 4 hours) │ │ │ (2 hours) │ │ + │ ╰──────────────╯ │ ╰────────────╯ │ + │ │ PR+merge │ │ PR+merge │ + ▼ ▼ ▼ ▼ ▼ + commit merge commit merge commit + │ │ + └── CI runs ✅ └── CI runs ✅ +``` + +**The core rules:** + +1. **Branches live less than a day.** If a branch lasts more than 24 hours, it's a signal that the task needs to be split into smaller pieces — not that the branch should keep growing. + +2. **Main is always deployable.** Every commit merged to `main` must pass CI. The pipeline is the gatekeeper — there is no "integration branch" to hide broken code. + +3. **Unfinished features ship behind feature flags.** Code can be merged before the feature is complete, as long as the flag keeps it invisible to users. This separates *deployment* (code goes to production) from *release* (users can see it). + +**Why it works better for CI than Git Flow or GitHub Flow:** + +| | Git Flow | GitHub Flow | Trunk-Based Dev | +|---------------------|-----------------|-------------|-----------------| +| Branch lifetime | Days–weeks | Hours–days | Hours (< 1 day) | +| Merge conflicts | Large, painful | Moderate | Minimal | +| CI feedback speed | Delayed | Fast | Immediate | +| Integration risk | High at release | Low | Very low | +| Feature flag needed | No | Rarely | Yes | + +**What a typical TBD day looks like:** + +```text +09:00 Developer picks up a ticket +09:15 git checkout -b feat/add-email-validation + (writes code + tests) +11:30 git push → opens PR +11:45 CI passes ✅, teammate reviews +12:00 Merged to main → CI runs again ✅ → auto-deployed to staging + (feature is behind a flag: email_validation_v2 = false) + +14:00 Developer picks up next ticket + ...same cycle repeats +``` + +**Feature flags in practice:** + +The flag lets you merge incomplete or experimental code safely. The feature is invisible in production until the team is confident enough to flip the switch — independently of any deployment. + +```typescript +// The feature ships in the codebase, but is off by default +const useNewEmailValidation = featureFlags.get('email_validation_v2', false); + +if (useNewEmailValidation) { + return validateWithZod(email); // New path — dark until flag is on +} else { + return legacyValidate(email); // Old path — still running +} +``` + +Flag lifecycle: + +```text +Merge (flag off) ──► QA testing (flag on for testers) ──► Gradual rollout ──► 100% ──► Remove flag + day 1 day 2–3 day 4–5 day 6 day 7+ +``` + +**When NOT to use TBD:** + +- Open source projects with external contributors who submit infrequent large PRs +- Teams without a fast CI pipeline (if CI takes 30+ minutes, short-lived branches become impractical) +- Projects with formal release gating (regulated industries, firmware) — Git Flow is more appropriate + +### Conventional Commits + +A machine-readable commit format that enables automated changelogs and semantic versioning. + +```text +[optional scope]: + +[optional body] + +[optional footer(s)] +``` + +| Type | Meaning | Version Bump | +|------------|-------------------------|---------------| +| `feat` | New feature | Minor (1.x.0) | +| `fix` | Bug fix | Patch (1.0.x) | +| `feat!` | Breaking change | Major (x.0.0) | +| `docs` | Documentation only | None | +| `chore` | Maintenance tasks | None | +| `test` | Adding or fixing tests | None | +| `ci` | CI config changes | None | +| `perf` | Performance improvement | Patch | +| `refactor` | Code restructure | None | + +**Examples:** + +```bash +git commit -m "feat(auth): add OAuth2 login with Google" +git commit -m "fix(api): handle null response from upstream service" +git commit -m "feat!: remove deprecated /v1 endpoints + +BREAKING CHANGE: /v1/users and /v1/orders removed. Use /v2 equivalents." +git commit -m "chore(deps): bump axios from 1.3.0 to 1.6.0" +``` + +### .gitignore for CI Workflows + +```gitignore +# Dependencies +node_modules/ +vendor/ +__pycache__/ +*.pyc +.venv/ + +# Build outputs +dist/ +build/ +*.egg-info/ +target/ + +# Secrets — NEVER commit these +.env +.env.local +.env.*.local +*.pem +*.key +secrets.yaml + +# CI artifacts +coverage/ +.nyc_output/ +junit.xml +test-results/ +*.log + +# IDE +.idea/ +.vscode/ +*.swp + +# OS +.DS_Store +Thumbs.db +``` + +### Branch Protection Rules + +Branch protection rules are the enforcement layer that makes CI meaningful. Without them, a developer can merge a PR even if every check is failing — the pipeline becomes advisory, not mandatory. Rules are configured per branch, typically targeting `main` (and sometimes `develop` or `release/*`). + +**What happens without protection vs with it:** + +```text +Without rules: With rules: +┌─────────────────────────────┐ ┌─────────────────────────────┐ +│ CI: ❌ Tests failing │ │ CI: ❌ Tests failing │ +│ CI: ❌ Lint failing │ │ CI: ❌ Lint failing │ +│ │ │ │ +│ [ Merge ] ← still clickable│ │ [ Merge ] ← BLOCKED 🔒 │ +└─────────────────────────────┘ └─────────────────────────────┘ + Developer can ignore CI CI is a hard gate +``` + +#### Setting Up on GitHub (UI path) + +```text +Repository → Settings → Branches → Add branch protection rule + Pattern: main (supports wildcards: release/*, v*.*) +``` + +The key options and what each one actually does: + +```text +☑ Require a pull request before merging + └─ Prevents direct pushes to main — all changes must come through a PR. + Nobody, including repo admins, can bypass this without explicitly + overriding the rule. + + ☑ Require approvals: 1 + │ └─ At least one reviewer must explicitly approve before merging. + │ Prevents rubber-stamp self-merges on solo projects or + │ "I'll review it after" patterns. + │ + └─ ☑ Dismiss stale pull request approvals when new commits are pushed + └─ If someone approves a PR and the author then pushes new + commits, the approval is revoked. Forces re-review of changes + made after approval — common attack vector if left unchecked. + +☑ Require status checks to pass before merging + │ └─ The actual CI gate. Lists which jobs MUST be green. + │ The job names here must exactly match the `name:` field in your + │ workflow (GitHub Actions) or the job key (GitLab CI). + │ + ├─ "CI / build-and-test" ← matches `name: build-and-test` in workflow + ├─ "CI / lint" + └─ "CI / security-scan" + + ☑ Require branches to be up to date before merging + └─ The PR's branch must include all commits currently on main. + Without this, two PRs could both pass CI individually but + conflict when merged together ("works in isolation" problem). + Forces the author to rebase/merge main before merging their PR. + +☑ Require conversation resolution before merging + └─ All review comments must be marked as resolved. Prevents merging + with open questions or unaddressed feedback. + +☑ Do not allow bypassing the above settings + └─ Applies rules even to repository administrators and owners. + Without this, admins can force-push or merge bypassing all checks. + Essential for compliance and auditability. +``` + +#### How Status Check Names Are Determined + +The name you register in the branch protection rule must match **exactly** what the CI system reports. Here is how to find the correct name for each platform: + +**GitHub Actions** — the check name is `{workflow name} / {job id}`: + +```yaml +# .github/workflows/ci.yml +name: CI # ← workflow name + +jobs: + build-and-test: # ← job id + ... + lint: + ... + +# Results in checks named: +# CI / build-and-test +# CI / lint +``` + +**GitLab CI** — the check name is the job key: + +```yaml +# .gitlab-ci.yml +unit-tests: # ← this is the status check name reported to GitHub + script: npm test + +lint: # ← and this one + script: npm run lint +``` + +#### ⚠️ Checks Only Appear After the First Pipeline Run + +This is one of the most common points of confusion when setting up branch protection for the first time. GitHub does **not** know which checks exist until the pipeline has actually run at least once against your repository. Until then, the search box under "Status checks that are required" returns no results — not because the configuration is wrong, but because GitHub has never seen those check names before. + +```text +First time setup order: + + Step 1 — Push your CI workflow file (.github/workflows/ci.yml) + ↓ + Step 2 — Let it run once (on any branch or a test PR) + ↓ + Step 3 — Go to Settings → Branches → Add rule → main + Type the check name in the search box + ↓ now it appears as an autocomplete suggestion + Step 4 — Select and save +``` + +**What the UI looks like before vs after the first run:** + +```text +Before first run: + Status checks that are required + ┌─────────────────────────────────────────┐ + │ Search for status checks... │ + └─────────────────────────────────────────┘ + No results found for "CI / lint" ← frustrating, but expected + +After first run: + Status checks that are required + ┌─────────────────────────────────────────┐ + │ CI / │ + └─────────────────────────────────────────┘ + ✓ CI / build-and-test ← now selectable + ✓ CI / lint + ✓ CI / security-scan +``` + +**Practical bootstrap sequence for a new repository:** + +```bash +# 1. Add your workflow file and push to a feature branch +git checkout -b ci/setup +mkdir -p .github/workflows +# ... create ci.yml ... +git add .github/workflows/ci.yml +git commit -m "ci: add initial CI pipeline" +git push origin ci/setup + +# 2. Open a PR — this triggers the pipeline for the first time +# Wait for all jobs to complete (green or red doesn't matter, +# GitHub just needs to register the check names) + +# 3. Now go to Settings → Branches → Add rule +# The check names will appear in the autocomplete search + +# 4. Merge this PR — branch protection is now active for all future PRs +``` + +> **Tip:** If you rename a job in your workflow YAML later (e.g., `lint` → `lint-and-format`), GitHub will stop receiving reports for the old name. The branch protection rule will reference a check that no longer exists, blocking all PRs. Always update the required checks in Settings immediately after renaming a job, or run the pipeline once so the new name appears in the search before saving the rule. + +**What happens when a required check never runs:** + +```text +Scenario: You added "CI / lint" as required, but the workflow only + triggers on pull_request, and someone pushes directly to + a non-protected branch. + +Status shown on PR: "Expected — Waiting for status to be reported" +Result: PR is permanently blocked — the check never fires. + +This is intentional and safe. It forces you to ensure your workflow +trigger covers the protected branch context. Common fix: + + on: + push: + branches: [main] + pull_request: ← this is the trigger that populates the check + branches: [main] ← on PRs targeting main +``` + +#### Full Recommended Configuration + +```yaml +# GitHub: Settings → Branches → Add rule +# Pattern: main + +# --- Pull Request Requirements --- +require_pull_request: true +required_approving_review_count: 1 # Increase to 2 for critical repos +dismiss_stale_reviews: true # Re-review required after new commits +require_review_from_codeowners: true # CODEOWNERS file defines who reviews what + +# --- CI Status Checks (must match exact job names) --- +require_status_checks: true +require_branches_up_to_date: true # Branch must be current with main +required_checks: + - "CI / lint" + - "CI / typecheck" + - "CI / unit-tests" + - "CI / integration-tests" + - "CI / security-scan" + +# --- Commit Requirements --- +require_signed_commits: true # GPG-signed commits (audit trail) +require_linear_history: true # Disables merge commits, enforces rebase + # Keeps git log clean and bisectable + +# --- Push Restrictions --- +restrict_pushes: true # Only allow pushes via PRs +allow_force_pushes: false # Prevents history rewriting on main +allow_deletions: false # Prevents accidental branch deletion + +# --- Admin Override --- +bypass_pull_request_allowances: [] # Nobody bypasses — not even admins +``` + +#### CODEOWNERS — Automatic Review Assignment + +Pair branch protection with a `CODEOWNERS` file to automatically assign reviewers based on which files changed: + +```bash +# .github/CODEOWNERS (or CODEOWNERS at repo root) + +# Default owner for everything +* @org/backend-team + +# Frontend — assigned to frontend team automatically +/frontend/ @org/frontend-team +*.tsx @org/frontend-team +*.css @org/frontend-team + +# Infrastructure changes require DevOps review +/infrastructure/ @org/devops-team +Dockerfile @org/devops-team +docker-compose*.yml @org/devops-team + +# CI config requires a senior engineer +.github/workflows/ @org/senior-engineers +.gitlab-ci.yml @org/senior-engineers + +# Secrets and sensitive config require security team +*secret* @org/security-team +*credential* @org/security-team +``` + +When a PR touches `/infrastructure/Dockerfile`, GitHub will automatically request a review from `@org/devops-team` and block merging until they approve. + +#### GitLab Equivalent — Protected Branches + +```yaml +# GitLab: Settings → Repository → Protected branches + +Branch: main +Allowed to merge: Developers + Maintainers +Allowed to push: No one # Forces all changes through MRs +Allowed to force push: false + +# Settings → Merge requests +Pipelines must succeed: true # CI gate +All discussions must be resolved: true +``` + +--- + +## 1.3 Core CI Concepts + +### CI vs Continuous Delivery vs Continuous Deployment + +```text +────────────────────────────────────────────────────────────── + CONTINUOUS INTEGRATION + [Code] ──► [Build] ──► [Test] + +────────────────────────────────────────────────────────────── + + CONTINUOUS DELIVERY (human approves deploy) + [Code] ──► [Build] ──► [Test] ──► [Staging] ──► [👤 Approve] ──► [Prod] + +────────────────────────────────────────────────────────────── + + CONTINUOUS DEPLOYMENT (fully automated) + [Code] ──► [Build] ──► [Test] ──► [Staging] ──► [Auto ✅] ──► [Prod] +────────────────────────────────────────────────────────────── +``` + +### The 8 Principles of CI + +1. **Single source repository** — one canonical repo +2. **Automate the build** — `make build` or equivalent, no manual steps +3. **Self-testing build** — failing tests = failing build +4. **Commit to mainline daily** — small, frequent integration reduces risk +5. **Every commit triggers CI** — the server watches every push +6. **Fix broken builds immediately** — broken CI blocks the whole team +7. **Keep builds fast** — target under 10 minutes +8. **Test in a production-like environment** — use containers for parity + +--- diff --git a/Lessons/02-core-ci-skills.md b/Lessons/02-core-ci-skills.md new file mode 100644 index 0000000..c76d6af --- /dev/null +++ b/Lessons/02-core-ci-skills.md @@ -0,0 +1,803 @@ +# Level 2 — Core CI Skills + +## 2.1 CI Platform Comparison + +| Platform | Config File | Free Tier (cloud) | Self-hostable | Best For | +|----------|-------------|-------------------|---------------|----------| +| **GitHub Actions** | `.github/workflows/*.yml` | 2,000 min/mo (public repos: unlimited) | ✅ | GitHub projects | +| **GitLab CI/CD** | `.gitlab-ci.yml` | 400 min/mo | ✅ | Full DevOps suite | +| **Jenkins** | `Jenkinsfile` | ❌ (no cloud offering) | ✅ free | Enterprise customization | +| **CircleCI** | `.circleci/config.yml` | 6,000 min/mo | ✅ | Docker-native speed | +| **Bitbucket Pipelines** | `bitbucket-pipelines.yml` | 50 min/mo | ❌ | Atlassian ecosystem | +| **Azure DevOps** | `azure-pipelines.yml` | 1,800 min/mo (public: unlimited) | ✅ | Microsoft/.NET shops | +| **Drone CI** | `.drone.yml` | ❌ (no cloud offering) | ✅ free | Lightweight, container-native | + +> **Recommendation:** Start with **GitHub Actions** or **GitLab CI** — best documentation, native integration, and large community. + +### Understanding the Free Tier + +"Free tier" in cloud CI platforms means **compute minutes per month** billed against a shared runner fleet. It is not unlimited free usage — it is a monthly quota that resets on a billing cycle. Understanding how minutes are counted and when you will exhaust them avoids surprise bills. + +#### How Minutes Are Counted + +A "minute" in CI is a **runner-minute**: one minute of wall-clock time on one runner. Parallel jobs each consume their own minutes simultaneously. + +```text +Example pipeline with 3 parallel jobs, each taking 5 minutes: + + lint ████████████ 5 min + test ████████████ 5 min ← all run at the same time + build ████████████ 5 min + + Wall-clock time elapsed: 5 minutes + Minutes billed: 15 minutes (3 jobs × 5 min) +``` + +Runner OS also affects billing on some platforms. GitHub Actions applies a multiplier to non-Linux runners: + +| OS | GitHub Actions multiplier | +|----|--------------------------| +| Linux (ubuntu-*) | 1× (base rate) | +| Windows (windows-*) | 2× | +| macOS (macos-*) | 10× | + +A 10-minute macOS job costs 100 minutes of your free quota. This is relevant if you are building mobile apps or testing cross-platform CLI tools. + +#### Free Tier Realities Per Platform + +##### GitHub Actions — 2,000 min/mo on private repos + +The most important exception: **public repositories get unlimited free minutes** on GitHub-hosted runners. This makes GitHub Actions the default choice for open source projects. + +```text +Private repo: 2,000 min/mo on Linux (~33 hours of single-job pipelines) + 200 min/mo equivalent on macOS (10× multiplier) +Public repo: Unlimited on all OS +``` + +A team running 20 PRs/day with a 6-minute pipeline uses roughly: + +```text +20 PRs × 6 min × 22 working days = 2,640 min/mo +→ Exceeds the free tier on private repos +→ Fine on public repos +``` + +##### GitLab CI — 400 min/mo (shared runners) + +GitLab's free tier is significantly smaller. It suits individuals and very small teams but is quickly exhausted by active development workflows. The practical alternative: **register your own machine as a GitLab Runner** — this is free, has no minute limit, and is how most self-hosted GitLab setups operate. + +```text +400 min/mo ÷ 22 working days ≈ 18 min/day of shared runner time + +If your pipeline takes 8 min and you push 3 times a day: + 3 × 8 min = 24 min/day → quota runs out mid-month +``` + +##### CircleCI — 6,000 min/mo + +The largest free cloud quota, but limited to 1 concurrent job on free plans. Parallelism (multiple jobs running at the same time) requires a paid plan. + +```text +Free plan: 6,000 min/mo, 1 concurrent job + Jobs queue rather than run in parallel + → a pipeline with 4 jobs runs sequentially, not in parallel +``` + +##### Bitbucket Pipelines — 50 min/mo + +Effectively a trial tier. 50 minutes is consumed by a single working day of normal development. Only viable for very low-frequency pipelines or as a trigger for external systems. + +##### Jenkins and Drone CI — no cloud offering + +These platforms have no managed cloud runner. "Free" means the software licence is free, but **you provide and pay for the compute yourself** (a VM, a bare-metal server, or a container on your own infrastructure). There is no minute quota, but there are real costs: hardware, electricity, maintenance, and runner uptime. + +#### When to Move Off the Free Tier + +Signs you have outgrown a cloud free tier: + +```text +1. Pipelines are queuing — jobs wait minutes before a runner picks them up +2. You are rationing pushes to save minutes +3. You are skipping CI runs to preserve quota +4. Your monthly bill spikes unexpectedly at day 18 + +Solutions, in order of preference: + a) Self-host one or more runners → unlimited minutes on your own hardware + b) Optimize pipelines to reduce per-run minute consumption + c) Upgrade to a paid plan for the parallelism and quota you actually need +``` + +#### Self-Hosted Runners: Zero Minute Cost + +Every platform that supports self-hosted runners exempts them from minute quotas entirely. Jobs on your own machine do not consume any cloud minutes regardless of how long they run. + +```text +GitHub Actions self-hosted runner: + runs-on: self-hosted ← this job costs $0 in GitHub minutes + runs-on: [self-hosted, linux, gpu] ← same, with label targeting +``` + +This is the standard solution for teams with GPU workloads, private network access, or simply high pipeline volume. See [Section 4.2 — Self-Hosted Runners](04-advanced-ci.md#42-self-hosted-runners) for setup details. + +--- + +## 2.2 Anatomy of a Pipeline + +### Pipeline Lifecycle + +```text +┌──────────┐ +│ TRIGGER │ push / pull_request / schedule / manual / webhook +└────┬─────┘ + │ +┌────▼─────┐ +│ CHECKOUT │ clone repository at the specific commit SHA +└────┬─────┘ + │ +┌────▼─────┐ +│ RESTORE │ restore dependency cache (npm, pip, cargo, maven…) +│ CACHE │ +└────┬─────┘ + │ +┌────▼─────────────────────────────────┐ +│ PARALLEL JOBS │ +│ ┌─────────┐ ┌──────┐ ┌────────┐ │ +│ │ LINT │ │BUILD │ │ TEST │ │ +│ └─────────┘ └──────┘ └────────┘ │ +└────┬─────────────────────────────────┘ + │ +┌────▼─────┐ +│ REPORT │ upload coverage, test results, build logs +└────┬─────┘ + │ +┌────▼─────┐ +│ NOTIFY │ Slack, email, PR comment, commit status +└──────────┘ +``` + +### Key Concepts Glossary + +| Term | Definition | +|------------------|----------------------------------------------------------| +| **Trigger** | Event that starts the pipeline (push, PR, cron, webhook) | +| **Runner/Agent** | The machine where jobs execute | +| **Job** | An isolated unit of work with its own runner | +| **Step** | A single command or action within a job | +| **Stage** | Logical group of jobs (test, build, deploy) | +| **Artifact** | File(s) produced by a job, passed to downstream jobs | +| **Cache** | Persisted files reused across pipeline runs | +| **Environment** | Named deployment target (dev, staging, prod) | +| **Secret** | Encrypted value injected as an environment variable | + +--- + +## 2.3 Writing Pipelines + +### GitHub Actions — Complete Annotated Example + +```yaml +# .github/workflows/ci.yml + +name: CI # Displayed in the GitHub Actions UI + +on: # Triggers + push: + branches: [main, develop] + pull_request: + branches: [main] + workflow_dispatch: # Allow manual trigger from UI + +env: # Global environment variables + NODE_VERSION: '20' + REGISTRY: ghcr.io + +jobs: + # ────────────────────────────────────────── + # JOB 1: Lint + # ────────────────────────────────────────── + lint: + name: Lint & Format Check + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: ${{ env.NODE_VERSION }} + cache: 'npm' # Automatically caches node_modules + + - name: Install dependencies + run: npm ci # Faster than npm install, uses lock file + + - name: Run ESLint + run: npm run lint + + - name: Check formatting + run: npm run format:check + + # ────────────────────────────────────────── + # JOB 2: Test (depends on nothing, runs in parallel with lint) + # ────────────────────────────────────────── + test: + name: Unit & Integration Tests + runs-on: ubuntu-latest + + services: # Spin up a real Postgres for integration tests + postgres: + image: postgres:16 + env: + POSTGRES_USER: testuser + POSTGRES_PASSWORD: testpass + POSTGRES_DB: testdb + ports: + - 5432:5432 + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: ${{ env.NODE_VERSION }} + cache: 'npm' + + - run: npm ci + + - name: Run tests with coverage + run: npm test -- --coverage + env: + DATABASE_URL: postgresql://testuser:testpass@localhost:5432/testdb + NODE_ENV: test + + - name: Upload coverage report + uses: actions/upload-artifact@v4 + with: + name: coverage-report + path: coverage/ + retention-days: 7 + + - name: Comment coverage on PR + if: github.event_name == 'pull_request' + uses: davelosert/vitest-coverage-report-action@v2 + + # ────────────────────────────────────────── + # JOB 3: Build (runs only after lint + test pass) + # ────────────────────────────────────────── + build: + name: Build + runs-on: ubuntu-latest + needs: [lint, test] # Waits for both jobs to succeed + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: ${{ env.NODE_VERSION }} + cache: 'npm' + + - run: npm ci + + - name: Build application + run: npm run build + env: + NODE_ENV: production + + - name: Upload build artifact + uses: actions/upload-artifact@v4 + with: + name: build-output + path: dist/ + retention-days: 14 + + # ────────────────────────────────────────── + # JOB 4: Notify (always runs, reports final status) + # ────────────────────────────────────────── + notify: + name: Notify + runs-on: ubuntu-latest + needs: [build] + if: always() # Run even if previous jobs failed + + steps: + - name: Notify Slack + uses: slackapi/slack-github-action@v1.26.0 + with: + payload: | + { + "text": "CI ${{ needs.build.result == 'success' && '✅ Passed' || '❌ Failed' }}: ${{ github.repository }}@${{ github.ref_name }}" + } + env: + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} +``` + +--- + +### GitLab CI — Complete Annotated Example + +```yaml +# .gitlab-ci.yml + +# ─── Global Configuration ─────────────────────────────────────────── +default: + image: node:20-alpine + before_script: + - npm ci --cache .npm --prefer-offline + cache: + key: + files: + - package-lock.json # Cache invalidates when lock file changes + paths: + - .npm/ + - node_modules/ + +variables: + NODE_ENV: test + POSTGRES_DB: testdb + POSTGRES_USER: testuser + POSTGRES_PASSWORD: testpass + +stages: + - validate + - test + - build + - publish + +# ─── Stage: validate ──────────────────────────────────────────────── +lint: + stage: validate + script: + - npm run lint + - npm run format:check + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + +typecheck: + stage: validate + script: + - npm run typecheck + +# ─── Stage: test ──────────────────────────────────────────────────── +unit-tests: + stage: test + script: + - npm run test:unit -- --coverage + coverage: '/Lines\s*:\s*(\d+\.?\d*)%/' # Extract coverage % for GitLab badge + artifacts: + when: always + reports: + junit: junit.xml # GitLab parses this for test result UI + coverage_report: + coverage_format: cobertura + path: coverage/cobertura-coverage.xml + paths: + - coverage/ + expire_in: 1 week + +integration-tests: + stage: test + services: + - postgres:16 + script: + - npm run test:integration + variables: + DATABASE_URL: "postgresql://testuser:testpass@postgres:5432/testdb" + +# ─── Stage: build ─────────────────────────────────────────────────── +build: + stage: build + script: + - NODE_ENV=production npm run build + artifacts: + paths: + - dist/ + expire_in: 2 weeks + only: + - main + - tags + +# ─── Stage: publish ───────────────────────────────────────────────── +publish-image: + stage: publish + image: docker:24 + services: + - docker:24-dind + before_script: + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY + script: + - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA . + - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest + - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA + - docker push $CI_REGISTRY_IMAGE:latest + only: + - main +``` + +--- + +### Jenkins — Declarative Pipeline Example + +```groovy +// Jenkinsfile + +pipeline { + agent any + + environment { + NODE_VERSION = '20' + REGISTRY = 'registry.example.com' + } + + tools { + nodejs "${NODE_VERSION}" + } + + options { + timeout(time: 30, unit: 'MINUTES') + disableConcurrentBuilds() // Prevent parallel runs on same branch + buildDiscarder(logRotator(numToKeepStr: '10')) + } + + stages { + stage('Checkout') { + steps { + checkout scm + } + } + + stage('Install') { + steps { + sh 'npm ci' + } + } + + stage('Validate') { + parallel { + stage('Lint') { + steps { sh 'npm run lint' } + } + stage('Type Check') { + steps { sh 'npm run typecheck' } + } + } + } + + stage('Test') { + steps { + sh 'npm test -- --coverage' + } + post { + always { + junit 'junit.xml' + publishHTML([ + reportDir: 'coverage', + reportFiles: 'index.html', + reportName: 'Coverage Report' + ]) + } + } + } + + stage('Build') { + when { + anyOf { + branch 'main' + tag pattern: 'v\\d+\\.\\d+\\.\\d+', comparator: 'REGEXP' + } + } + steps { + sh 'NODE_ENV=production npm run build' + } + } + } + + post { + success { + slackSend(color: 'good', message: "✅ Build passed: ${env.JOB_NAME} #${env.BUILD_NUMBER}") + } + failure { + slackSend(color: 'danger', message: "❌ Build failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}\n${env.BUILD_URL}") + } + } +} +``` + +--- + +## 2.4 Automated Testing + +### The Test Pyramid + +```text + ┌───────────┐ + /│ E2E │\ Few, slow, expensive + / └───────────┘ \ Run: nightly or on release + /─────────────────\ + / │ Integration │ \ Moderate number + / └───────────────┘ \ Run: every PR + /───────────────────────\ + / │ Unit Tests │ \ Many, fast, cheap + / └─────────────────┘ \ Run: every commit + /─────────────────────────────\ +``` + +**Guideline:** 70% unit / 20% integration / 10% E2E + +### Unit Tests — Node.js with Vitest + +```typescript +// src/utils/validate.ts +export function validateEmail(email: string): boolean { + return /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(email); +} + +export function validateAge(age: number): boolean { + return Number.isInteger(age) && age >= 0 && age <= 150; +} +``` + +```typescript +// src/utils/validate.test.ts +import { describe, it, expect } from 'vitest'; +import { validateEmail, validateAge } from './validate'; + +describe('validateEmail', () => { + it('accepts a valid email', () => { + expect(validateEmail('user@example.com')).toBe(true); + }); + + it('rejects email without @', () => { + expect(validateEmail('userexample.com')).toBe(false); + }); + + it('rejects empty string', () => { + expect(validateEmail('')).toBe(false); + }); +}); + +describe('validateAge', () => { + it('accepts age 0', () => expect(validateAge(0)).toBe(true)); + it('accepts age 25', () => expect(validateAge(25)).toBe(true)); + it('rejects negative age', () => expect(validateAge(-1)).toBe(false)); + it('rejects float', () => expect(validateAge(25.5)).toBe(false)); +}); +``` + +### Unit Tests — Python with pytest + +```python +# src/utils/validate.py +import re + +def validate_email(email: str) -> bool: + pattern = r'^[^\s@]+@[^\s@]+\.[^\s@]+$' + return bool(re.match(pattern, email)) + +def validate_age(age: int) -> bool: + return isinstance(age, int) and 0 <= age <= 150 +``` + +```python +# tests/test_validate.py +import pytest +from src.utils.validate import validate_email, validate_age + +class TestValidateEmail: + def test_valid_email(self): + assert validate_email("user@example.com") is True + + def test_missing_at(self): + assert validate_email("userexample.com") is False + + def test_empty_string(self): + assert validate_email("") is False + + @pytest.mark.parametrize("email", [ + "a@b.co", + "user+tag@domain.org", + "first.last@sub.domain.com", + ]) + def test_valid_emails(self, email): + assert validate_email(email) is True + + +class TestValidateAge: + def test_zero(self): + assert validate_age(0) is True + + def test_normal_age(self): + assert validate_age(25) is True + + def test_negative(self): + assert validate_age(-1) is False + + def test_float_rejected(self): + assert validate_age(25.5) is False # type: ignore +``` + +### Integration Test — API with Supertest + +```typescript +// tests/api/users.test.ts +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import request from 'supertest'; +import { app } from '../../src/app'; +import { db } from '../../src/db'; + +beforeAll(async () => { + await db.migrate.latest(); + await db.seed.run(); +}); + +afterAll(async () => { + await db.destroy(); +}); + +describe('POST /api/users', () => { + it('creates a user with valid data', async () => { + const res = await request(app) + .post('/api/users') + .send({ name: 'Alice', email: 'alice@example.com' }) + .expect(201); + + expect(res.body).toMatchObject({ + id: expect.any(Number), + name: 'Alice', + email: 'alice@example.com', + }); + }); + + it('rejects invalid email', async () => { + await request(app) + .post('/api/users') + .send({ name: 'Bob', email: 'not-an-email' }) + .expect(422); + }); +}); +``` + +### E2E Test — Playwright + +```typescript +// e2e/login.spec.ts +import { test, expect } from '@playwright/test'; + +test.describe('Login flow', () => { + test('user can log in with valid credentials', async ({ page }) => { + await page.goto('/login'); + + await page.fill('[data-testid="email"]', 'user@example.com'); + await page.fill('[data-testid="password"]', 'password123'); + await page.click('[data-testid="submit"]'); + + await expect(page).toHaveURL('/dashboard'); + await expect(page.locator('h1')).toContainText('Welcome'); + }); + + test('shows error on invalid credentials', async ({ page }) => { + await page.goto('/login'); + + await page.fill('[data-testid="email"]', 'bad@example.com'); + await page.fill('[data-testid="password"]', 'wrong'); + await page.click('[data-testid="submit"]'); + + await expect(page.locator('[data-testid="error"]')).toBeVisible(); + await expect(page.locator('[data-testid="error"]')).toContainText('Invalid credentials'); + }); +}); +``` + +#### playwright.config.ts — CI-optimized settings + +```typescript +import { defineConfig } from '@playwright/test'; + +export default defineConfig({ + testDir: './e2e', + fullyParallel: true, + retries: process.env.CI ? 2 : 0, // Retry flaky tests in CI only + workers: process.env.CI ? 4 : undefined, + reporter: process.env.CI + ? [['junit', { outputFile: 'e2e-results.xml' }], ['html']] + : 'list', + use: { + baseURL: process.env.BASE_URL || 'http://localhost:3000', + screenshot: 'only-on-failure', + video: 'retain-on-failure', + }, +}); +``` + +### Coverage Enforcement in CI + +```yaml +# GitHub Actions — fail if coverage drops below 80% +- name: Run tests with coverage + run: | + npm test -- --coverage --coverageThreshold='{"global":{"lines":80,"functions":80,"branches":75}}' +``` + +```ini +# pytest.ini — enforce coverage threshold +[tool:pytest] +addopts = --cov=src --cov-report=xml --cov-report=term-missing --cov-fail-under=80 +``` + +--- + +## 2.5 Build Automation + +### Makefile — Universal Build Interface + +```makefile +# Makefile +.PHONY: install lint test build clean docker-build + +install: + npm ci + +lint: + npm run lint + npm run format:check + +test: + npm test -- --coverage + +build: + NODE_ENV=production npm run build + +clean: + rm -rf dist/ coverage/ node_modules/ + +docker-build: + docker build -t myapp:$(shell git rev-parse --short HEAD) . + +# CI-specific target — run everything in order +ci: install lint test build +``` + +### package.json Scripts + +```json +{ + "scripts": { + "dev": "vite", + "build": "tsc && vite build", + "preview": "vite preview", + "lint": "eslint src --ext .ts,.tsx --max-warnings 0", + "format": "prettier --write .", + "format:check": "prettier --check .", + "typecheck": "tsc --noEmit", + "test": "vitest run", + "test:watch": "vitest", + "test:ui": "vitest --ui", + "test:e2e": "playwright test", + "ci": "npm run lint && npm run typecheck && npm run test && npm run build" + } +} +``` + +### Dependency Locking Best Practices + +```bash +# Always commit lock files — they ensure reproducible builds +git add package-lock.json # Node.js +git add poetry.lock # Python +git add Cargo.lock # Rust +git add go.sum # Go + +# Use exact install commands in CI (not `npm install`) +npm ci # Node — uses lock file, fails if mismatched +pip install -r requirements.txt # Python +cargo build # Rust — uses Cargo.lock automatically +``` + +--- diff --git a/Lessons/03-intermediate-ci.md b/Lessons/03-intermediate-ci.md new file mode 100644 index 0000000..2723f3e --- /dev/null +++ b/Lessons/03-intermediate-ci.md @@ -0,0 +1,425 @@ +# Level 3 — Intermediate CI + +## 3.1 Docker in CI + +### Multi-Stage Dockerfile + +A well-structured Dockerfile that keeps the production image small and the build environment separate: + +```dockerfile +# ─── Stage 1: Dependencies ─────────────────────────────────────────── +FROM node:20-alpine AS deps +WORKDIR /app +COPY package*.json ./ +RUN npm ci --only=production && cp -r node_modules /tmp/prod-deps +RUN npm ci # Install all deps for build + +# ─── Stage 2: Build ────────────────────────────────────────────────── +FROM node:20-alpine AS builder +WORKDIR /app +COPY --from=deps /app/node_modules ./node_modules +COPY . . +RUN npm run build + +# ─── Stage 3: Production Image ─────────────────────────────────────── +FROM node:20-alpine AS runner +WORKDIR /app + +ENV NODE_ENV=production + +# Run as non-root user (security best practice) +RUN addgroup --system --gid 1001 nodejs +RUN adduser --system --uid 1001 appuser + +COPY --from=deps /tmp/prod-deps ./node_modules +COPY --from=builder /app/dist ./dist +COPY --from=builder /app/package.json ./ + +USER appuser + +EXPOSE 3000 +CMD ["node", "dist/index.js"] +``` + +### Building and Pushing in GitHub Actions + +```yaml +# .github/workflows/docker.yml + +name: Build & Push Docker Image + +on: + push: + branches: [main] + tags: ['v*'] + +jobs: + docker: + runs-on: ubuntu-latest + permissions: + contents: read + packages: write # Required to push to GHCR + + steps: + - uses: actions/checkout@v4 + + # Enable Docker layer caching via GitHub cache + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Log in to GitHub Container Registry + uses: docker/login-action@v3 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Generate image tags based on git context + - name: Extract metadata + id: meta + uses: docker/metadata-action@v5 + with: + images: ghcr.io/${{ github.repository }} + tags: | + type=ref,event=branch + type=semver,pattern={{version}} + type=semver,pattern={{major}}.{{minor}} + type=sha,prefix=sha- + + - name: Build and push + uses: docker/build-push-action@v5 + with: + context: . + push: true + tags: ${{ steps.meta.outputs.tags }} + labels: ${{ steps.meta.outputs.labels }} + cache-from: type=gha # Use GitHub Actions cache for layers + cache-to: type=gha,mode=max +``` + +### Docker Layer Caching Strategy + +```dockerfile +# ✅ GOOD: Copy package files BEFORE source code +# This way, npm install is only re-run when package.json changes +COPY package*.json ./ + +# Cached unless package*.json changed +RUN npm ci +# Source changes don't invalidate the npm layer +COPY src/ ./src/ +RUN npm run build + +# ❌ BAD: Copy everything at once — any source change busts the cache +COPY . . +RUN npm ci +RUN npm run build +``` + +--- + +## 3.2 Code Quality Gates + +### ESLint Configuration + +#### .eslintrc.json + +```json +{ + "extends": [ + "eslint:recommended", + "plugin:@typescript-eslint/recommended", + "plugin:import/recommended" + ], + "rules": { + "no-console": "warn", + "no-unused-vars": "off", + "@typescript-eslint/no-unused-vars": "error", + "@typescript-eslint/no-explicit-any": "warn", + "import/order": ["error", { "alphabetize": { "order": "asc" } }] + } +} +``` + +### Prettier Configuration + +#### .prettierrc + +```json +{ + "semi": true, + "singleQuote": true, + "tabWidth": 2, + "trailingComma": "es5", + "printWidth": 100, + "arrowParens": "avoid" +} +``` + +### Pre-commit Hooks with Husky + +Run quality checks locally before committing — mirrors what CI will do: + +```bash +npm install --save-dev husky lint-staged +npx husky init +``` + +#### package.json + +```json +{ + "lint-staged": { + "*.{ts,tsx}": ["eslint --fix", "prettier --write"], + "*.{json,md,yml}": ["prettier --write"] + } +} +``` + +```bash +# .husky/pre-commit +#!/bin/sh +npx lint-staged +``` + +### SonarQube Integration in CI + +```yaml +# GitHub Actions SonarQube scan +- name: SonarQube Scan + uses: SonarSource/sonarqube-scan-action@master + env: + SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} + SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }} +``` + +```properties +# sonar-project.properties +sonar.projectKey=my-project +sonar.sources=src +sonar.tests=tests +sonar.javascript.lcov.reportPaths=coverage/lcov.info +sonar.testExecutionReportPaths=junit.xml + +# Quality gate thresholds +sonar.qualitygate.wait=true # Fail CI if quality gate fails +``` + +--- + +## 3.3 Security Scanning + +### Secret Detection — Preventing Leaks Before They Happen + +```yaml +# GitHub Actions — scan for secrets in every PR +- name: Scan for secrets (Gitleaks) + uses: gitleaks/gitleaks-action@v2 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} +``` + +```toml +# .gitleaks.toml — custom rules +[allowlist] + description = "Allowlist for known safe patterns" + regexes = [ + '''EXAMPLE_KEY''', + '''test_.*_key''' + ] + paths = [ + '''tests/fixtures/.*''' + ] +``` + +### Dependency Vulnerability Scanning + +```yaml +# Scan Node.js dependencies +- name: Audit dependencies + run: npm audit --audit-level=high + # Fails if HIGH or CRITICAL vulnerabilities found + +# Or use Snyk for richer reports +- name: Snyk vulnerability scan + uses: snyk/actions/node@master + env: + SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }} + with: + args: --severity-threshold=high +``` + +```yaml +# Scan Python dependencies +- name: Safety check + run: | + pip install safety + safety check -r requirements.txt --json > safety-report.json +``` + +### Container Image Scanning with Trivy + +```yaml +- name: Scan Docker image for vulnerabilities + uses: aquasecurity/trivy-action@master + with: + image-ref: 'ghcr.io/${{ github.repository }}:${{ github.sha }}' + format: 'sarif' + output: 'trivy-results.sarif' + severity: 'CRITICAL,HIGH' + exit-code: '1' # Fail the pipeline if vulnerabilities found + +- name: Upload Trivy results to GitHub Security tab + uses: github/codeql-action/upload-sarif@v3 + with: + sarif_file: 'trivy-results.sarif' +``` + +### CodeQL Static Analysis + +```yaml +# .github/workflows/codeql.yml +name: CodeQL Analysis + +on: + push: + branches: [main] + pull_request: + branches: [main] + schedule: + - cron: '0 6 * * 1' # Weekly scan every Monday at 6am + +jobs: + analyze: + runs-on: ubuntu-latest + permissions: + security-events: write + actions: read + contents: read + + strategy: + matrix: + language: [javascript, python] + + steps: + - uses: actions/checkout@v4 + + - name: Initialize CodeQL + uses: github/codeql-action/init@v3 + with: + languages: ${{ matrix.language }} + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v3 +``` + +--- + +## 3.4 Artifact Management + +### Semantic Versioning in CI + +```yaml +# .github/workflows/release.yml +name: Release + +on: + push: + branches: [main] + +jobs: + release: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 # Full history needed for changelog + + - name: Semantic Release + uses: cycjimmy/semantic-release-action@v4 + with: + semantic_version: 23 + extra_plugins: | + @semantic-release/changelog + @semantic-release/git + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + NPM_TOKEN: ${{ secrets.NPM_TOKEN }} +``` + +#### .releaserc.json — semantic-release config + +```json +{ + "branches": ["main"], + "plugins": [ + "@semantic-release/commit-analyzer", + "@semantic-release/release-notes-generator", + ["@semantic-release/changelog", { + "changelogFile": "CHANGELOG.md" + }], + ["@semantic-release/npm", { + "npmPublish": true + }], + ["@semantic-release/git", { + "assets": ["CHANGELOG.md", "package.json"], + "message": "chore(release): ${nextRelease.version} [skip ci]" + }], + "@semantic-release/github" + ] +} +``` + +### Publishing to PyPI from CI + +```yaml +- name: Build Python package + run: | + pip install build + python -m build + +- name: Publish to PyPI + uses: pypa/gh-action-pypi-publish@release/v1 + with: + password: ${{ secrets.PYPI_API_TOKEN }} + # Use TestPyPI first: repository-url: https://test.pypi.org/legacy/ +``` + +--- + +## 3.5 Notifications & Observability + +### Slack Notifications + +```yaml +- name: Notify Slack on failure + if: failure() + uses: slackapi/slack-github-action@v1.26.0 + with: + payload: | + { + "blocks": [ + { + "type": "section", + "text": { + "type": "mrkdwn", + "text": "❌ *CI Failed*: <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Run>\n*Repo:* ${{ github.repository }}\n*Branch:* `${{ github.ref_name }}`\n*Commit:* `${{ github.sha }}`\n*Author:* ${{ github.actor }}" + } + } + ] + } + env: + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} + SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK +``` + +### Build Status Badge + +```markdown + +![CI](https://github.com/owner/repo/actions/workflows/ci.yml/badge.svg) +![Coverage](https://codecov.io/gh/owner/repo/branch/main/graph/badge.svg) +![Security](https://snyk.io/test/github/owner/repo/badge.svg) +``` + +--- diff --git a/Lessons/04-advanced-ci.md b/Lessons/04-advanced-ci.md new file mode 100644 index 0000000..5e136ea --- /dev/null +++ b/Lessons/04-advanced-ci.md @@ -0,0 +1,521 @@ +# Level 4 — Advanced CI + +## 4.1 Pipeline Optimization + +### Parallelization with Matrix Builds + +```yaml +# Test across multiple Node.js versions and operating systems +jobs: + test: + strategy: + fail-fast: false # Don't cancel other matrix jobs on failure + matrix: + os: [ubuntu-latest, windows-latest, macos-latest] + node: [18, 20, 22] + exclude: + - os: windows-latest + node: 18 # Skip this combination + + runs-on: ${{ matrix.os }} + name: Test (Node ${{ matrix.node }} / ${{ matrix.os }}) + + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: ${{ matrix.node }} + - run: npm ci + - run: npm test +``` + +### Conditional Execution — Path-Based Triggers + +```yaml +# Only run frontend tests when frontend code changes +jobs: + frontend-tests: + if: | + contains(github.event.head_commit.modified, 'frontend/') || + contains(github.event.head_commit.added, 'frontend/') + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: cd frontend && npm ci && npm test +``` + +```yaml +# GitLab: rules with changes filter +frontend-test: + script: cd frontend && npm test + rules: + - changes: + - frontend/**/* + - package.json + +backend-test: + script: cd backend && go test ./... + rules: + - changes: + - backend/**/* + - go.mod + - go.sum +``` + +### Dependency Graph — DAG Pipelines in GitLab + +```yaml +# Jobs run in parallel when they don't share a stage dependency +stages: [build, test, deploy] + +build-frontend: + stage: build + script: npm run build:frontend + artifacts: + paths: [dist/frontend] + +build-backend: + stage: build # Runs in PARALLEL with build-frontend + script: go build ./... + artifacts: + paths: [bin/] + +test-frontend: + stage: test + needs: [build-frontend] # Only waits for build-frontend, not build-backend + script: npm test + +test-backend: + stage: test + needs: [build-backend] # Only waits for build-backend + script: go test ./... + +deploy: + stage: deploy + needs: [test-frontend, test-backend] # Waits for BOTH tests + script: ./deploy.sh +``` + +### Advanced Caching Strategies + +```yaml +# GitHub Actions — cache npm with composite key +- name: Cache node modules + uses: actions/cache@v4 + with: + path: ~/.npm + key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }} + restore-keys: | + ${{ runner.os }}-node- # Partial match fallback + +# Cache pip packages +- name: Cache pip + uses: actions/cache@v4 + with: + path: ~/.cache/pip + key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }} + restore-keys: ${{ runner.os }}-pip- + +# Cache Rust/Cargo (more complex — cache both registry and target) +- name: Cache Cargo + uses: Swatinem/rust-cache@v2 + with: + workspaces: ". -> target" + cache-on-failure: true +``` + +--- + +## 4.2 Self-Hosted Runners + +### GitHub Actions Self-Hosted Runner Setup + +```bash +# On your server/VM — register a runner +mkdir actions-runner && cd actions-runner +curl -o actions-runner-linux-x64.tar.gz -L \ + https://github.com/actions/runner/releases/download/v2.317.0/actions-runner-linux-x64-2.317.0.tar.gz +tar xzf ./actions-runner-linux-x64.tar.gz + +# Register with your repo +./config.sh --url https://github.com/OWNER/REPO --token YOUR_TOKEN + +# Install and start as a service +sudo ./svc.sh install +sudo ./svc.sh start +``` + +```yaml +# Use your self-hosted runner in a workflow +jobs: + gpu-job: + runs-on: [self-hosted, linux, gpu] # Target by labels + steps: + - run: nvidia-smi # Only works on your GPU runner +``` + +### Ephemeral Runners with Docker + +```dockerfile +# Dockerfile.runner +FROM ubuntu:22.04 +RUN apt-get update && apt-get install -y \ + curl git jq libicu70 \ + && rm -rf /var/lib/apt/lists/* + +# Install GitHub Actions runner +ARG RUNNER_VERSION=2.317.0 +RUN curl -o /tmp/runner.tar.gz -L \ + https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \ + && tar xzf /tmp/runner.tar.gz -C /opt/actions-runner + +COPY entrypoint.sh /entrypoint.sh +RUN chmod +x /entrypoint.sh + +ENTRYPOINT ["/entrypoint.sh"] +``` + +```bash +#!/bin/bash +# entrypoint.sh — register, run once, then deregister (ephemeral) +/opt/actions-runner/config.sh \ + --url "${GITHUB_URL}" \ + --token "${RUNNER_TOKEN}" \ + --name "ephemeral-$(hostname)" \ + --ephemeral \ + --unattended + +/opt/actions-runner/run.sh +``` + +--- + +## 4.3 Advanced Testing Strategies + +### Test Sharding — Split a Large Suite Across Runners + +```yaml +# GitHub Actions — shard Playwright E2E tests across 4 runners +jobs: + e2e: + strategy: + matrix: + shard: [1, 2, 3, 4] + + steps: + - uses: actions/checkout@v4 + - run: npm ci + - run: npx playwright test --shard=${{ matrix.shard }}/4 + - uses: actions/upload-artifact@v4 + with: + name: blob-report-${{ matrix.shard }} + path: blob-report/ + + # Merge shard reports into one HTML report + merge-reports: + needs: [e2e] + steps: + - uses: actions/download-artifact@v4 + with: + pattern: blob-report-* + merge-multiple: true + path: all-blob-reports + - run: npx playwright merge-reports --reporter html ./all-blob-reports +``` + +### Contract Testing with Pact + +```typescript +// consumer.pact.test.ts — define the contract from the consumer's perspective +import { PactV3, MatchersV3 } from '@pact-foundation/pact'; + +const provider = new PactV3({ + consumer: 'OrderService', + provider: 'UserService', +}); + +describe('UserService contract', () => { + it('returns a user by ID', () => { + return provider + .given('User 123 exists') + .uponReceiving('a request for user 123') + .withRequest({ method: 'GET', path: '/users/123' }) + .willRespondWith({ + status: 200, + body: { + id: MatchersV3.integer(123), + name: MatchersV3.string('Alice'), + email: MatchersV3.email('alice@example.com'), + }, + }) + .executeTest(async (mockServer) => { + const client = new UserClient(mockServer.url); + const user = await client.getUser(123); + expect(user.name).toBe('Alice'); + }); + }); +}); +``` + +### Performance Testing with k6 + +```javascript +// load-test.js +import http from 'k6/http'; +import { check, sleep } from 'k6'; +import { Rate } from 'k6/metrics'; + +const errorRate = new Rate('errors'); + +export const options = { + stages: [ + { duration: '30s', target: 20 }, // Ramp up to 20 users + { duration: '1m', target: 20 }, // Stay at 20 + { duration: '20s', target: 0 }, // Ramp down + ], + thresholds: { + http_req_duration: ['p(95)<500'], // 95% of requests under 500ms + errors: ['rate<0.01'], // Error rate under 1% + }, +}; + +export default function () { + const res = http.get(`${__ENV.BASE_URL}/api/users`); + + check(res, { + 'status is 200': (r) => r.status === 200, + 'response time < 500ms': (r) => r.timings.duration < 500, + }); + + errorRate.add(res.status !== 200); + sleep(1); +} +``` + +```yaml +# Run k6 in CI +- name: Run k6 load test + uses: grafana/k6-action@v0.3.1 + with: + filename: load-test.js + env: + BASE_URL: https://staging.example.com + K6_CLOUD_TOKEN: ${{ secrets.K6_CLOUD_TOKEN }} +``` + +--- + +## 4.4 GitOps & CI/CD Integration + +### CI → CD Trigger Pattern + +```yaml +# CI pipeline (build repo) — triggers CD on success +- name: Trigger deployment pipeline + if: github.ref == 'refs/heads/main' && success() + uses: peter-evans/repository-dispatch@v3 + with: + token: ${{ secrets.CD_DISPATCH_TOKEN }} + repository: org/cd-config-repo + event-type: deploy-staging + client-payload: | + { + "image": "ghcr.io/${{ github.repository }}", + "tag": "${{ github.sha }}", + "environment": "staging" + } +``` + +### Kubernetes Manifest Update (GitOps) + +```yaml +# In the CD repo — update image tag and commit (ArgoCD/Flux picks it up) +- name: Update image tag in Kubernetes manifest + run: | + IMAGE_TAG=${{ github.event.client_payload.tag }} + sed -i "s|image: .*|image: ghcr.io/org/app:${IMAGE_TAG}|" k8s/staging/deployment.yaml + git config user.name "ci-bot" + git config user.email "ci-bot@example.com" + git add k8s/staging/deployment.yaml + git commit -m "chore: deploy ${IMAGE_TAG} to staging [skip ci]" + git push +``` + +### Environment Promotion Pipeline + +```yaml +# GitLab CI — promote through environments +stages: [build, deploy-dev, deploy-staging, deploy-prod] + +.deploy-template: &deploy + image: bitnami/kubectl:latest + script: + - kubectl set image deployment/app app=$IMAGE:$CI_COMMIT_SHA -n $NAMESPACE + +deploy-dev: + <<: *deploy + stage: deploy-dev + variables: + NAMESPACE: dev + environment: + name: development + url: https://dev.example.com + only: [main] + +deploy-staging: + <<: *deploy + stage: deploy-staging + variables: + NAMESPACE: staging + environment: + name: staging + url: https://staging.example.com + when: manual # Require human approval + only: [main] + +deploy-prod: + <<: *deploy + stage: deploy-prod + variables: + NAMESPACE: production + environment: + name: production + url: https://example.com + when: manual + only: [main] + allow_failure: false +``` + +--- + +## 4.5 Feature Flags & Progressive Delivery + +### Feature Flag Pattern in Code + +```typescript +// Using OpenFeature SDK +import { OpenFeature } from '@openfeature/server-sdk'; + +const client = OpenFeature.getClient('my-app'); + +export async function handleCheckout(userId: string) { + const useNewCheckout = await client.getBooleanValue( + 'new-checkout-flow', + false, // Default: disabled + { targetingKey: userId } + ); + + if (useNewCheckout) { + return newCheckoutFlow(userId); + } else { + return legacyCheckoutFlow(userId); + } +} +``` + +### Canary Release with GitHub Actions + Kubernetes + +```yaml +- name: Deploy canary (10% traffic) + run: | + # Deploy new version as canary + kubectl apply -f k8s/canary/deployment.yaml + + # Set traffic split: 90% stable, 10% canary + kubectl apply -f - < 0.01" | bc -l) )); then + echo "Error rate too high, rolling back" + kubectl rollout undo deployment/app-canary + exit 1 + fi +``` + +--- + +## 4.6 Infrastructure as Code in CI + +### Terraform Validation Pipeline + +```yaml +# .github/workflows/terraform.yml +name: Terraform CI + +on: + pull_request: + paths: + - 'infrastructure/**' + +jobs: + terraform: + runs-on: ubuntu-latest + defaults: + run: + working-directory: infrastructure/ + + steps: + - uses: actions/checkout@v4 + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_version: 1.9.0 + + - name: Terraform Format Check + run: terraform fmt -check -recursive + + - name: Terraform Init + run: terraform init -backend=false # Skip remote backend in PR + + - name: Terraform Validate + run: terraform validate + + - name: Run tflint (Terraform linter) + uses: terraform-linters/setup-tflint@v4 + - run: tflint --recursive + + - name: Run Checkov (security scan) + uses: bridgecrewio/checkov-action@master + with: + directory: infrastructure/ + framework: terraform + output_format: sarif + output_file_path: checkov.sarif + + - name: Terraform Plan + run: terraform plan -out=plan.tfplan + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + + - name: Comment plan on PR + uses: actions/github-script@v7 + with: + script: | + const plan = require('fs').readFileSync('plan.txt', 'utf8'); + github.rest.issues.createComment({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + body: `## Terraform Plan\n\`\`\`hcl\n${plan.slice(0, 60000)}\n\`\`\`` + }); +``` + +--- diff --git a/Lessons/05-mastery.md b/Lessons/05-mastery.md new file mode 100644 index 0000000..a5a199d --- /dev/null +++ b/Lessons/05-mastery.md @@ -0,0 +1,239 @@ +# Level 5 — Mastery + +## 5.1 CI Architecture at Scale + +### Monorepo CI with Nx (Affected-Only Builds) + +#### nx.json + +```json +{ + "affected": { + "defaultBase": "main" + }, + "tasksRunnerOptions": { + "default": { + "runner": "@nrwl/nx-cloud", + "options": { + "cacheableOperations": ["build", "test", "lint"], + "accessToken": "your-nx-cloud-token" + } + } + } +} +``` + +`nrwl/nx-cloud` used as distributed cache + +```yaml +# Only test/build affected projects — not the entire monorepo +- name: Get affected projects + run: | + AFFECTED=$(npx nx show projects --affected --type=app | tr '\n' ',') + echo "AFFECTED=${AFFECTED}" >> $GITHUB_ENV + +- name: Run affected tests + run: npx nx affected --target=test --parallel=4 + +- name: Build affected apps + run: npx nx affected --target=build --parallel=2 +``` + +### Distributed Caching with Bazel + +```python +# .bazelrc — remote cache configuration +build --remote_cache=grpcs://cache.example.com:443 +build --remote_header=Authorization=Bearer $BAZEL_CACHE_TOKEN +build --disk_cache=~/.cache/bazel + +# Enable remote execution for CI +build:ci --remote_executor=grpcs://rbe.example.com:443 +build:ci --remote_instance_name=default +``` + +--- + +## 5.2 Platform Engineering + +### Reusable Workflow Library (GitHub) + +```yaml +# .github/workflows/_reusable-node-ci.yml — shared template + +name: Reusable Node.js CI + +on: + workflow_call: + inputs: + node-version: + type: string + default: '20' + working-directory: + type: string + default: '.' + test-command: + type: string + default: 'npm test' + secrets: + SLACK_WEBHOOK_URL: + required: false + +jobs: + ci: + runs-on: ubuntu-latest + defaults: + run: + working-directory: ${{ inputs.working-directory }} + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: ${{ inputs.node-version }} + cache: 'npm' + cache-dependency-path: ${{ inputs.working-directory }}/package-lock.json + - run: npm ci + - run: npm run lint + - run: ${{ inputs.test-command }} + - run: npm run build +``` + +```yaml +# Consuming the reusable workflow in any project +name: CI +on: [push, pull_request] + +jobs: + ci: + uses: org/.github/.github/workflows/_reusable-node-ci.yml@main + with: + node-version: '22' + test-command: 'npm run test:coverage' + secrets: + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} +``` + +### GitLab CI/CD Components (Catalog) + +```yaml +# components/node-ci/templates/node.yml +spec: + inputs: + node_version: + default: '20' + coverage_threshold: + default: '80' + +--- + +node-lint: + image: node:$[[ inputs.node_version ]]-alpine + script: + - npm ci + - npm run lint + +node-test: + image: node:$[[ inputs.node_version ]]-alpine + script: + - npm ci + - npm test -- --coverage --coverageThreshold='{"global":{"lines":$[[ inputs.coverage_threshold ]]}}' + coverage: '/Lines\s*:\s*(\d+\.?\d*)%/' +``` + +```yaml +# Consuming the component +include: + - component: $CI_SERVER_FQDN/org/ci-catalog/node-ci@v1.0 + inputs: + node_version: '22' + coverage_threshold: '85' +``` + +--- + +## 5.3 DORA Metrics + +### Tracking the Four Key Metrics + +| Metric | How to Measure | Target (Elite) | +|---------------------------|--------------------------------------------------|----------------| +| **Deployment Frequency** | Count deploys to prod per day via CI logs | Multiple/day | +| **Lead Time for Changes** | Time from first commit to prod deploy | < 1 hour | +| **Change Failure Rate** | % of deploys that trigger a rollback or incident | < 5% | +| **MTTR** | Time between incident alert and resolved deploy | < 1 hour | + +### Collecting Metrics from GitHub Actions + +```python +# scripts/collect_dora.py — parse workflow runs and emit metrics + +import requests +import datetime + +GITHUB_TOKEN = os.environ['GITHUB_TOKEN'] +REPO = 'org/my-app' + +headers = {'Authorization': f'Bearer {GITHUB_TOKEN}'} + +def get_deploy_frequency(days=30): + since = (datetime.datetime.now() - datetime.timedelta(days=days)).isoformat() + runs = requests.get( + f'https://api.github.com/repos/{REPO}/actions/workflows/deploy.yml/runs', + params={'status': 'success', 'branch': 'main', 'created': f'>{since}'}, + headers=headers + ).json() + + total_deploys = runs['total_count'] + per_day = total_deploys / days + return {'total': total_deploys, 'per_day': round(per_day, 2)} + +def get_lead_time(): + # Compare first commit timestamp vs successful deploy timestamp + # Emit to Grafana/Datadog/custom dashboard + pass +``` + +--- + +## 5.4 CI Culture + +### The CI Social Contract + +These are team agreements — equally as important as the technical setup: + +```markdown +## Our CI Contract + +1. **Green main is sacred** — never merge a PR that breaks main. +2. **Fix the build before anything else** — a broken pipeline blocks everyone. +3. **You broke it, you fix it** — the author of the breaking commit owns the fix. +4. **Don't disable tests to make CI pass** — fix the root cause. +5. **PRs stay small** — large PRs are review liabilities and merge risks. +6. **Short-lived branches** — branches older than 2 days need a plan. +7. **No "works on my machine"** — if CI fails, it's a real problem. +8. **Review pipeline metrics weekly** — slow pipelines are tech debt. +``` + +### Trunk-Based Development Checklist + +```markdown +## TBD Readiness Checklist + +Infrastructure: +- [ ] CI pipeline runs in < 10 minutes +- [ ] Branch protection enforces CI before merge +- [ ] Automated deployment to at least one environment on merge to main + +Practices: +- [ ] Feature flags are available for hiding incomplete features +- [ ] All developers commit to main (or short-lived branches < 1 day) +- [ ] PR review SLA: < 2 hours during business hours +- [ ] On-call rotation to handle build failures quickly + +Code Health: +- [ ] Test suite is fast enough to run locally in < 2 minutes +- [ ] Flaky tests are quarantined and tracked +- [ ] Build produces the same output regardless of environment +``` + +--- diff --git a/Lessons/99-reference.md b/Lessons/99-reference.md new file mode 100644 index 0000000..bce5287 --- /dev/null +++ b/Lessons/99-reference.md @@ -0,0 +1,206 @@ +# Reference: Full Pipeline Examples + +## Complete Node.js TypeScript CI/CD Pipeline + +```yaml +# .github/workflows/full-ci-cd.yml +name: Full CI/CD Pipeline + +on: + push: + branches: [main] + tags: ['v*'] + pull_request: + branches: [main] + +concurrency: # Cancel in-progress runs for same PR + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +env: + REGISTRY: ghcr.io + IMAGE_NAME: ${{ github.repository }} + +jobs: + # ─── Quality Checks (parallel) ────────────────────────────────────── + lint: + name: Lint & Type Check + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: { node-version: '20', cache: 'npm' } + - run: npm ci + - run: npm run lint && npm run typecheck + + security: + name: Security Scan + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: { node-version: '20', cache: 'npm' } + - run: npm ci + - run: npm audit --audit-level=high + - uses: gitleaks/gitleaks-action@v2 + env: { GITHUB_TOKEN: '${{ secrets.GITHUB_TOKEN }}' } + + # ─── Tests (parallel) ─────────────────────────────────────────────── + unit-tests: + name: Unit Tests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: { node-version: '20', cache: 'npm' } + - run: npm ci + - run: npm run test:unit -- --coverage + - uses: codecov/codecov-action@v4 + with: + token: ${{ secrets.CODECOV_TOKEN }} + + integration-tests: + name: Integration Tests + runs-on: ubuntu-latest + services: + postgres: + image: postgres:16-alpine + env: + POSTGRES_USER: test + POSTGRES_PASSWORD: test + POSTGRES_DB: testdb + options: --health-cmd pg_isready --health-interval 10s --health-retries 5 + ports: ['5432:5432'] + redis: + image: redis:7-alpine + ports: ['6379:6379'] + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: { node-version: '20', cache: 'npm' } + - run: npm ci + - run: npm run test:integration + env: + DATABASE_URL: postgresql://test:test@localhost:5432/testdb + REDIS_URL: redis://localhost:6379 + + # ─── Build ────────────────────────────────────────────────────────── + build: + name: Build & Push Image + runs-on: ubuntu-latest + needs: [lint, security, unit-tests, integration-tests] + permissions: + contents: read + packages: write + outputs: + image-tag: ${{ steps.meta.outputs.version }} + + steps: + - uses: actions/checkout@v4 + - uses: docker/setup-buildx-action@v3 + - uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + - id: meta + uses: docker/metadata-action@v5 + with: + images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} + tags: | + type=semver,pattern={{version}} + type=sha,prefix=sha- + type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }} + - uses: docker/build-push-action@v5 + with: + push: ${{ github.event_name != 'pull_request' }} + tags: ${{ steps.meta.outputs.tags }} + cache-from: type=gha + cache-to: type=gha,mode=max + + # ─── Deploy to Staging (on merge to main) ─────────────────────────── + deploy-staging: + name: Deploy to Staging + runs-on: ubuntu-latest + needs: build + if: github.ref == 'refs/heads/main' + environment: + name: staging + url: https://staging.example.com + + steps: + - run: | + echo "Deploying ${{ needs.build.outputs.image-tag }} to staging" + # kubectl / helm / ArgoCD / etc. + + # ─── Deploy to Production (on tag) ────────────────────────────────── + deploy-prod: + name: Deploy to Production + runs-on: ubuntu-latest + needs: [build, deploy-staging] + if: startsWith(github.ref, 'refs/tags/v') + environment: + name: production + url: https://example.com + + steps: + - run: | + echo "Deploying ${{ needs.build.outputs.image-tag }} to production" +``` + +--- + +## Complete Python FastAPI CI Pipeline + +```yaml +# .github/workflows/python-ci.yml +name: Python CI + +on: [push, pull_request] + +jobs: + ci: + runs-on: ubuntu-latest + strategy: + matrix: + python-version: ['3.11', '3.12'] + + steps: + - uses: actions/checkout@v4 + + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + cache: 'pip' + + - name: Install dependencies + run: | + pip install --upgrade pip + pip install -r requirements.txt + pip install -r requirements-dev.txt + + - name: Lint with Ruff + run: ruff check . --output-format=github + + - name: Format check with Black + run: black --check . + + - name: Type check with mypy + run: mypy src/ + + - name: Run tests + run: pytest -v --cov=src --cov-report=xml --cov-fail-under=80 + + - name: Upload coverage + uses: codecov/codecov-action@v4 + with: + token: ${{ secrets.CODECOV_TOKEN }} + files: coverage.xml +``` + +--- + +*This guide is a living document. CI tooling evolves rapidly — always verify against official documentation.* +*Last updated: March 2026* + diff --git a/Lessons/ci.md b/Lessons/ci.md deleted file mode 100644 index f8c34df..0000000 --- a/Lessons/ci.md +++ /dev/null @@ -1,2746 +0,0 @@ -# Continuous Integration — Complete Practical Guide - -> A hands-on, sample-driven companion to the CI Learning Path. -> Every concept is paired with working code, real configurations, and actionable explanations. - ---- - -## Table of Contents - -1. [Level 1 — Foundations](#level-1--foundations) -2. [Level 2 — Core CI Skills](#level-2--core-ci-skills) -3. [Level 3 — Intermediate CI](#level-3--intermediate-ci) -4. [Level 4 — Advanced CI](#level-4--advanced-ci) -5. [Level 5 — Mastery](#level-5--mastery) -6. [Reference: Full Pipeline Examples](#reference-full-pipeline-examples) - ---- - -# Level 1 — Foundations - -## 1.1 The SDLC and Where CI Fits - -### The Traditional Problem - -Before CI, teams worked in isolation for days or weeks, then attempted to merge everything at once — a painful process known as **"integration hell"**. - -```text -Developer A (2 weeks of work) ──┐ -Developer B (2 weeks of work) ──┼──► MERGE ──► 💥 Conflicts everywhere -Developer C (2 weeks of work) ──┘ -``` - -### The CI Solution - -Integrate continuously — every change triggers an automated pipeline: - -```text -Developer A (1 commit) ──► Pipeline ──► ✅ Merged -Developer B (1 commit) ──► Pipeline ──► ✅ Merged -Developer C (1 commit) ──► Pipeline ──► ❌ Tests fail → Fix immediately -``` - -### The SDLC with CI - -```text -┌─────────┐ ┌──────┐ ┌───────┐ ┌──────┐ ┌─────────┐ ┌────────┐ -│ PLAN │──►│ CODE │──►│ BUILD │──►│ TEST │──►│ RELEASE │──►│ DEPLOY │ -└─────────┘ └──────┘ └───────┘ └──────┘ └─────────┘ └────────┘ - │ ▲ - └───── CI ─────┘ (automated feedback on every commit) -``` - -| Stage | Without CI | With CI | -|---------|-----------------------------|-------------------------------------------| -| Code | Manual reviews | Automated linting, type checking | -| Build | "Works on my machine" | Reproducible builds in clean environments | -| Test | Run manually, often skipped | Mandatory, automated on every commit | -| Release | Manual, error-prone | Automated versioning and changelogs | - ---- - -## 1.2 Git Mastery for CI - -### Essential Git Configuration - -```bash -# Identity -git config --global user.name "Your Name" -git config --global user.email "you@example.com" - -# Default branch -git config --global init.defaultBranch main - -# Useful aliases -git config --global alias.st "status --short" -git config --global alias.lg "log --oneline --graph --decorate --all" -git config --global alias.recent "branch --sort=-committerdate" -``` - -### Branching Strategies Compared - -#### Git Flow — scheduled releases - -```text -main ──────────────────────────────────────────────► (production tags) - └── develop ─────────────────────────────────────► (integration) - ├── feature/login ──────────► merge to develop - ├── feature/checkout ───────► merge to develop - └── release/1.2.0 ──────────► merge to main + develop - └── hotfix/critical ──► merge to main + develop -``` - -#### GitHub Flow — continuous web delivery - -```text -main ─────────────────────────────────────────────► (always deployable) - ├── feature/new-api ──► PR ──► merge ──► deploy - └── fix/auth-bug ─────► PR ──► merge ──► deploy -``` - -#### Trunk-Based Development — recommended for CI - -The model used by Google, Meta, and most high-performing engineering teams. Everyone integrates into `main` continuously, keeping the trunk always in a deployable state. - -```text - Day 1 Day 2 Day 3 - │ │ │ -main ───────┼─────────────────────┼─────────────────────┼──────► (always green) - │ ╭─ feat/login ─╮ │ ╭─ fix/auth ─╮ │ - │ │ (< 4 hours) │ │ │ (2 hours) │ │ - │ ╰──────────────╯ │ ╰────────────╯ │ - │ │ PR+merge │ │ PR+merge │ - ▼ ▼ ▼ ▼ ▼ - commit merge commit merge commit - │ │ - └── CI runs ✅ └── CI runs ✅ -``` - -**The core rules:** - -1. **Branches live less than a day.** If a branch lasts more than 24 hours, it's a signal that the task needs to be split into smaller pieces — not that the branch should keep growing. - -2. **Main is always deployable.** Every commit merged to `main` must pass CI. The pipeline is the gatekeeper — there is no "integration branch" to hide broken code. - -3. **Unfinished features ship behind feature flags.** Code can be merged before the feature is complete, as long as the flag keeps it invisible to users. This separates *deployment* (code goes to production) from *release* (users can see it). - -**Why it works better for CI than Git Flow or GitHub Flow:** - -| | Git Flow | GitHub Flow | Trunk-Based Dev | -|---------------------|-----------------|-------------|-----------------| -| Branch lifetime | Days–weeks | Hours–days | Hours (< 1 day) | -| Merge conflicts | Large, painful | Moderate | Minimal | -| CI feedback speed | Delayed | Fast | Immediate | -| Integration risk | High at release | Low | Very low | -| Feature flag needed | No | Rarely | Yes | - -**What a typical TBD day looks like:** - -```text -09:00 Developer picks up a ticket -09:15 git checkout -b feat/add-email-validation - (writes code + tests) -11:30 git push → opens PR -11:45 CI passes ✅, teammate reviews -12:00 Merged to main → CI runs again ✅ → auto-deployed to staging - (feature is behind a flag: email_validation_v2 = false) - -14:00 Developer picks up next ticket - ...same cycle repeats -``` - -**Feature flags in practice:** - -The flag lets you merge incomplete or experimental code safely. The feature is invisible in production until the team is confident enough to flip the switch — independently of any deployment. - -```typescript -// The feature ships in the codebase, but is off by default -const useNewEmailValidation = featureFlags.get('email_validation_v2', false); - -if (useNewEmailValidation) { - return validateWithZod(email); // New path — dark until flag is on -} else { - return legacyValidate(email); // Old path — still running -} -``` - -Flag lifecycle: - -```text -Merge (flag off) ──► QA testing (flag on for testers) ──► Gradual rollout ──► 100% ──► Remove flag - day 1 day 2–3 day 4–5 day 6 day 7+ -``` - -**When NOT to use TBD:** - -- Open source projects with external contributors who submit infrequent large PRs -- Teams without a fast CI pipeline (if CI takes 30+ minutes, short-lived branches become impractical) -- Projects with formal release gating (regulated industries, firmware) — Git Flow is more appropriate - -### Conventional Commits - -A machine-readable commit format that enables automated changelogs and semantic versioning. - -```text -[optional scope]: - -[optional body] - -[optional footer(s)] -``` - -| Type | Meaning | Version Bump | -|------------|-------------------------|---------------| -| `feat` | New feature | Minor (1.x.0) | -| `fix` | Bug fix | Patch (1.0.x) | -| `feat!` | Breaking change | Major (x.0.0) | -| `docs` | Documentation only | None | -| `chore` | Maintenance tasks | None | -| `test` | Adding or fixing tests | None | -| `ci` | CI config changes | None | -| `perf` | Performance improvement | Patch | -| `refactor` | Code restructure | None | - -**Examples:** - -```bash -git commit -m "feat(auth): add OAuth2 login with Google" -git commit -m "fix(api): handle null response from upstream service" -git commit -m "feat!: remove deprecated /v1 endpoints - -BREAKING CHANGE: /v1/users and /v1/orders removed. Use /v2 equivalents." -git commit -m "chore(deps): bump axios from 1.3.0 to 1.6.0" -``` - -### .gitignore for CI Workflows - -```gitignore -# Dependencies -node_modules/ -vendor/ -__pycache__/ -*.pyc -.venv/ - -# Build outputs -dist/ -build/ -*.egg-info/ -target/ - -# Secrets — NEVER commit these -.env -.env.local -.env.*.local -*.pem -*.key -secrets.yaml - -# CI artifacts -coverage/ -.nyc_output/ -junit.xml -test-results/ -*.log - -# IDE -.idea/ -.vscode/ -*.swp - -# OS -.DS_Store -Thumbs.db -``` - -### Branch Protection Rules - -Branch protection rules are the enforcement layer that makes CI meaningful. Without them, a developer can merge a PR even if every check is failing — the pipeline becomes advisory, not mandatory. Rules are configured per branch, typically targeting `main` (and sometimes `develop` or `release/*`). - -**What happens without protection vs with it:** - -```text -Without rules: With rules: -┌─────────────────────────────┐ ┌─────────────────────────────┐ -│ CI: ❌ Tests failing │ │ CI: ❌ Tests failing │ -│ CI: ❌ Lint failing │ │ CI: ❌ Lint failing │ -│ │ │ │ -│ [ Merge ] ← still clickable│ │ [ Merge ] ← BLOCKED 🔒 │ -└─────────────────────────────┘ └─────────────────────────────┘ - Developer can ignore CI CI is a hard gate -``` - -#### Setting Up on GitHub (UI path) - -```text -Repository → Settings → Branches → Add branch protection rule - Pattern: main (supports wildcards: release/*, v*.*) -``` - -The key options and what each one actually does: - -```text -☑ Require a pull request before merging - └─ Prevents direct pushes to main — all changes must come through a PR. - Nobody, including repo admins, can bypass this without explicitly - overriding the rule. - - ☑ Require approvals: 1 - │ └─ At least one reviewer must explicitly approve before merging. - │ Prevents rubber-stamp self-merges on solo projects or - │ "I'll review it after" patterns. - │ - └─ ☑ Dismiss stale pull request approvals when new commits are pushed - └─ If someone approves a PR and the author then pushes new - commits, the approval is revoked. Forces re-review of changes - made after approval — common attack vector if left unchecked. - -☑ Require status checks to pass before merging - │ └─ The actual CI gate. Lists which jobs MUST be green. - │ The job names here must exactly match the `name:` field in your - │ workflow (GitHub Actions) or the job key (GitLab CI). - │ - ├─ "CI / build-and-test" ← matches `name: build-and-test` in workflow - ├─ "CI / lint" - └─ "CI / security-scan" - - ☑ Require branches to be up to date before merging - └─ The PR's branch must include all commits currently on main. - Without this, two PRs could both pass CI individually but - conflict when merged together ("works in isolation" problem). - Forces the author to rebase/merge main before merging their PR. - -☑ Require conversation resolution before merging - └─ All review comments must be marked as resolved. Prevents merging - with open questions or unaddressed feedback. - -☑ Do not allow bypassing the above settings - └─ Applies rules even to repository administrators and owners. - Without this, admins can force-push or merge bypassing all checks. - Essential for compliance and auditability. -``` - -#### How Status Check Names Are Determined - -The name you register in the branch protection rule must match **exactly** what the CI system reports. Here is how to find the correct name for each platform: - -**GitHub Actions** — the check name is `{workflow name} / {job id}`: - -```yaml -# .github/workflows/ci.yml -name: CI # ← workflow name - -jobs: - build-and-test: # ← job id - ... - lint: - ... - -# Results in checks named: -# CI / build-and-test -# CI / lint -``` - -**GitLab CI** — the check name is the job key: - -```yaml -# .gitlab-ci.yml -unit-tests: # ← this is the status check name reported to GitHub - script: npm test - -lint: # ← and this one - script: npm run lint -``` - -#### ⚠️ Checks Only Appear After the First Pipeline Run - -This is one of the most common points of confusion when setting up branch protection for the first time. GitHub does **not** know which checks exist until the pipeline has actually run at least once against your repository. Until then, the search box under "Status checks that are required" returns no results — not because the configuration is wrong, but because GitHub has never seen those check names before. - -```text -First time setup order: - - Step 1 — Push your CI workflow file (.github/workflows/ci.yml) - ↓ - Step 2 — Let it run once (on any branch or a test PR) - ↓ - Step 3 — Go to Settings → Branches → Add rule → main - Type the check name in the search box - ↓ now it appears as an autocomplete suggestion - Step 4 — Select and save -``` - -**What the UI looks like before vs after the first run:** - -```text -Before first run: - Status checks that are required - ┌─────────────────────────────────────────┐ - │ Search for status checks... │ - └─────────────────────────────────────────┘ - No results found for "CI / lint" ← frustrating, but expected - -After first run: - Status checks that are required - ┌─────────────────────────────────────────┐ - │ CI / │ - └─────────────────────────────────────────┘ - ✓ CI / build-and-test ← now selectable - ✓ CI / lint - ✓ CI / security-scan -``` - -**Practical bootstrap sequence for a new repository:** - -```bash -# 1. Add your workflow file and push to a feature branch -git checkout -b ci/setup -mkdir -p .github/workflows -# ... create ci.yml ... -git add .github/workflows/ci.yml -git commit -m "ci: add initial CI pipeline" -git push origin ci/setup - -# 2. Open a PR — this triggers the pipeline for the first time -# Wait for all jobs to complete (green or red doesn't matter, -# GitHub just needs to register the check names) - -# 3. Now go to Settings → Branches → Add rule -# The check names will appear in the autocomplete search - -# 4. Merge this PR — branch protection is now active for all future PRs -``` - -> **Tip:** If you rename a job in your workflow YAML later (e.g., `lint` → `lint-and-format`), GitHub will stop receiving reports for the old name. The branch protection rule will reference a check that no longer exists, blocking all PRs. Always update the required checks in Settings immediately after renaming a job, or run the pipeline once so the new name appears in the search before saving the rule. - -**What happens when a required check never runs:** - -```text -Scenario: You added "CI / lint" as required, but the workflow only - triggers on pull_request, and someone pushes directly to - a non-protected branch. - -Status shown on PR: "Expected — Waiting for status to be reported" -Result: PR is permanently blocked — the check never fires. - -This is intentional and safe. It forces you to ensure your workflow -trigger covers the protected branch context. Common fix: - - on: - push: - branches: [main] - pull_request: ← this is the trigger that populates the check - branches: [main] ← on PRs targeting main -``` - -#### Full Recommended Configuration - -```yaml -# GitHub: Settings → Branches → Add rule -# Pattern: main - -# --- Pull Request Requirements --- -require_pull_request: true -required_approving_review_count: 1 # Increase to 2 for critical repos -dismiss_stale_reviews: true # Re-review required after new commits -require_review_from_codeowners: true # CODEOWNERS file defines who reviews what - -# --- CI Status Checks (must match exact job names) --- -require_status_checks: true -require_branches_up_to_date: true # Branch must be current with main -required_checks: - - "CI / lint" - - "CI / typecheck" - - "CI / unit-tests" - - "CI / integration-tests" - - "CI / security-scan" - -# --- Commit Requirements --- -require_signed_commits: true # GPG-signed commits (audit trail) -require_linear_history: true # Disables merge commits, enforces rebase - # Keeps git log clean and bisectable - -# --- Push Restrictions --- -restrict_pushes: true # Only allow pushes via PRs -allow_force_pushes: false # Prevents history rewriting on main -allow_deletions: false # Prevents accidental branch deletion - -# --- Admin Override --- -bypass_pull_request_allowances: [] # Nobody bypasses — not even admins -``` - -#### CODEOWNERS — Automatic Review Assignment - -Pair branch protection with a `CODEOWNERS` file to automatically assign reviewers based on which files changed: - -```bash -# .github/CODEOWNERS (or CODEOWNERS at repo root) - -# Default owner for everything -* @org/backend-team - -# Frontend — assigned to frontend team automatically -/frontend/ @org/frontend-team -*.tsx @org/frontend-team -*.css @org/frontend-team - -# Infrastructure changes require DevOps review -/infrastructure/ @org/devops-team -Dockerfile @org/devops-team -docker-compose*.yml @org/devops-team - -# CI config requires a senior engineer -.github/workflows/ @org/senior-engineers -.gitlab-ci.yml @org/senior-engineers - -# Secrets and sensitive config require security team -*secret* @org/security-team -*credential* @org/security-team -``` - -When a PR touches `/infrastructure/Dockerfile`, GitHub will automatically request a review from `@org/devops-team` and block merging until they approve. - -#### GitLab Equivalent — Protected Branches - -```yaml -# GitLab: Settings → Repository → Protected branches - -Branch: main -Allowed to merge: Developers + Maintainers -Allowed to push: No one # Forces all changes through MRs -Allowed to force push: false - -# Settings → Merge requests -Pipelines must succeed: true # CI gate -All discussions must be resolved: true -``` - ---- - -## 1.3 Core CI Concepts - -### CI vs Continuous Delivery vs Continuous Deployment - -```text -────────────────────────────────────────────────────────────── - CONTINUOUS INTEGRATION - [Code] ──► [Build] ──► [Test] - -────────────────────────────────────────────────────────────── - + CONTINUOUS DELIVERY (human approves deploy) - [Code] ──► [Build] ──► [Test] ──► [Staging] ──► [👤 Approve] ──► [Prod] - -────────────────────────────────────────────────────────────── - + CONTINUOUS DEPLOYMENT (fully automated) - [Code] ──► [Build] ──► [Test] ──► [Staging] ──► [Auto ✅] ──► [Prod] -────────────────────────────────────────────────────────────── -``` - -### The 8 Principles of CI - -1. **Single source repository** — one canonical repo -2. **Automate the build** — `make build` or equivalent, no manual steps -3. **Self-testing build** — failing tests = failing build -4. **Commit to mainline daily** — small, frequent integration reduces risk -5. **Every commit triggers CI** — the server watches every push -6. **Fix broken builds immediately** — broken CI blocks the whole team -7. **Keep builds fast** — target under 10 minutes -8. **Test in a production-like environment** — use containers for parity - ---- - -# Level 2 — Core CI Skills - -## 2.1 CI Platform Comparison - -| Platform | Config File | Free Tier (cloud) | Self-hostable | Best For | -|----------|-------------|-------------------|---------------|----------| -| **GitHub Actions** | `.github/workflows/*.yml` | 2,000 min/mo (public repos: unlimited) | ✅ | GitHub projects | -| **GitLab CI/CD** | `.gitlab-ci.yml` | 400 min/mo | ✅ | Full DevOps suite | -| **Jenkins** | `Jenkinsfile` | ❌ (no cloud offering) | ✅ free | Enterprise customization | -| **CircleCI** | `.circleci/config.yml` | 6,000 min/mo | ✅ | Docker-native speed | -| **Bitbucket Pipelines** | `bitbucket-pipelines.yml` | 50 min/mo | ❌ | Atlassian ecosystem | -| **Azure DevOps** | `azure-pipelines.yml` | 1,800 min/mo (public: unlimited) | ✅ | Microsoft/.NET shops | -| **Drone CI** | `.drone.yml` | ❌ (no cloud offering) | ✅ free | Lightweight, container-native | - -> **Recommendation:** Start with **GitHub Actions** or **GitLab CI** — best documentation, native integration, and large community. - -### Understanding the Free Tier - -"Free tier" in cloud CI platforms means **compute minutes per month** billed against a shared runner fleet. It is not unlimited free usage — it is a monthly quota that resets on a billing cycle. Understanding how minutes are counted and when you will exhaust them avoids surprise bills. - -#### How Minutes Are Counted - -A "minute" in CI is a **runner-minute**: one minute of wall-clock time on one runner. Parallel jobs each consume their own minutes simultaneously. - -```text -Example pipeline with 3 parallel jobs, each taking 5 minutes: - - lint ████████████ 5 min - test ████████████ 5 min ← all run at the same time - build ████████████ 5 min - - Wall-clock time elapsed: 5 minutes - Minutes billed: 15 minutes (3 jobs × 5 min) -``` - -Runner OS also affects billing on some platforms. GitHub Actions applies a multiplier to non-Linux runners: - -| OS | GitHub Actions multiplier | -|----|--------------------------| -| Linux (ubuntu-*) | 1× (base rate) | -| Windows (windows-*) | 2× | -| macOS (macos-*) | 10× | - -A 10-minute macOS job costs 100 minutes of your free quota. This is relevant if you are building mobile apps or testing cross-platform CLI tools. - -#### Free Tier Realities Per Platform - -##### GitHub Actions — 2,000 min/mo on private repos - -The most important exception: **public repositories get unlimited free minutes** on GitHub-hosted runners. This makes GitHub Actions the default choice for open source projects. - -```text -Private repo: 2,000 min/mo on Linux (~33 hours of single-job pipelines) - 200 min/mo equivalent on macOS (10× multiplier) -Public repo: Unlimited on all OS -``` - -A team running 20 PRs/day with a 6-minute pipeline uses roughly: - -```text -20 PRs × 6 min × 22 working days = 2,640 min/mo -→ Exceeds the free tier on private repos -→ Fine on public repos -``` - -##### GitLab CI — 400 min/mo (shared runners) - -GitLab's free tier is significantly smaller. It suits individuals and very small teams but is quickly exhausted by active development workflows. The practical alternative: **register your own machine as a GitLab Runner** — this is free, has no minute limit, and is how most self-hosted GitLab setups operate. - -```text -400 min/mo ÷ 22 working days ≈ 18 min/day of shared runner time - -If your pipeline takes 8 min and you push 3 times a day: - 3 × 8 min = 24 min/day → quota runs out mid-month -``` - -##### CircleCI — 6,000 min/mo - -The largest free cloud quota, but limited to 1 concurrent job on free plans. Parallelism (multiple jobs running at the same time) requires a paid plan. - -```text -Free plan: 6,000 min/mo, 1 concurrent job - Jobs queue rather than run in parallel - → a pipeline with 4 jobs runs sequentially, not in parallel -``` - -##### Bitbucket Pipelines — 50 min/mo - -Effectively a trial tier. 50 minutes is consumed by a single working day of normal development. Only viable for very low-frequency pipelines or as a trigger for external systems. - -##### Jenkins and Drone CI — no cloud offering - -These platforms have no managed cloud runner. "Free" means the software licence is free, but **you provide and pay for the compute yourself** (a VM, a bare-metal server, or a container on your own infrastructure). There is no minute quota, but there are real costs: hardware, electricity, maintenance, and runner uptime. - -#### When to Move Off the Free Tier - -Signs you have outgrown a cloud free tier: - -```text -1. Pipelines are queuing — jobs wait minutes before a runner picks them up -2. You are rationing pushes to save minutes -3. You are skipping CI runs to preserve quota -4. Your monthly bill spikes unexpectedly at day 18 - -Solutions, in order of preference: - a) Self-host one or more runners → unlimited minutes on your own hardware - b) Optimize pipelines to reduce per-run minute consumption - c) Upgrade to a paid plan for the parallelism and quota you actually need -``` - -#### Self-Hosted Runners: Zero Minute Cost - -Every platform that supports self-hosted runners exempts them from minute quotas entirely. Jobs on your own machine do not consume any cloud minutes regardless of how long they run. - -```text -GitHub Actions self-hosted runner: - runs-on: self-hosted ← this job costs $0 in GitHub minutes - runs-on: [self-hosted, linux, gpu] ← same, with label targeting -``` - -This is the standard solution for teams with GPU workloads, private network access, or simply high pipeline volume. See [Section 4.2 — Self-Hosted Runners](#42-self-hosted-runners) for setup details. - ---- - -## 2.2 Anatomy of a Pipeline - -### Pipeline Lifecycle - -```text -┌──────────┐ -│ TRIGGER │ push / pull_request / schedule / manual / webhook -└────┬─────┘ - │ -┌────▼─────┐ -│ CHECKOUT │ clone repository at the specific commit SHA -└────┬─────┘ - │ -┌────▼─────┐ -│ RESTORE │ restore dependency cache (npm, pip, cargo, maven…) -│ CACHE │ -└────┬─────┘ - │ -┌────▼─────────────────────────────────┐ -│ PARALLEL JOBS │ -│ ┌─────────┐ ┌──────┐ ┌────────┐ │ -│ │ LINT │ │BUILD │ │ TEST │ │ -│ └─────────┘ └──────┘ └────────┘ │ -└────┬─────────────────────────────────┘ - │ -┌────▼─────┐ -│ REPORT │ upload coverage, test results, build logs -└────┬─────┘ - │ -┌────▼─────┐ -│ NOTIFY │ Slack, email, PR comment, commit status -└──────────┘ -``` - -### Key Concepts Glossary - -| Term | Definition | -|------------------|----------------------------------------------------------| -| **Trigger** | Event that starts the pipeline (push, PR, cron, webhook) | -| **Runner/Agent** | The machine where jobs execute | -| **Job** | An isolated unit of work with its own runner | -| **Step** | A single command or action within a job | -| **Stage** | Logical group of jobs (test, build, deploy) | -| **Artifact** | File(s) produced by a job, passed to downstream jobs | -| **Cache** | Persisted files reused across pipeline runs | -| **Environment** | Named deployment target (dev, staging, prod) | -| **Secret** | Encrypted value injected as an environment variable | - ---- - -## 2.3 Writing Pipelines - -### GitHub Actions — Complete Annotated Example - -```yaml -# .github/workflows/ci.yml - -name: CI # Displayed in the GitHub Actions UI - -on: # Triggers - push: - branches: [main, develop] - pull_request: - branches: [main] - workflow_dispatch: # Allow manual trigger from UI - -env: # Global environment variables - NODE_VERSION: '20' - REGISTRY: ghcr.io - -jobs: - # ────────────────────────────────────────── - # JOB 1: Lint - # ────────────────────────────────────────── - lint: - name: Lint & Format Check - runs-on: ubuntu-latest - steps: - - name: Checkout code - uses: actions/checkout@v4 - - - name: Setup Node.js - uses: actions/setup-node@v4 - with: - node-version: ${{ env.NODE_VERSION }} - cache: 'npm' # Automatically caches node_modules - - - name: Install dependencies - run: npm ci # Faster than npm install, uses lock file - - - name: Run ESLint - run: npm run lint - - - name: Check formatting - run: npm run format:check - - # ────────────────────────────────────────── - # JOB 2: Test (depends on nothing, runs in parallel with lint) - # ────────────────────────────────────────── - test: - name: Unit & Integration Tests - runs-on: ubuntu-latest - - services: # Spin up a real Postgres for integration tests - postgres: - image: postgres:16 - env: - POSTGRES_USER: testuser - POSTGRES_PASSWORD: testpass - POSTGRES_DB: testdb - ports: - - 5432:5432 - options: >- - --health-cmd pg_isready - --health-interval 10s - --health-timeout 5s - --health-retries 5 - - steps: - - uses: actions/checkout@v4 - - - uses: actions/setup-node@v4 - with: - node-version: ${{ env.NODE_VERSION }} - cache: 'npm' - - - run: npm ci - - - name: Run tests with coverage - run: npm test -- --coverage - env: - DATABASE_URL: postgresql://testuser:testpass@localhost:5432/testdb - NODE_ENV: test - - - name: Upload coverage report - uses: actions/upload-artifact@v4 - with: - name: coverage-report - path: coverage/ - retention-days: 7 - - - name: Comment coverage on PR - if: github.event_name == 'pull_request' - uses: davelosert/vitest-coverage-report-action@v2 - - # ────────────────────────────────────────── - # JOB 3: Build (runs only after lint + test pass) - # ────────────────────────────────────────── - build: - name: Build - runs-on: ubuntu-latest - needs: [lint, test] # Waits for both jobs to succeed - - steps: - - uses: actions/checkout@v4 - - - uses: actions/setup-node@v4 - with: - node-version: ${{ env.NODE_VERSION }} - cache: 'npm' - - - run: npm ci - - - name: Build application - run: npm run build - env: - NODE_ENV: production - - - name: Upload build artifact - uses: actions/upload-artifact@v4 - with: - name: build-output - path: dist/ - retention-days: 14 - - # ────────────────────────────────────────── - # JOB 4: Notify (always runs, reports final status) - # ────────────────────────────────────────── - notify: - name: Notify - runs-on: ubuntu-latest - needs: [build] - if: always() # Run even if previous jobs failed - - steps: - - name: Notify Slack - uses: slackapi/slack-github-action@v1.26.0 - with: - payload: | - { - "text": "CI ${{ needs.build.result == 'success' && '✅ Passed' || '❌ Failed' }}: ${{ github.repository }}@${{ github.ref_name }}" - } - env: - SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} -``` - ---- - -### GitLab CI — Complete Annotated Example - -```yaml -# .gitlab-ci.yml - -# ─── Global Configuration ─────────────────────────────────────────── -default: - image: node:20-alpine - before_script: - - npm ci --cache .npm --prefer-offline - cache: - key: - files: - - package-lock.json # Cache invalidates when lock file changes - paths: - - .npm/ - - node_modules/ - -variables: - NODE_ENV: test - POSTGRES_DB: testdb - POSTGRES_USER: testuser - POSTGRES_PASSWORD: testpass - -stages: - - validate - - test - - build - - publish - -# ─── Stage: validate ──────────────────────────────────────────────── -lint: - stage: validate - script: - - npm run lint - - npm run format:check - rules: - - if: $CI_PIPELINE_SOURCE == "merge_request_event" - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - -typecheck: - stage: validate - script: - - npm run typecheck - -# ─── Stage: test ──────────────────────────────────────────────────── -unit-tests: - stage: test - script: - - npm run test:unit -- --coverage - coverage: '/Lines\s*:\s*(\d+\.?\d*)%/' # Extract coverage % for GitLab badge - artifacts: - when: always - reports: - junit: junit.xml # GitLab parses this for test result UI - coverage_report: - coverage_format: cobertura - path: coverage/cobertura-coverage.xml - paths: - - coverage/ - expire_in: 1 week - -integration-tests: - stage: test - services: - - postgres:16 - script: - - npm run test:integration - variables: - DATABASE_URL: "postgresql://testuser:testpass@postgres:5432/testdb" - -# ─── Stage: build ─────────────────────────────────────────────────── -build: - stage: build - script: - - NODE_ENV=production npm run build - artifacts: - paths: - - dist/ - expire_in: 2 weeks - only: - - main - - tags - -# ─── Stage: publish ───────────────────────────────────────────────── -publish-image: - stage: publish - image: docker:24 - services: - - docker:24-dind - before_script: - - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY - script: - - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA . - - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest - - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA - - docker push $CI_REGISTRY_IMAGE:latest - only: - - main -``` - ---- - -### Jenkins — Declarative Pipeline Example - -```groovy -// Jenkinsfile - -pipeline { - agent any - - environment { - NODE_VERSION = '20' - REGISTRY = 'registry.example.com' - } - - tools { - nodejs "${NODE_VERSION}" - } - - options { - timeout(time: 30, unit: 'MINUTES') - disableConcurrentBuilds() // Prevent parallel runs on same branch - buildDiscarder(logRotator(numToKeepStr: '10')) - } - - stages { - stage('Checkout') { - steps { - checkout scm - } - } - - stage('Install') { - steps { - sh 'npm ci' - } - } - - stage('Validate') { - parallel { - stage('Lint') { - steps { sh 'npm run lint' } - } - stage('Type Check') { - steps { sh 'npm run typecheck' } - } - } - } - - stage('Test') { - steps { - sh 'npm test -- --coverage' - } - post { - always { - junit 'junit.xml' - publishHTML([ - reportDir: 'coverage', - reportFiles: 'index.html', - reportName: 'Coverage Report' - ]) - } - } - } - - stage('Build') { - when { - anyOf { - branch 'main' - tag pattern: 'v\\d+\\.\\d+\\.\\d+', comparator: 'REGEXP' - } - } - steps { - sh 'NODE_ENV=production npm run build' - } - } - } - - post { - success { - slackSend(color: 'good', message: "✅ Build passed: ${env.JOB_NAME} #${env.BUILD_NUMBER}") - } - failure { - slackSend(color: 'danger', message: "❌ Build failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}\n${env.BUILD_URL}") - } - } -} -``` - ---- - -## 2.4 Automated Testing - -### The Test Pyramid - -```text - ┌───────────┐ - /│ E2E │\ Few, slow, expensive - / └───────────┘ \ Run: nightly or on release - /─────────────────\ - / │ Integration │ \ Moderate number - / └───────────────┘ \ Run: every PR - /───────────────────────\ - / │ Unit Tests │ \ Many, fast, cheap - / └─────────────────┘ \ Run: every commit - /─────────────────────────────\ -``` - -**Guideline:** 70% unit / 20% integration / 10% E2E - -### Unit Tests — Node.js with Vitest - -```typescript -// src/utils/validate.ts -export function validateEmail(email: string): boolean { - return /^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(email); -} - -export function validateAge(age: number): boolean { - return Number.isInteger(age) && age >= 0 && age <= 150; -} -``` - -```typescript -// src/utils/validate.test.ts -import { describe, it, expect } from 'vitest'; -import { validateEmail, validateAge } from './validate'; - -describe('validateEmail', () => { - it('accepts a valid email', () => { - expect(validateEmail('user@example.com')).toBe(true); - }); - - it('rejects email without @', () => { - expect(validateEmail('userexample.com')).toBe(false); - }); - - it('rejects empty string', () => { - expect(validateEmail('')).toBe(false); - }); -}); - -describe('validateAge', () => { - it('accepts age 0', () => expect(validateAge(0)).toBe(true)); - it('accepts age 25', () => expect(validateAge(25)).toBe(true)); - it('rejects negative age', () => expect(validateAge(-1)).toBe(false)); - it('rejects float', () => expect(validateAge(25.5)).toBe(false)); -}); -``` - -### Unit Tests — Python with pytest - -```python -# src/utils/validate.py -import re - -def validate_email(email: str) -> bool: - pattern = r'^[^\s@]+@[^\s@]+\.[^\s@]+$' - return bool(re.match(pattern, email)) - -def validate_age(age: int) -> bool: - return isinstance(age, int) and 0 <= age <= 150 -``` - -```python -# tests/test_validate.py -import pytest -from src.utils.validate import validate_email, validate_age - -class TestValidateEmail: - def test_valid_email(self): - assert validate_email("user@example.com") is True - - def test_missing_at(self): - assert validate_email("userexample.com") is False - - def test_empty_string(self): - assert validate_email("") is False - - @pytest.mark.parametrize("email", [ - "a@b.co", - "user+tag@domain.org", - "first.last@sub.domain.com", - ]) - def test_valid_emails(self, email): - assert validate_email(email) is True - - -class TestValidateAge: - def test_zero(self): - assert validate_age(0) is True - - def test_normal_age(self): - assert validate_age(25) is True - - def test_negative(self): - assert validate_age(-1) is False - - def test_float_rejected(self): - assert validate_age(25.5) is False # type: ignore -``` - -### Integration Test — API with Supertest - -```typescript -// tests/api/users.test.ts -import { describe, it, expect, beforeAll, afterAll } from 'vitest'; -import request from 'supertest'; -import { app } from '../../src/app'; -import { db } from '../../src/db'; - -beforeAll(async () => { - await db.migrate.latest(); - await db.seed.run(); -}); - -afterAll(async () => { - await db.destroy(); -}); - -describe('POST /api/users', () => { - it('creates a user with valid data', async () => { - const res = await request(app) - .post('/api/users') - .send({ name: 'Alice', email: 'alice@example.com' }) - .expect(201); - - expect(res.body).toMatchObject({ - id: expect.any(Number), - name: 'Alice', - email: 'alice@example.com', - }); - }); - - it('rejects invalid email', async () => { - await request(app) - .post('/api/users') - .send({ name: 'Bob', email: 'not-an-email' }) - .expect(422); - }); -}); -``` - -### E2E Test — Playwright - -```typescript -// e2e/login.spec.ts -import { test, expect } from '@playwright/test'; - -test.describe('Login flow', () => { - test('user can log in with valid credentials', async ({ page }) => { - await page.goto('/login'); - - await page.fill('[data-testid="email"]', 'user@example.com'); - await page.fill('[data-testid="password"]', 'password123'); - await page.click('[data-testid="submit"]'); - - await expect(page).toHaveURL('/dashboard'); - await expect(page.locator('h1')).toContainText('Welcome'); - }); - - test('shows error on invalid credentials', async ({ page }) => { - await page.goto('/login'); - - await page.fill('[data-testid="email"]', 'bad@example.com'); - await page.fill('[data-testid="password"]', 'wrong'); - await page.click('[data-testid="submit"]'); - - await expect(page.locator('[data-testid="error"]')).toBeVisible(); - await expect(page.locator('[data-testid="error"]')).toContainText('Invalid credentials'); - }); -}); -``` - -#### playwright.config.ts — CI-optimized settings - -```typescript -import { defineConfig } from '@playwright/test'; - -export default defineConfig({ - testDir: './e2e', - fullyParallel: true, - retries: process.env.CI ? 2 : 0, // Retry flaky tests in CI only - workers: process.env.CI ? 4 : undefined, - reporter: process.env.CI - ? [['junit', { outputFile: 'e2e-results.xml' }], ['html']] - : 'list', - use: { - baseURL: process.env.BASE_URL || 'http://localhost:3000', - screenshot: 'only-on-failure', - video: 'retain-on-failure', - }, -}); -``` - -### Coverage Enforcement in CI - -```yaml -# GitHub Actions — fail if coverage drops below 80% -- name: Run tests with coverage - run: | - npm test -- --coverage --coverageThreshold='{"global":{"lines":80,"functions":80,"branches":75}}' -``` - -```ini -# pytest.ini — enforce coverage threshold -[tool:pytest] -addopts = --cov=src --cov-report=xml --cov-report=term-missing --cov-fail-under=80 -``` - ---- - -## 2.5 Build Automation - -### Makefile — Universal Build Interface - -```makefile -# Makefile -.PHONY: install lint test build clean docker-build - -install: - npm ci - -lint: - npm run lint - npm run format:check - -test: - npm test -- --coverage - -build: - NODE_ENV=production npm run build - -clean: - rm -rf dist/ coverage/ node_modules/ - -docker-build: - docker build -t myapp:$(shell git rev-parse --short HEAD) . - -# CI-specific target — run everything in order -ci: install lint test build -``` - -### package.json Scripts - -```json -{ - "scripts": { - "dev": "vite", - "build": "tsc && vite build", - "preview": "vite preview", - "lint": "eslint src --ext .ts,.tsx --max-warnings 0", - "format": "prettier --write .", - "format:check": "prettier --check .", - "typecheck": "tsc --noEmit", - "test": "vitest run", - "test:watch": "vitest", - "test:ui": "vitest --ui", - "test:e2e": "playwright test", - "ci": "npm run lint && npm run typecheck && npm run test && npm run build" - } -} -``` - -### Dependency Locking Best Practices - -```bash -# Always commit lock files — they ensure reproducible builds -git add package-lock.json # Node.js -git add poetry.lock # Python -git add Cargo.lock # Rust -git add go.sum # Go - -# Use exact install commands in CI (not `npm install`) -npm ci # Node — uses lock file, fails if mismatched -pip install -r requirements.txt # Python -cargo build # Rust — uses Cargo.lock automatically -``` - ---- - -# Level 3 — Intermediate CI - -## 3.1 Docker in CI - -### Multi-Stage Dockerfile - -A well-structured Dockerfile that keeps the production image small and the build environment separate: - -```dockerfile -# ─── Stage 1: Dependencies ─────────────────────────────────────────── -FROM node:20-alpine AS deps -WORKDIR /app -COPY package*.json ./ -RUN npm ci --only=production && cp -r node_modules /tmp/prod-deps -RUN npm ci # Install all deps for build - -# ─── Stage 2: Build ────────────────────────────────────────────────── -FROM node:20-alpine AS builder -WORKDIR /app -COPY --from=deps /app/node_modules ./node_modules -COPY . . -RUN npm run build - -# ─── Stage 3: Production Image ─────────────────────────────────────── -FROM node:20-alpine AS runner -WORKDIR /app - -ENV NODE_ENV=production - -# Run as non-root user (security best practice) -RUN addgroup --system --gid 1001 nodejs -RUN adduser --system --uid 1001 appuser - -COPY --from=deps /tmp/prod-deps ./node_modules -COPY --from=builder /app/dist ./dist -COPY --from=builder /app/package.json ./ - -USER appuser - -EXPOSE 3000 -CMD ["node", "dist/index.js"] -``` - -### Building and Pushing in GitHub Actions - -```yaml -# .github/workflows/docker.yml - -name: Build & Push Docker Image - -on: - push: - branches: [main] - tags: ['v*'] - -jobs: - docker: - runs-on: ubuntu-latest - permissions: - contents: read - packages: write # Required to push to GHCR - - steps: - - uses: actions/checkout@v4 - - # Enable Docker layer caching via GitHub cache - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 - - - name: Log in to GitHub Container Registry - uses: docker/login-action@v3 - with: - registry: ghcr.io - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - - # Generate image tags based on git context - - name: Extract metadata - id: meta - uses: docker/metadata-action@v5 - with: - images: ghcr.io/${{ github.repository }} - tags: | - type=ref,event=branch - type=semver,pattern={{version}} - type=semver,pattern={{major}}.{{minor}} - type=sha,prefix=sha- - - - name: Build and push - uses: docker/build-push-action@v5 - with: - context: . - push: true - tags: ${{ steps.meta.outputs.tags }} - labels: ${{ steps.meta.outputs.labels }} - cache-from: type=gha # Use GitHub Actions cache for layers - cache-to: type=gha,mode=max -``` - -### Docker Layer Caching Strategy - -```dockerfile -# ✅ GOOD: Copy package files BEFORE source code -# This way, npm install is only re-run when package.json changes -COPY package*.json ./ - -# Cached unless package*.json changed -RUN npm ci -# Source changes don't invalidate the npm layer -COPY src/ ./src/ -RUN npm run build - -# ❌ BAD: Copy everything at once — any source change busts the cache -COPY . . -RUN npm ci -RUN npm run build -``` - ---- - -## 3.2 Code Quality Gates - -### ESLint Configuration - -#### .eslintrc.json - -```json -{ - "extends": [ - "eslint:recommended", - "plugin:@typescript-eslint/recommended", - "plugin:import/recommended" - ], - "rules": { - "no-console": "warn", - "no-unused-vars": "off", - "@typescript-eslint/no-unused-vars": "error", - "@typescript-eslint/no-explicit-any": "warn", - "import/order": ["error", { "alphabetize": { "order": "asc" } }] - } -} -``` - -### Prettier Configuration - -#### .prettierrc - -```json -{ - "semi": true, - "singleQuote": true, - "tabWidth": 2, - "trailingComma": "es5", - "printWidth": 100, - "arrowParens": "avoid" -} -``` - -### Pre-commit Hooks with Husky - -Run quality checks locally before committing — mirrors what CI will do: - -```bash -npm install --save-dev husky lint-staged -npx husky init -``` - -#### package.json - -```json -{ - "lint-staged": { - "*.{ts,tsx}": ["eslint --fix", "prettier --write"], - "*.{json,md,yml}": ["prettier --write"] - } -} -``` - -```bash -# .husky/pre-commit -#!/bin/sh -npx lint-staged -``` - -### SonarQube Integration in CI - -```yaml -# GitHub Actions SonarQube scan -- name: SonarQube Scan - uses: SonarSource/sonarqube-scan-action@master - env: - SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} - SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }} -``` - -```properties -# sonar-project.properties -sonar.projectKey=my-project -sonar.sources=src -sonar.tests=tests -sonar.javascript.lcov.reportPaths=coverage/lcov.info -sonar.testExecutionReportPaths=junit.xml - -# Quality gate thresholds -sonar.qualitygate.wait=true # Fail CI if quality gate fails -``` - ---- - -## 3.3 Security Scanning - -### Secret Detection — Preventing Leaks Before They Happen - -```yaml -# GitHub Actions — scan for secrets in every PR -- name: Scan for secrets (Gitleaks) - uses: gitleaks/gitleaks-action@v2 - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} -``` - -```toml -# .gitleaks.toml — custom rules -[allowlist] - description = "Allowlist for known safe patterns" - regexes = [ - '''EXAMPLE_KEY''', - '''test_.*_key''' - ] - paths = [ - '''tests/fixtures/.*''' - ] -``` - -### Dependency Vulnerability Scanning - -```yaml -# Scan Node.js dependencies -- name: Audit dependencies - run: npm audit --audit-level=high - # Fails if HIGH or CRITICAL vulnerabilities found - -# Or use Snyk for richer reports -- name: Snyk vulnerability scan - uses: snyk/actions/node@master - env: - SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }} - with: - args: --severity-threshold=high -``` - -```yaml -# Scan Python dependencies -- name: Safety check - run: | - pip install safety - safety check -r requirements.txt --json > safety-report.json -``` - -### Container Image Scanning with Trivy - -```yaml -- name: Scan Docker image for vulnerabilities - uses: aquasecurity/trivy-action@master - with: - image-ref: 'ghcr.io/${{ github.repository }}:${{ github.sha }}' - format: 'sarif' - output: 'trivy-results.sarif' - severity: 'CRITICAL,HIGH' - exit-code: '1' # Fail the pipeline if vulnerabilities found - -- name: Upload Trivy results to GitHub Security tab - uses: github/codeql-action/upload-sarif@v3 - with: - sarif_file: 'trivy-results.sarif' -``` - -### CodeQL Static Analysis - -```yaml -# .github/workflows/codeql.yml -name: CodeQL Analysis - -on: - push: - branches: [main] - pull_request: - branches: [main] - schedule: - - cron: '0 6 * * 1' # Weekly scan every Monday at 6am - -jobs: - analyze: - runs-on: ubuntu-latest - permissions: - security-events: write - actions: read - contents: read - - strategy: - matrix: - language: [javascript, python] - - steps: - - uses: actions/checkout@v4 - - - name: Initialize CodeQL - uses: github/codeql-action/init@v3 - with: - languages: ${{ matrix.language }} - - - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@v3 -``` - ---- - -## 3.4 Artifact Management - -### Semantic Versioning in CI - -```yaml -# .github/workflows/release.yml -name: Release - -on: - push: - branches: [main] - -jobs: - release: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 # Full history needed for changelog - - - name: Semantic Release - uses: cycjimmy/semantic-release-action@v4 - with: - semantic_version: 23 - extra_plugins: | - @semantic-release/changelog - @semantic-release/git - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - NPM_TOKEN: ${{ secrets.NPM_TOKEN }} -``` - -#### .releaserc.json — semantic-release config - -```json -{ - "branches": ["main"], - "plugins": [ - "@semantic-release/commit-analyzer", - "@semantic-release/release-notes-generator", - ["@semantic-release/changelog", { - "changelogFile": "CHANGELOG.md" - }], - ["@semantic-release/npm", { - "npmPublish": true - }], - ["@semantic-release/git", { - "assets": ["CHANGELOG.md", "package.json"], - "message": "chore(release): ${nextRelease.version} [skip ci]" - }], - "@semantic-release/github" - ] -} -``` - -### Publishing to PyPI from CI - -```yaml -- name: Build Python package - run: | - pip install build - python -m build - -- name: Publish to PyPI - uses: pypa/gh-action-pypi-publish@release/v1 - with: - password: ${{ secrets.PYPI_API_TOKEN }} - # Use TestPyPI first: repository-url: https://test.pypi.org/legacy/ -``` - ---- - -## 3.5 Notifications & Observability - -### Slack Notifications - -```yaml -- name: Notify Slack on failure - if: failure() - uses: slackapi/slack-github-action@v1.26.0 - with: - payload: | - { - "blocks": [ - { - "type": "section", - "text": { - "type": "mrkdwn", - "text": "❌ *CI Failed*: <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View Run>\n*Repo:* ${{ github.repository }}\n*Branch:* `${{ github.ref_name }}`\n*Commit:* `${{ github.sha }}`\n*Author:* ${{ github.actor }}" - } - } - ] - } - env: - SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} - SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK -``` - -### Build Status Badge - -```markdown - -![CI](https://github.com/owner/repo/actions/workflows/ci.yml/badge.svg) -![Coverage](https://codecov.io/gh/owner/repo/branch/main/graph/badge.svg) -![Security](https://snyk.io/test/github/owner/repo/badge.svg) -``` - ---- - -# Level 4 — Advanced CI - -## 4.1 Pipeline Optimization - -### Parallelization with Matrix Builds - -```yaml -# Test across multiple Node.js versions and operating systems -jobs: - test: - strategy: - fail-fast: false # Don't cancel other matrix jobs on failure - matrix: - os: [ubuntu-latest, windows-latest, macos-latest] - node: [18, 20, 22] - exclude: - - os: windows-latest - node: 18 # Skip this combination - - runs-on: ${{ matrix.os }} - name: Test (Node ${{ matrix.node }} / ${{ matrix.os }}) - - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: ${{ matrix.node }} - - run: npm ci - - run: npm test -``` - -### Conditional Execution — Path-Based Triggers - -```yaml -# Only run frontend tests when frontend code changes -jobs: - frontend-tests: - if: | - contains(github.event.head_commit.modified, 'frontend/') || - contains(github.event.head_commit.added, 'frontend/') - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - run: cd frontend && npm ci && npm test -``` - -```yaml -# GitLab: rules with changes filter -frontend-test: - script: cd frontend && npm test - rules: - - changes: - - frontend/**/* - - package.json - -backend-test: - script: cd backend && go test ./... - rules: - - changes: - - backend/**/* - - go.mod - - go.sum -``` - -### Dependency Graph — DAG Pipelines in GitLab - -```yaml -# Jobs run in parallel when they don't share a stage dependency -stages: [build, test, deploy] - -build-frontend: - stage: build - script: npm run build:frontend - artifacts: - paths: [dist/frontend] - -build-backend: - stage: build # Runs in PARALLEL with build-frontend - script: go build ./... - artifacts: - paths: [bin/] - -test-frontend: - stage: test - needs: [build-frontend] # Only waits for build-frontend, not build-backend - script: npm test - -test-backend: - stage: test - needs: [build-backend] # Only waits for build-backend - script: go test ./... - -deploy: - stage: deploy - needs: [test-frontend, test-backend] # Waits for BOTH tests - script: ./deploy.sh -``` - -### Advanced Caching Strategies - -```yaml -# GitHub Actions — cache npm with composite key -- name: Cache node modules - uses: actions/cache@v4 - with: - path: ~/.npm - key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }} - restore-keys: | - ${{ runner.os }}-node- # Partial match fallback - -# Cache pip packages -- name: Cache pip - uses: actions/cache@v4 - with: - path: ~/.cache/pip - key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }} - restore-keys: ${{ runner.os }}-pip- - -# Cache Rust/Cargo (more complex — cache both registry and target) -- name: Cache Cargo - uses: Swatinem/rust-cache@v2 - with: - workspaces: ". -> target" - cache-on-failure: true -``` - ---- - -## 4.2 Self-Hosted Runners - -### GitHub Actions Self-Hosted Runner Setup - -```bash -# On your server/VM — register a runner -mkdir actions-runner && cd actions-runner -curl -o actions-runner-linux-x64.tar.gz -L \ - https://github.com/actions/runner/releases/download/v2.317.0/actions-runner-linux-x64-2.317.0.tar.gz -tar xzf ./actions-runner-linux-x64.tar.gz - -# Register with your repo -./config.sh --url https://github.com/OWNER/REPO --token YOUR_TOKEN - -# Install and start as a service -sudo ./svc.sh install -sudo ./svc.sh start -``` - -```yaml -# Use your self-hosted runner in a workflow -jobs: - gpu-job: - runs-on: [self-hosted, linux, gpu] # Target by labels - steps: - - run: nvidia-smi # Only works on your GPU runner -``` - -### Ephemeral Runners with Docker - -```dockerfile -# Dockerfile.runner -FROM ubuntu:22.04 -RUN apt-get update && apt-get install -y \ - curl git jq libicu70 \ - && rm -rf /var/lib/apt/lists/* - -# Install GitHub Actions runner -ARG RUNNER_VERSION=2.317.0 -RUN curl -o /tmp/runner.tar.gz -L \ - https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz \ - && tar xzf /tmp/runner.tar.gz -C /opt/actions-runner - -COPY entrypoint.sh /entrypoint.sh -RUN chmod +x /entrypoint.sh - -ENTRYPOINT ["/entrypoint.sh"] -``` - -```bash -#!/bin/bash -# entrypoint.sh — register, run once, then deregister (ephemeral) -/opt/actions-runner/config.sh \ - --url "${GITHUB_URL}" \ - --token "${RUNNER_TOKEN}" \ - --name "ephemeral-$(hostname)" \ - --ephemeral \ - --unattended - -/opt/actions-runner/run.sh -``` - ---- - -## 4.3 Advanced Testing Strategies - -### Test Sharding — Split a Large Suite Across Runners - -```yaml -# GitHub Actions — shard Playwright E2E tests across 4 runners -jobs: - e2e: - strategy: - matrix: - shard: [1, 2, 3, 4] - - steps: - - uses: actions/checkout@v4 - - run: npm ci - - run: npx playwright test --shard=${{ matrix.shard }}/4 - - uses: actions/upload-artifact@v4 - with: - name: blob-report-${{ matrix.shard }} - path: blob-report/ - - # Merge shard reports into one HTML report - merge-reports: - needs: [e2e] - steps: - - uses: actions/download-artifact@v4 - with: - pattern: blob-report-* - merge-multiple: true - path: all-blob-reports - - run: npx playwright merge-reports --reporter html ./all-blob-reports -``` - -### Contract Testing with Pact - -```typescript -// consumer.pact.test.ts — define the contract from the consumer's perspective -import { PactV3, MatchersV3 } from '@pact-foundation/pact'; - -const provider = new PactV3({ - consumer: 'OrderService', - provider: 'UserService', -}); - -describe('UserService contract', () => { - it('returns a user by ID', () => { - return provider - .given('User 123 exists') - .uponReceiving('a request for user 123') - .withRequest({ method: 'GET', path: '/users/123' }) - .willRespondWith({ - status: 200, - body: { - id: MatchersV3.integer(123), - name: MatchersV3.string('Alice'), - email: MatchersV3.email('alice@example.com'), - }, - }) - .executeTest(async (mockServer) => { - const client = new UserClient(mockServer.url); - const user = await client.getUser(123); - expect(user.name).toBe('Alice'); - }); - }); -}); -``` - -### Performance Testing with k6 - -```javascript -// load-test.js -import http from 'k6/http'; -import { check, sleep } from 'k6'; -import { Rate } from 'k6/metrics'; - -const errorRate = new Rate('errors'); - -export const options = { - stages: [ - { duration: '30s', target: 20 }, // Ramp up to 20 users - { duration: '1m', target: 20 }, // Stay at 20 - { duration: '20s', target: 0 }, // Ramp down - ], - thresholds: { - http_req_duration: ['p(95)<500'], // 95% of requests under 500ms - errors: ['rate<0.01'], // Error rate under 1% - }, -}; - -export default function () { - const res = http.get(`${__ENV.BASE_URL}/api/users`); - - check(res, { - 'status is 200': (r) => r.status === 200, - 'response time < 500ms': (r) => r.timings.duration < 500, - }); - - errorRate.add(res.status !== 200); - sleep(1); -} -``` - -```yaml -# Run k6 in CI -- name: Run k6 load test - uses: grafana/k6-action@v0.3.1 - with: - filename: load-test.js - env: - BASE_URL: https://staging.example.com - K6_CLOUD_TOKEN: ${{ secrets.K6_CLOUD_TOKEN }} -``` - ---- - -## 4.4 GitOps & CI/CD Integration - -### CI → CD Trigger Pattern - -```yaml -# CI pipeline (build repo) — triggers CD on success -- name: Trigger deployment pipeline - if: github.ref == 'refs/heads/main' && success() - uses: peter-evans/repository-dispatch@v3 - with: - token: ${{ secrets.CD_DISPATCH_TOKEN }} - repository: org/cd-config-repo - event-type: deploy-staging - client-payload: | - { - "image": "ghcr.io/${{ github.repository }}", - "tag": "${{ github.sha }}", - "environment": "staging" - } -``` - -### Kubernetes Manifest Update (GitOps) - -```yaml -# In the CD repo — update image tag and commit (ArgoCD/Flux picks it up) -- name: Update image tag in Kubernetes manifest - run: | - IMAGE_TAG=${{ github.event.client_payload.tag }} - sed -i "s|image: .*|image: ghcr.io/org/app:${IMAGE_TAG}|" k8s/staging/deployment.yaml - git config user.name "ci-bot" - git config user.email "ci-bot@example.com" - git add k8s/staging/deployment.yaml - git commit -m "chore: deploy ${IMAGE_TAG} to staging [skip ci]" - git push -``` - -### Environment Promotion Pipeline - -```yaml -# GitLab CI — promote through environments -stages: [build, deploy-dev, deploy-staging, deploy-prod] - -.deploy-template: &deploy - image: bitnami/kubectl:latest - script: - - kubectl set image deployment/app app=$IMAGE:$CI_COMMIT_SHA -n $NAMESPACE - -deploy-dev: - <<: *deploy - stage: deploy-dev - variables: - NAMESPACE: dev - environment: - name: development - url: https://dev.example.com - only: [main] - -deploy-staging: - <<: *deploy - stage: deploy-staging - variables: - NAMESPACE: staging - environment: - name: staging - url: https://staging.example.com - when: manual # Require human approval - only: [main] - -deploy-prod: - <<: *deploy - stage: deploy-prod - variables: - NAMESPACE: production - environment: - name: production - url: https://example.com - when: manual - only: [main] - allow_failure: false -``` - ---- - -## 4.5 Feature Flags & Progressive Delivery - -### Feature Flag Pattern in Code - -```typescript -// Using OpenFeature SDK -import { OpenFeature } from '@openfeature/server-sdk'; - -const client = OpenFeature.getClient('my-app'); - -export async function handleCheckout(userId: string) { - const useNewCheckout = await client.getBooleanValue( - 'new-checkout-flow', - false, // Default: disabled - { targetingKey: userId } - ); - - if (useNewCheckout) { - return newCheckoutFlow(userId); - } else { - return legacyCheckoutFlow(userId); - } -} -``` - -### Canary Release with GitHub Actions + Kubernetes - -```yaml -- name: Deploy canary (10% traffic) - run: | - # Deploy new version as canary - kubectl apply -f k8s/canary/deployment.yaml - - # Set traffic split: 90% stable, 10% canary - kubectl apply -f - < 0.01" | bc -l) )); then - echo "Error rate too high, rolling back" - kubectl rollout undo deployment/app-canary - exit 1 - fi -``` - ---- - -## 4.6 Infrastructure as Code in CI - -### Terraform Validation Pipeline - -```yaml -# .github/workflows/terraform.yml -name: Terraform CI - -on: - pull_request: - paths: - - 'infrastructure/**' - -jobs: - terraform: - runs-on: ubuntu-latest - defaults: - run: - working-directory: infrastructure/ - - steps: - - uses: actions/checkout@v4 - - - uses: hashicorp/setup-terraform@v3 - with: - terraform_version: 1.9.0 - - - name: Terraform Format Check - run: terraform fmt -check -recursive - - - name: Terraform Init - run: terraform init -backend=false # Skip remote backend in PR - - - name: Terraform Validate - run: terraform validate - - - name: Run tflint (Terraform linter) - uses: terraform-linters/setup-tflint@v4 - - run: tflint --recursive - - - name: Run Checkov (security scan) - uses: bridgecrewio/checkov-action@master - with: - directory: infrastructure/ - framework: terraform - output_format: sarif - output_file_path: checkov.sarif - - - name: Terraform Plan - run: terraform plan -out=plan.tfplan - env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - - - name: Comment plan on PR - uses: actions/github-script@v7 - with: - script: | - const plan = require('fs').readFileSync('plan.txt', 'utf8'); - github.rest.issues.createComment({ - issue_number: context.issue.number, - owner: context.repo.owner, - repo: context.repo.repo, - body: `## Terraform Plan\n\`\`\`hcl\n${plan.slice(0, 60000)}\n\`\`\`` - }); -``` - ---- - -# Level 5 — Mastery - -## 5.1 CI Architecture at Scale - -### Monorepo CI with Nx (Affected-Only Builds) - -#### nx.json - -```json -{ - "affected": { - "defaultBase": "main" - }, - "tasksRunnerOptions": { - "default": { - "runner": "@nrwl/nx-cloud", - "options": { - "cacheableOperations": ["build", "test", "lint"], - "accessToken": "your-nx-cloud-token" - } - } - } -} -``` - -`nrwl/nx-cloud` used as distributed cache - -```yaml -# Only test/build affected projects — not the entire monorepo -- name: Get affected projects - run: | - AFFECTED=$(npx nx show projects --affected --type=app | tr '\n' ',') - echo "AFFECTED=${AFFECTED}" >> $GITHUB_ENV - -- name: Run affected tests - run: npx nx affected --target=test --parallel=4 - -- name: Build affected apps - run: npx nx affected --target=build --parallel=2 -``` - -### Distributed Caching with Bazel - -```python -# .bazelrc — remote cache configuration -build --remote_cache=grpcs://cache.example.com:443 -build --remote_header=Authorization=Bearer $BAZEL_CACHE_TOKEN -build --disk_cache=~/.cache/bazel - -# Enable remote execution for CI -build:ci --remote_executor=grpcs://rbe.example.com:443 -build:ci --remote_instance_name=default -``` - ---- - -## 5.2 Platform Engineering - -### Reusable Workflow Library (GitHub) - -```yaml -# .github/workflows/_reusable-node-ci.yml — shared template - -name: Reusable Node.js CI - -on: - workflow_call: - inputs: - node-version: - type: string - default: '20' - working-directory: - type: string - default: '.' - test-command: - type: string - default: 'npm test' - secrets: - SLACK_WEBHOOK_URL: - required: false - -jobs: - ci: - runs-on: ubuntu-latest - defaults: - run: - working-directory: ${{ inputs.working-directory }} - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: ${{ inputs.node-version }} - cache: 'npm' - cache-dependency-path: ${{ inputs.working-directory }}/package-lock.json - - run: npm ci - - run: npm run lint - - run: ${{ inputs.test-command }} - - run: npm run build -``` - -```yaml -# Consuming the reusable workflow in any project -name: CI -on: [push, pull_request] - -jobs: - ci: - uses: org/.github/.github/workflows/_reusable-node-ci.yml@main - with: - node-version: '22' - test-command: 'npm run test:coverage' - secrets: - SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} -``` - -### GitLab CI/CD Components (Catalog) - -```yaml -# components/node-ci/templates/node.yml -spec: - inputs: - node_version: - default: '20' - coverage_threshold: - default: '80' - ---- -node-lint: - image: node:$[[ inputs.node_version ]]-alpine - script: - - npm ci - - npm run lint - -node-test: - image: node:$[[ inputs.node_version ]]-alpine - script: - - npm ci - - npm test -- --coverage --coverageThreshold='{"global":{"lines":$[[ inputs.coverage_threshold ]]}}' - coverage: '/Lines\s*:\s*(\d+\.?\d*)%/' -``` - -```yaml -# Consuming the component -include: - - component: $CI_SERVER_FQDN/org/ci-catalog/node-ci@v1.0 - inputs: - node_version: '22' - coverage_threshold: '85' -``` - ---- - -## 5.3 DORA Metrics - -### Tracking the Four Key Metrics - -| Metric | How to Measure | Target (Elite) | -|---------------------------|--------------------------------------------------|----------------| -| **Deployment Frequency** | Count deploys to prod per day via CI logs | Multiple/day | -| **Lead Time for Changes** | Time from first commit to prod deploy | < 1 hour | -| **Change Failure Rate** | % of deploys that trigger a rollback or incident | < 5% | -| **MTTR** | Time between incident alert and resolved deploy | < 1 hour | - -### Collecting Metrics from GitHub Actions - -```python -# scripts/collect_dora.py — parse workflow runs and emit metrics - -import requests -import datetime - -GITHUB_TOKEN = os.environ['GITHUB_TOKEN'] -REPO = 'org/my-app' - -headers = {'Authorization': f'Bearer {GITHUB_TOKEN}'} - -def get_deploy_frequency(days=30): - since = (datetime.datetime.now() - datetime.timedelta(days=days)).isoformat() - runs = requests.get( - f'https://api.github.com/repos/{REPO}/actions/workflows/deploy.yml/runs', - params={'status': 'success', 'branch': 'main', 'created': f'>{since}'}, - headers=headers - ).json() - - total_deploys = runs['total_count'] - per_day = total_deploys / days - return {'total': total_deploys, 'per_day': round(per_day, 2)} - -def get_lead_time(): - # Compare first commit timestamp vs successful deploy timestamp - # Emit to Grafana/Datadog/custom dashboard - pass -``` - ---- - -## 5.4 CI Culture - -### The CI Social Contract - -These are team agreements — equally as important as the technical setup: - -```markdown -## Our CI Contract - -1. **Green main is sacred** — never merge a PR that breaks main. -2. **Fix the build before anything else** — a broken pipeline blocks everyone. -3. **You broke it, you fix it** — the author of the breaking commit owns the fix. -4. **Don't disable tests to make CI pass** — fix the root cause. -5. **PRs stay small** — large PRs are review liabilities and merge risks. -6. **Short-lived branches** — branches older than 2 days need a plan. -7. **No "works on my machine"** — if CI fails, it's a real problem. -8. **Review pipeline metrics weekly** — slow pipelines are tech debt. -``` - -### Trunk-Based Development Checklist - -```markdown -## TBD Readiness Checklist - -Infrastructure: -- [ ] CI pipeline runs in < 10 minutes -- [ ] Branch protection enforces CI before merge -- [ ] Automated deployment to at least one environment on merge to main - -Practices: -- [ ] Feature flags are available for hiding incomplete features -- [ ] All developers commit to main (or short-lived branches < 1 day) -- [ ] PR review SLA: < 2 hours during business hours -- [ ] On-call rotation to handle build failures quickly - -Code Health: -- [ ] Test suite is fast enough to run locally in < 2 minutes -- [ ] Flaky tests are quarantined and tracked -- [ ] Build produces the same output regardless of environment -``` - ---- - -# Reference: Full Pipeline Examples - -## Complete Node.js TypeScript CI/CD Pipeline - -```yaml -# .github/workflows/full-ci-cd.yml -name: Full CI/CD Pipeline - -on: - push: - branches: [main] - tags: ['v*'] - pull_request: - branches: [main] - -concurrency: # Cancel in-progress runs for same PR - group: ${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: true - -env: - REGISTRY: ghcr.io - IMAGE_NAME: ${{ github.repository }} - -jobs: - # ─── Quality Checks (parallel) ────────────────────────────────────── - lint: - name: Lint & Type Check - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: { node-version: '20', cache: 'npm' } - - run: npm ci - - run: npm run lint && npm run typecheck - - security: - name: Security Scan - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: { node-version: '20', cache: 'npm' } - - run: npm ci - - run: npm audit --audit-level=high - - uses: gitleaks/gitleaks-action@v2 - env: { GITHUB_TOKEN: '${{ secrets.GITHUB_TOKEN }}' } - - # ─── Tests (parallel) ─────────────────────────────────────────────── - unit-tests: - name: Unit Tests - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: { node-version: '20', cache: 'npm' } - - run: npm ci - - run: npm run test:unit -- --coverage - - uses: codecov/codecov-action@v4 - with: - token: ${{ secrets.CODECOV_TOKEN }} - - integration-tests: - name: Integration Tests - runs-on: ubuntu-latest - services: - postgres: - image: postgres:16-alpine - env: - POSTGRES_USER: test - POSTGRES_PASSWORD: test - POSTGRES_DB: testdb - options: --health-cmd pg_isready --health-interval 10s --health-retries 5 - ports: ['5432:5432'] - redis: - image: redis:7-alpine - ports: ['6379:6379'] - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: { node-version: '20', cache: 'npm' } - - run: npm ci - - run: npm run test:integration - env: - DATABASE_URL: postgresql://test:test@localhost:5432/testdb - REDIS_URL: redis://localhost:6379 - - # ─── Build ────────────────────────────────────────────────────────── - build: - name: Build & Push Image - runs-on: ubuntu-latest - needs: [lint, security, unit-tests, integration-tests] - permissions: - contents: read - packages: write - outputs: - image-tag: ${{ steps.meta.outputs.version }} - - steps: - - uses: actions/checkout@v4 - - uses: docker/setup-buildx-action@v3 - - uses: docker/login-action@v3 - with: - registry: ${{ env.REGISTRY }} - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - - id: meta - uses: docker/metadata-action@v5 - with: - images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} - tags: | - type=semver,pattern={{version}} - type=sha,prefix=sha- - type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }} - - uses: docker/build-push-action@v5 - with: - push: ${{ github.event_name != 'pull_request' }} - tags: ${{ steps.meta.outputs.tags }} - cache-from: type=gha - cache-to: type=gha,mode=max - - # ─── Deploy to Staging (on merge to main) ─────────────────────────── - deploy-staging: - name: Deploy to Staging - runs-on: ubuntu-latest - needs: build - if: github.ref == 'refs/heads/main' - environment: - name: staging - url: https://staging.example.com - - steps: - - run: | - echo "Deploying ${{ needs.build.outputs.image-tag }} to staging" - # kubectl / helm / ArgoCD / etc. - - # ─── Deploy to Production (on tag) ────────────────────────────────── - deploy-prod: - name: Deploy to Production - runs-on: ubuntu-latest - needs: [build, deploy-staging] - if: startsWith(github.ref, 'refs/tags/v') - environment: - name: production - url: https://example.com - - steps: - - run: | - echo "Deploying ${{ needs.build.outputs.image-tag }} to production" -``` - ---- - -## Complete Python FastAPI CI Pipeline - -```yaml -# .github/workflows/python-ci.yml -name: Python CI - -on: [push, pull_request] - -jobs: - ci: - runs-on: ubuntu-latest - strategy: - matrix: - python-version: ['3.11', '3.12'] - - steps: - - uses: actions/checkout@v4 - - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v5 - with: - python-version: ${{ matrix.python-version }} - cache: 'pip' - - - name: Install dependencies - run: | - pip install --upgrade pip - pip install -r requirements.txt - pip install -r requirements-dev.txt - - - name: Lint with Ruff - run: ruff check . --output-format=github - - - name: Format check with Black - run: black --check . - - - name: Type check with mypy - run: mypy src/ - - - name: Run tests - run: pytest -v --cov=src --cov-report=xml --cov-fail-under=80 - - - name: Upload coverage - uses: codecov/codecov-action@v4 - with: - token: ${{ secrets.CODECOV_TOKEN }} - files: coverage.xml -``` - ---- - -*This guide is a living document. CI tooling evolves rapidly — always verify against official documentation.* -*Last updated: March 2026* diff --git a/Lessons/index.md b/Lessons/index.md new file mode 100644 index 0000000..55d9235 --- /dev/null +++ b/Lessons/index.md @@ -0,0 +1,17 @@ +# Continuous Integration — Complete Practical Guide + +> A hands-on, sample-driven companion to the CI Learning Path. +> Every concept is paired with working code, real configurations, and actionable explanations. + +--- + +## Table of Contents + +1. [Level 1 — Foundations](01-foundations.md) +2. [Level 2 — Core CI Skills](02-core-ci-skills.md) +3. [Level 3 — Intermediate CI](03-intermediate-ci.md) +4. [Level 4 — Advanced CI](04-advanced-ci.md) +5. [Level 5 — Mastery](05-mastery.md) +6. [Reference: Full Pipeline Examples](99-reference.md) + +--- From eaa23119d1a1e34a784547c6fd2c003f74e01814 Mon Sep 17 00:00:00 2001 From: Ivan Buttinoni Date: Wed, 11 Mar 2026 10:03:17 +0100 Subject: [PATCH 2/5] fix formatting --- Lessons/99-reference.md | 1 - 1 file changed, 1 deletion(-) diff --git a/Lessons/99-reference.md b/Lessons/99-reference.md index bce5287..f31cb99 100644 --- a/Lessons/99-reference.md +++ b/Lessons/99-reference.md @@ -203,4 +203,3 @@ jobs: *This guide is a living document. CI tooling evolves rapidly — always verify against official documentation.* *Last updated: March 2026* - From 83b5435e7c7b2316cc0781952d5c83a94a8667e8 Mon Sep 17 00:00:00 2001 From: Ivan Buttinoni Date: Wed, 11 Mar 2026 14:57:08 +0100 Subject: [PATCH 3/5] Add comprehensive Docker in CI documentation - Explain why Docker is used in CI pipelines for environment parity, reproducibility, and isolation. - Describe the goal of producing a tagged, versioned image in a container registry. - Detail the three-file setup: Dockerfile, .dockerignore, and GitHub Actions workflow. - Expand on multi-stage Dockerfile best practices and layer caching strategies. - Clarify the workflow from push to pullable image, including authentication and tagging. - Add examples for different ecosystems (Python, Go, Java) and common mistakes to avoid. --- Lessons/03-intermediate-ci.md | 360 ++++++++++++++++++++++++++++++++-- 1 file changed, 346 insertions(+), 14 deletions(-) diff --git a/Lessons/03-intermediate-ci.md b/Lessons/03-intermediate-ci.md index 2723f3e..604720a 100644 --- a/Lessons/03-intermediate-ci.md +++ b/Lessons/03-intermediate-ci.md @@ -2,6 +2,88 @@ ## 3.1 Docker in CI +### Why Docker in a CI Pipeline + +Without Docker, a CI pipeline runs directly on the host runner — a shared machine with a pre-installed OS, language runtimes, and system libraries. This creates a class of problems that become more painful as a project grows: + +``` +Without Docker: + + Runner OS: Ubuntu 22.04 + Node version: 18.x (whatever the runner has installed) + Python version: 3.10 (same) + System libs: libpq 14, openssl 1.1 ... + + Your app expects: Node 20, Python 3.12, libpq 16 + Result: "works on my machine" — fails in CI, or worse, + passes in CI and fails in production +``` + +Docker solves this by packaging the application **together with its exact environment** into a self-contained image. The image runs identically on a developer's laptop, a CI runner, a staging server, and a production Kubernetes cluster — because they are all running the exact same filesystem snapshot. + +**The three reasons to use Docker in CI:** + +1. **Environment parity.** The image that CI builds and tests is the exact same artifact that gets deployed. Not "the same code built in a different environment" — the same binary image, byte for byte. If it passes tests in CI, it will behave the same way in production. + +2. **Reproducibility.** The Dockerfile is a precise, version-controlled recipe. Anyone can rebuild the image from scratch and get an identical result. No hidden state, no "it worked last week" surprises. + +3. **Isolation.** Multiple projects with conflicting dependencies (different Node versions, different system libraries) can run on the same CI runner without interfering with each other, because each runs inside its own container. + +### The Goal: From Source Code to a Pullable Image + +The end goal of the Docker section of a CI pipeline is to produce a **tagged, versioned image stored in a container registry** (in this guide, GHCR — GitHub Container Registry). Once the image is there, any downstream system — a deployment pipeline, a Kubernetes cluster, a developer's local machine — can pull and run it without needing the source code, build tools, or any knowledge of how it was built. + +``` +Source code Container Registry (GHCR) +───────────── ────────────────────────── +src/ +package.json ──► CI ──► ghcr.io/org/repo:main +Dockerfile ghcr.io/org/repo:1.2.0 + ghcr.io/org/repo:sha-a3f2c1d + │ + ▼ + Kubernetes / Docker / any server + docker pull ghcr.io/org/repo:sha-a3f2c1d + docker run ghcr.io/org/repo:sha-a3f2c1d +``` + +**Why push to a registry rather than build on every server?** + +Building an image takes time and requires build tools, source code, and network access to download dependencies. A registry decouples the build from the deployment: you build once in CI, store the result, and deploy the stored artifact everywhere. Servers in production do not need compilers, package managers, or source code — they only need `docker pull`. + +**Why tag by commit SHA?** + +The `sha-a3f2c1d` tag is immutable — it always refers to the exact image built from that specific commit. Tags like `:latest` or `:main` are mutable and move with every push, making it impossible to know which version is actually running. In production deployments, always pin to the SHA tag. + +``` +Mutable tag (risky in production): + image: ghcr.io/org/repo:latest ← changes with every push, no audit trail + +Immutable tag (safe): + image: ghcr.io/org/repo:sha-a3f2c1d ← always this exact build +``` + +### The Three-File Setup + +A complete Docker-in-CI setup involves three files working together: + +``` +Repository: + ├── Dockerfile (1) defines the image build + ├── .dockerignore (2) controls what enters the build context + └── .github/workflows/docker.yml (3) automates build + push on every commit +``` + +| File | Responsibility | Runs on | +|------|---------------|---------| +| `Dockerfile` | How to build the image, layer by layer | Docker daemon on the CI runner | +| `.dockerignore` | Which files to exclude from the build context | Docker daemon, before any layer | +| `docker.yml` | When to trigger the build, how to authenticate, where to push | GitHub Actions runner | + +The sections below cover each in detail. + +--- + ### Multi-Stage Dockerfile A well-structured Dockerfile that keeps the production image small and the build environment separate: @@ -43,6 +125,52 @@ CMD ["node", "dist/index.js"] ### Building and Pushing in GitHub Actions +The workflow file and the Dockerfile are two separate files with distinct responsibilities that work together as a pipeline: + +``` +Repository on disk: + ├── Dockerfile ← defines HOW to build the image + └── .github/ + └── workflows/ + └── docker.yml ← defines WHEN to build and WHERE to push it +``` + +The Dockerfile is not referenced by name anywhere in the workflow YAML. Instead, the `docker/build-push-action` step uses the `context: .` parameter — meaning "the root of the checked-out repository" — and Docker automatically looks for a file named `Dockerfile` in that directory. The multi-stage build defined in the Dockerfile is what runs when the action executes `docker build`. + +**End-to-end flow from a push to a pullable image:** + +``` +1. Developer pushes to main (or creates a tag like v1.2.0) + │ + ▼ +2. GitHub Actions triggers docker.yml + │ + ▼ +3. Runner checks out the repository + (Dockerfile is now present on the runner at ./Dockerfile) + │ + ▼ +4. docker/build-push-action runs docker build + ┌─────────────────────────────────────────────────┐ + │ Executes the Dockerfile stages in order: │ + │ Stage 1 (deps) — installs node_modules │ + │ Stage 2 (builder) — runs npm run build │ + │ Stage 3 (runner) — assembles production image │ + └─────────────────────────────────────────────────┘ + │ + ▼ +5. The final image (Stage 3 only) is tagged and pushed to GHCR + ghcr.io/org/repo:main + ghcr.io/org/repo:sha-a3f2c1d + ghcr.io/org/repo:1.2.0 ← if triggered by a tag + │ + ▼ +6. Image is now pullable by anyone with access to the repo: + docker pull ghcr.io/org/repo:main +``` + +> **Important:** only the final stage of the multi-stage Dockerfile becomes the pushed image. The intermediate stages (`deps`, `builder`) are used during the build and then discarded. They never appear in GHCR. + ```yaml # .github/workflows/docker.yml @@ -51,71 +179,274 @@ name: Build & Push Docker Image on: push: branches: [main] - tags: ['v*'] + tags: ['v*'] # Matches v1.0.0, v2.3.1, etc. jobs: docker: runs-on: ubuntu-latest permissions: contents: read - packages: write # Required to push to GHCR + packages: write # Required to push to GHCR — without this + # the GITHUB_TOKEN cannot write to the registry steps: - - uses: actions/checkout@v4 + - name: Checkout repository + uses: actions/checkout@v4 + # After this step, the runner has the full repo on disk, + # including the Dockerfile at ./Dockerfile # Enable Docker layer caching via GitHub cache - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 + # Buildx is an extended Docker build client that supports + # multi-platform builds and the GitHub Actions cache backend + # (cache-from: type=gha). The standard `docker build` command + # does not support the GHA cache backend. - name: Log in to GitHub Container Registry uses: docker/login-action@v3 with: registry: ghcr.io - username: ${{ github.actor }} + username: ${{ github.actor }} # The user who triggered the workflow password: ${{ secrets.GITHUB_TOKEN }} - + # GITHUB_TOKEN is automatically available in every workflow — + # no manual secret setup required. It is scoped to this repo + # and expires when the workflow run ends. + # The `packages: write` permission above is what allows it to push. + # Generate image tags based on git context - - name: Extract metadata + - name: Extract metadata (tags and labels) id: meta uses: docker/metadata-action@v5 with: images: ghcr.io/${{ github.repository }} + # github.repository = "org/repo-name" + # Full image name = "ghcr.io/org/repo-name" tags: | type=ref,event=branch + # → ghcr.io/org/repo:main (on push to main) + type=semver,pattern={{version}} + # → ghcr.io/org/repo:1.2.0 (on tag v1.2.0) + type=semver,pattern={{major}}.{{minor}} + # → ghcr.io/org/repo:1.2 (floating minor tag) + type=sha,prefix=sha- + # → ghcr.io/org/repo:sha-a3f2c1d (always unique, per commit) + # The sha tag is the most useful for deployment pipelines — + # it is immutable and always points to the exact commit that was built. - name: Build and push uses: docker/build-push-action@v5 with: context: . + # "." means: use the root of the checked-out repo as the build context. + # Docker will look for ./Dockerfile automatically. + # To use a different path: context: ./services/api + # To use a different filename: file: ./docker/Dockerfile.prod + push: true + # push: false would build the image without pushing — useful for + # PR validation where you want to confirm the image builds + # but don't want to publish it yet. + tags: ${{ steps.meta.outputs.tags }} + # The list of ghcr.io/... tags computed in the metadata step above. + # One image, multiple tags pointing to it. + labels: ${{ steps.meta.outputs.labels }} - cache-from: type=gha # Use GitHub Actions cache for layers + # OCI standard labels: org.opencontainers.image.source, + # .created, .revision, etc. Makes the image traceable back + # to the exact commit and workflow run that built it. + + cache-from: type=gha cache-to: type=gha,mode=max + # Stores Docker layer cache in GitHub Actions Cache (not in GHCR). + # On subsequent runs, unchanged layers are restored from cache + # instead of being rebuilt — the same benefit as the Dockerfile + # layer ordering described above, but persisted across runner instances. + # mode=max caches all intermediate layers, not just the final stage. +``` + +**Where the image lands in GHCR and how to pull it:** + +Once the workflow completes, the image is visible at: +``` +https://github.com/ORG/REPO/pkgs/container/REPO +``` + +By default, a newly pushed image inherits the visibility of the repository (public repo → public image, private repo → private image). To pull it: + +```bash +# Public image — no authentication needed +docker pull ghcr.io/org/repo:main + +# Private image — authenticate first +echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin +docker pull ghcr.io/org/repo:main + +# Pull a specific immutable build by commit SHA +docker pull ghcr.io/org/repo:sha-a3f2c1d + +# In a Kubernetes manifest +# image: ghcr.io/org/repo:sha-a3f2c1d ← prefer SHA over :latest in production ``` ### Docker Layer Caching Strategy +Docker builds images as a stack of layers. Each `COPY` and `RUN` instruction creates a new layer. When a layer changes, Docker invalidates that layer **and every layer above it** — forcing all subsequent instructions to re-run from scratch. Understanding this cascade is the key to fast, cache-efficient builds in CI. + +``` +Layer stack (top = last instruction): + + [5] RUN npm run build ← rebuilt if layer 4 changes + [4] COPY src/ ./src/ ← rebuilt if layer 3 changes or source files change + [3] RUN npm ci ← rebuilt if layer 2 changes + [2] COPY package*.json ./ ← rebuilt if base layer changes or package.json changes + [1] FROM node:20-alpine ← base image, rarely changes +``` + +The rule: **put instructions that change rarely near the bottom, instructions that change often near the top.** + +--- + +#### ✅ Correct Pattern — Separate Dependencies from Source + +Copy the package manifest files first, install dependencies, then copy source code. This way a source file change only reruns the build step, not the install step. + ```dockerfile -# ✅ GOOD: Copy package files BEFORE source code -# This way, npm install is only re-run when package.json changes -COPY package*.json ./ +FROM node:20-alpine AS builder +WORKDIR /app -# Cached unless package*.json changed +# Step 1 — copy only the files that define your dependencies +COPY package.json package-lock.json ./ + +# Step 2 — install; this layer is cached as long as the lock file +# does not change, regardless of any source code edits RUN npm ci -# Source changes don't invalidate the npm layer + +# Step 3 — now copy source; changes here only invalidate layers above COPY src/ ./src/ +COPY tsconfig.json ./ + +# Step 4 — build; re-runs only when source changes, not on every commit RUN npm run build +``` + +What gets cached and what gets rebuilt on a typical source-only change: + +```text +Commit: changed src/api/users.ts + + [1] FROM node:20-alpine ✅ cache hit + [2] COPY package*.json ✅ cache hit (manifest unchanged) + [3] RUN npm ci ✅ cache hit (skipped — ~60s saved) + [4] COPY src/ ❌ cache miss (source changed) + [5] RUN npm run build ❌ re-runs (~10s) + + Total: ~10s instead of ~70s +``` + +The same pattern applies to other ecosystems: + +```dockerfile +# Python +COPY requirements.txt ./ +RUN pip install --no-cache-dir -r requirements.txt +COPY src/ ./src/ + +# Go +COPY go.mod go.sum ./ +RUN go mod download +COPY . . +RUN go build ./... + +# Java / Maven +COPY pom.xml ./ +RUN mvn dependency:go-offline -q +COPY src/ ./src/ +RUN mvn package -DskipTests +``` + +--- + +#### ❌ Common Mistake — Copying Everything at Once + +The most common mistake: a single `COPY . .` before installing dependencies. Any change to any file in the project — including a one-line edit in a README — busts the dependency cache. + +```dockerfile +FROM node:20-alpine AS builder +WORKDIR /app -# ❌ BAD: Copy everything at once — any source change busts the cache +# ❌ This copies source, tests, docs, configs — everything COPY . . + +# ❌ npm ci re-runs on EVERY build because the layer above always changes +# Even if package.json hasn't been touched RUN npm ci RUN npm run build ``` +What happens on every single commit, including trivial ones: + +```text +Commit: fixed a typo in README.md + + [1] FROM node:20-alpine ✅ cache hit + [2] COPY . . ❌ cache miss (any file change breaks this) + [3] RUN npm ci ❌ re-runs (~60s wasted) + [4] RUN npm run build ❌ re-runs (~10s) + + Total: ~70s on every push, even for a typo fix +``` + +Other variants of the same mistake: + +```dockerfile +# ❌ Variant: copying the whole project before the lock file specifically +COPY . . +COPY package-lock.json ./ # too late — layer 1 already invalidated + +# ❌ Variant: using ADD instead of COPY (adds remote fetch overhead too) +ADD . /app + +# ❌ Variant: no .dockerignore — node_modules, .git, coverage/ all copied +# into the image context, slowing down even cache-hit builds +``` + +#### .dockerignore — Exclude What Docker Doesn't Need + +A missing `.dockerignore` silently breaks cache efficiency. Docker sends the entire build context to the daemon before evaluating any cache. If `node_modules/` (potentially hundreds of MB) is in the context, every build pays that transfer cost even on a cache hit. + +```dockerignore +# .dockerignore + +# Dependencies — rebuilt inside the image, not copied from host +node_modules/ +.venv/ +vendor/ + +# Version control +.git/ +.gitignore + +# CI and editor artifacts +coverage/ +.nyc_output/ +dist/ +build/ +*.log +.env +.env.* + +# Documentation and tooling not needed at runtime +README.md +CHANGELOG.md +.eslintrc* +.prettierrc* +``` + --- ## 3.2 Code Quality Gates @@ -176,8 +507,9 @@ npx husky init } ``` +#### .husky/pre-commit + ```bash -# .husky/pre-commit #!/bin/sh npx lint-staged ``` From bc94a7bed6b8e2c3beb3f5899c96292e0bcd227e Mon Sep 17 00:00:00 2001 From: Ivan Buttinoni Date: Wed, 11 Mar 2026 15:26:08 +0100 Subject: [PATCH 4/5] Add comprehensive GitHub Actions workflow guide --- Lessons/60-github-workflow-guide.md | 1256 +++++++++++++++++++++++++++ 1 file changed, 1256 insertions(+) create mode 100644 Lessons/60-github-workflow-guide.md diff --git a/Lessons/60-github-workflow-guide.md b/Lessons/60-github-workflow-guide.md new file mode 100644 index 0000000..237a64e --- /dev/null +++ b/Lessons/60-github-workflow-guide.md @@ -0,0 +1,1256 @@ +# Annex A — GitHub Actions Workflow Guide + +> A complete reference for GitHub Actions workflow structure, syntax, and control flow patterns. +> Designed as a companion to the CI Learning Path. + +--- + +## Table of Contents + +1. [What Is a Workflow?](#1-what-is-a-workflow) +2. [Anatomy of a Workflow File](#2-anatomy-of-a-workflow-file) +3. [Triggers (`on`)](#3-triggers-on) +4. [Runners (`runs-on`)](#4-runners-runs-on) +5. [Steps — The Unit of Work](#5-steps--the-unit-of-work) +6. [Sequential vs Parallel Jobs](#6-sequential-vs-parallel-jobs) +7. [Dependent Jobs (`needs`)](#7-dependent-jobs-needs) +8. [Conditional Flow (`if`)](#8-conditional-flow-if) +9. [Loop / Matrix Flow (`strategy.matrix`)](#9-loop--matrix-flow-strategymatrix) +10. [Expressions, Contexts, and Variables](#10-expressions-contexts-and-variables) +11. [Secrets and Environment Variables](#11-secrets-and-environment-variables) +12. [Artifacts and Data Passing Between Jobs](#12-artifacts-and-data-passing-between-jobs) +13. [Reusable Workflows and Composite Actions](#13-reusable-workflows-and-composite-actions) +14. [Permissions](#14-permissions) +15. [Concurrency Control](#15-concurrency-control) +16. [Timeouts and Failure Strategies](#16-timeouts-and-failure-strategies) +17. [Caching Dependencies](#17-caching-dependencies) +18. [Complete Reference Pipeline](#18-complete-reference-pipeline) + +--- + +## 1. What Is a Workflow? + +A **workflow** is an automated process defined in a YAML file stored under `.github/workflows/` in your repository. GitHub Actions detects and runs every `.yml` file in that directory according to its trigger conditions. + +``` +Repository +└── .github/ + └── workflows/ + ├── ci.yml ← runs on every push / PR + ├── docker.yml ← builds and pushes images + ├── release.yml ← creates releases on tag push + └── nightly.yml ← scheduled security scans +``` + +Each workflow is independent. They can run in parallel with each other. One workflow cannot directly call another — unless you use the `workflow_call` trigger (see [Reusable Workflows](#13-reusable-workflows-and-composite-actions)). + +--- + +## 2. Anatomy of a Workflow File + +A complete workflow is composed of four top-level keys: + +```yaml +name: CI Pipeline # (1) Display name shown in the GitHub UI + +on: [push, pull_request] # (2) Trigger conditions — when does this workflow run? + +env: # (3) Workflow-level environment variables (optional) + NODE_ENV: test + +jobs: # (4) The work to be done — one or more named jobs + build: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Run tests + run: npm test +``` + +**How the keys relate to each other:** + +``` +Workflow +│ +├── on → defines WHEN the workflow runs +├── env → variables available to ALL jobs +└── jobs → defines WHAT runs + ├── job-a → a named job + │ ├── runs-on → which machine type + │ ├── env → variables for this job only + │ └── steps → ordered list of commands and actions + └── job-b → another named job (runs in parallel by default) +``` + +--- + +## 3. Triggers (`on`) + +The `on` key controls when a workflow is activated. It is the most nuanced part of the file. + +### 3.1 Simple Event Triggers + +```yaml +on: push # Any push to any branch + +on: [push, pull_request] # Either event activates the workflow +``` + +### 3.2 Filtered Push Triggers — Branch Scoping + +This is the most common production pattern: run the workflow only when certain branches or tags are pushed. + +```yaml +on: + push: + branches: + - main # Exact name + - 'release/**' # Glob: any branch starting with release/ + - 'feature/**' + tags: + - 'v*' # Any tag starting with v (e.g. v1.0.0, v2.3.1) + paths: + - 'src/**' # Only run if files under src/ changed + - '!docs/**' # Exclude docs/ changes (! = negation) +``` + +**Branch filter logic:** + +``` +Push to main → workflow runs (matches 'main') +Push to release/1.0 → workflow runs (matches 'release/**') +Push to develop → workflow skipped (no matching rule) +Push tag v1.2.0 → workflow runs (matches 'v*') +Push tag beta-1 → workflow skipped (no match) +Push to main (docs only) → workflow skipped (paths exclude docs/**) +``` + +### 3.3 Pull Request Triggers + +```yaml +on: + pull_request: + branches: + - main # Only PRs targeting main + - 'release/**' + types: + - opened # PR was opened + - synchronize # New commits pushed to the PR branch + - reopened # PR was re-opened after being closed + # Other types: edited, labeled, unlabeled, closed, ready_for_review +``` + +> **Important:** `pull_request` workflows from forked repositories run with read-only permissions and no access to secrets, for security reasons. Use `pull_request_target` (with caution) if you need secret access in fork PRs. + +### 3.4 Scheduled Triggers (Cron) + +```yaml +on: + schedule: + - cron: '0 6 * * 1' # Every Monday at 06:00 UTC + - cron: '0 2 * * *' # Every day at 02:00 UTC +``` + +Cron format: `minute hour day-of-month month day-of-week` + +``` +'0 6 * * 1' + │ │ │ │ └── day of week: 1 = Monday (0=Sun, 7=Sun) + │ │ │ └──── month: * = every month + │ │ └────── day of month: * = every day + │ └──────── hour: 6 = 06:00 UTC + └────────── minute: 0 = top of the hour +``` + +### 3.5 Manual Trigger + +```yaml +on: + workflow_dispatch: + inputs: + environment: + description: 'Target environment' + required: true + default: 'staging' + type: choice + options: [staging, production] + debug: + description: 'Enable debug logging' + type: boolean + default: false +``` + +`workflow_dispatch` adds a "Run workflow" button in the GitHub UI. Inputs are available in steps as `${{ inputs.environment }}`. + +### 3.6 Other Useful Triggers + +```yaml +on: + release: + types: [published] # When a GitHub Release is published + + workflow_run: + workflows: ["CI"] # When another named workflow completes + types: [completed] + + workflow_call: # This workflow can be called by other workflows + inputs: + version: + type: string + required: true +``` + +--- + +## 4. Runners (`runs-on`) + +The runner is the virtual machine that executes the job. GitHub provides hosted runners; you can also register self-hosted runners. + +```yaml +jobs: + build: + runs-on: ubuntu-latest # Recommended for most CI tasks + + mac-build: + runs-on: macos-latest # Required for iOS/macOS builds + + windows-build: + runs-on: windows-latest # Required for Windows-specific testing +``` + +**Available GitHub-hosted runners:** + +| Label | OS | Notes | +|---|---|---| +| `ubuntu-latest` | Ubuntu 22.04 | Fastest, cheapest, most tooling pre-installed | +| `ubuntu-22.04` | Ubuntu 22.04 | Pin to a specific version | +| `ubuntu-20.04` | Ubuntu 20.04 | Legacy support | +| `macos-latest` | macOS 14 | Required for Xcode/Swift | +| `windows-latest` | Windows Server 2022 | Required for .NET/MSBuild | + +**Self-hosted runners:** + +```yaml +runs-on: [self-hosted, linux, x64, gpu] +# Labels allow targeting specific machine capabilities +``` + +--- + +## 5. Steps — The Unit of Work + +A job is a sequence of steps executed in order on the same runner. Each step is either a **shell command** (`run`) or a **pre-built action** (`uses`). + +### 5.1 Shell Commands (`run`) + +```yaml +steps: + - name: Install dependencies + run: npm ci + + - name: Multi-line command + run: | + echo "Running tests..." + npm test + echo "Done" + + - name: Use a different shell + shell: python + run: | + import sys + print(f"Python {sys.version}") +``` + +### 5.2 Pre-built Actions (`uses`) + +```yaml +steps: + - name: Checkout repository + uses: actions/checkout@v4 # Official GitHub action, version 4 + with: + fetch-depth: 0 # Clone full history (default: 1 shallow) + token: ${{ secrets.PAT }} # Use a PAT instead of GITHUB_TOKEN + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '20' + cache: 'npm' # Built-in cache for node_modules + + - name: Custom action from same repo + uses: ./.github/actions/my-action +``` + +### 5.3 Step Outputs + +Steps can emit outputs that later steps within the same job can read: + +```yaml +steps: + - name: Get version + id: version # ← assign an id to reference this step + run: | + VERSION=$(cat package.json | jq -r '.version') + echo "tag=v${VERSION}" >> $GITHUB_OUTPUT # write output + + - name: Print version + run: echo "Version is ${{ steps.version.outputs.tag }}" + # ↑ reference by step id +``` + +### 5.4 Step-Level Environment Variables + +```yaml +steps: + - name: Deploy + env: + API_URL: https://api.example.com + DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }} + run: ./scripts/deploy.sh +``` + +--- + +## 6. Sequential vs Parallel Jobs + +### 6.1 Parallel (Default) + +All jobs at the top level of `jobs:` run **in parallel** by default. GitHub spins up a separate runner VM for each job simultaneously. + +```yaml +jobs: + lint: # ┐ + runs-on: ubuntu-latest # │ these three start at the same time + steps: ... # │ + # │ + test: # │ + runs-on: ubuntu-latest # │ + steps: ... # │ + # │ + security: # ┘ + runs-on: ubuntu-latest + steps: ... +``` + +``` +Time → + +lint ████████████ +test ████████████████████ +security ████████████ +``` + +Use parallel jobs to cut wall-clock time. Each job has an isolated, fresh environment. + +### 6.2 Sequential (Using `needs`) + +To force sequential execution, use `needs:` to declare dependencies. See [Section 7](#7-dependent-jobs-needs) for the full reference. + +```yaml +jobs: + build: + runs-on: ubuntu-latest + steps: ... + + test: + needs: build # test waits for build to succeed + runs-on: ubuntu-latest + steps: ... + + deploy: + needs: test # deploy waits for test to succeed + runs-on: ubuntu-latest + steps: ... +``` + +``` +Time → + +build ████████ + test ████████████ + deploy ████ +``` + +### 6.3 Fan-out / Fan-in Pattern + +A common real-world pattern: one job triggers parallel work, then a final job waits for all of them. + +```yaml +jobs: + build: # runs first + ... + + test-unit: # run in parallel after build + needs: build + ... + + test-integration: + needs: build + ... + + test-e2e: + needs: build + ... + + deploy: # runs after ALL three test jobs pass + needs: [test-unit, test-integration, test-e2e] + ... +``` + +``` +Time → + +build ████████ + test-unit ████████ + test-integration ████████████ + test-e2e ██████ + deploy ████ +``` + +--- + +## 7. Dependent Jobs (`needs`) + +`needs` accepts either a single job name or a list of job names. The current job starts only after all listed dependencies complete successfully. + +```yaml +jobs: + a: + runs-on: ubuntu-latest + steps: + - run: echo "job a" + + b: + needs: a # depends on a + runs-on: ubuntu-latest + steps: + - run: echo "job b" + + c: + needs: a # also depends on a (parallel with b) + runs-on: ubuntu-latest + steps: + - run: echo "job c" + + d: + needs: [b, c] # depends on BOTH b AND c + runs-on: ubuntu-latest + steps: + - run: echo "job d" +``` + +### Accessing Outputs Across Jobs + +```yaml +jobs: + build: + runs-on: ubuntu-latest + outputs: + image-tag: ${{ steps.meta.outputs.version }} # expose step output as job output + steps: + - id: meta + run: echo "version=1.2.3" >> $GITHUB_OUTPUT + + deploy: + needs: build + runs-on: ubuntu-latest + steps: + - run: echo "Deploying ${{ needs.build.outputs.image-tag }}" + # ↑ access via needs..outputs. +``` + +### Running a Dependent Job Even on Failure + +By default, if a dependency fails, the dependent job is skipped. Use `always()` or `failure()` conditions to override: + +```yaml + notify: + needs: [build, test, deploy] + if: always() # run regardless of upstream results + runs-on: ubuntu-latest + steps: + - run: echo "Pipeline done, result = ${{ job.status }}" +``` + +--- + +## 8. Conditional Flow (`if`) + +The `if` key can be applied to an entire **job** or to an individual **step**. The job/step is skipped when the expression evaluates to false. + +### 8.1 Job-Level Conditions + +```yaml +jobs: + deploy-staging: + runs-on: ubuntu-latest + if: github.ref == 'refs/heads/develop' # only on the develop branch + + deploy-production: + runs-on: ubuntu-latest + if: startsWith(github.ref, 'refs/tags/v') # only on version tags + + notify-failure: + needs: [build, test] + if: failure() # only if any dependency failed +``` + +### 8.2 Step-Level Conditions + +```yaml +steps: + - name: Upload coverage + if: success() # default — only if previous steps succeeded + + - name: Notify on failure + if: failure() # only if a prior step failed + + - name: Always clean up + if: always() # runs regardless + + - name: Only on main + if: github.ref == 'refs/heads/main' + + - name: Only on PR + if: github.event_name == 'pull_request' + + - name: Only on manual trigger with debug enabled + if: github.event_name == 'workflow_dispatch' && inputs.debug == 'true' +``` + +### 8.3 Status Functions + +| Function | Evaluates to true when... | +|---|---| +| `success()` | All previous steps succeeded (default) | +| `failure()` | At least one previous step failed | +| `cancelled()` | The workflow was cancelled | +| `always()` | Unconditionally true | + +### 8.4 Expression Operators + +```yaml +if: github.actor == 'dependabot[bot]' # equality +if: github.ref != 'refs/heads/main' # inequality +if: contains(github.event.head_commit.message, '[skip ci]') # contains() +if: startsWith(github.ref, 'refs/tags/') # startsWith() +if: github.event_name == 'push' && github.ref == 'refs/heads/main' # AND +if: github.event_name == 'push' || github.event_name == 'workflow_dispatch' # OR +if: !contains(github.event.pull_request.labels.*.name, 'skip-ci') # NOT +``` + +--- + +## 9. Loop / Matrix Flow (`strategy.matrix`) + +A matrix builds a **job for each combination** of values you define. It is the workflow equivalent of a for-loop. Each cell in the matrix becomes an independent parallel job with its own runner. + +### 9.1 Single-Dimension Matrix + +```yaml +jobs: + test: + runs-on: ubuntu-latest + strategy: + matrix: + node-version: [18, 20, 22] # run once for each value + + steps: + - uses: actions/setup-node@v4 + with: + node-version: ${{ matrix.node-version }} # access current value + - run: npm test +``` + +This creates 3 parallel jobs: +- `test (18)` +- `test (20)` +- `test (22)` + +### 9.2 Multi-Dimension Matrix + +```yaml +strategy: + matrix: + os: [ubuntu-latest, macos-latest, windows-latest] + node: [18, 20] +``` + +This creates **3 × 2 = 6** parallel jobs, one for every combination: + +``` +ubuntu-latest + node 18 +ubuntu-latest + node 20 +macos-latest + node 18 +macos-latest + node 20 +windows-latest + node 18 +windows-latest + node 20 +``` + +### 9.3 Matrix with `include` (Add Specific Combinations) + +```yaml +strategy: + matrix: + os: [ubuntu-latest, windows-latest] + python: ['3.10', '3.12'] + include: + - os: ubuntu-latest + python: '3.9' # add this specific combo + experimental: true # add extra variable for this combo only +``` + +### 9.4 Matrix with `exclude` (Remove Specific Combinations) + +```yaml +strategy: + matrix: + os: [ubuntu-latest, macos-latest, windows-latest] + node: [18, 20] + exclude: + - os: windows-latest + node: 18 # skip Node 18 on Windows +``` + +### 9.5 Failure Handling in Matrices + +```yaml +strategy: + fail-fast: false # default is true — false lets all cells complete + # even if one fails (useful for experimental matrices) + max-parallel: 3 # limit concurrent jobs to avoid rate limits + matrix: + node: [18, 20, 22] +``` + +With `fail-fast: true` (default), if `node 18` fails, GitHub immediately cancels all other in-progress matrix jobs. With `fail-fast: false`, all 3 complete regardless. + +--- + +## 10. Expressions, Contexts, and Variables + +### 10.1 Expression Syntax + +Expressions are wrapped in `${{ }}` and can appear in most YAML values: + +```yaml +run: echo "Branch is ${{ github.ref_name }}" +if: github.actor != 'dependabot[bot]' +with: + node-version: ${{ matrix.node }} +``` + +### 10.2 Key Contexts + +**`github` context** — information about the event that triggered the workflow: + +```yaml +${{ github.repository }} # "org/repo-name" +${{ github.ref }} # "refs/heads/main" or "refs/tags/v1.0.0" +${{ github.ref_name }} # "main" or "v1.0.0" (short name) +${{ github.sha }} # Full commit SHA +${{ github.actor }} # Username who triggered the event +${{ github.event_name }} # "push", "pull_request", "workflow_dispatch" +${{ github.run_id }} # Unique ID for this workflow run +${{ github.run_number }} # Auto-incrementing run count for this workflow +${{ github.server_url }} # "https://github.com" +${{ github.workspace }} # Path to checked-out repo on runner +``` + +**`job` context** — the current job's status: + +```yaml +${{ job.status }} # "success", "failure", "cancelled" +``` + +**`steps` context** — outputs from earlier steps in the same job: + +```yaml +${{ steps..outputs. }} +${{ steps..outcome }} # "success", "failure", "skipped", "cancelled" +${{ steps..conclusion }} # final status after continue-on-error +``` + +**`needs` context** — outputs from upstream jobs: + +```yaml +${{ needs..outputs. }} +${{ needs..result }} # "success", "failure", "skipped", "cancelled" +``` + +**`matrix` context** — current matrix cell values: + +```yaml +${{ matrix.node-version }} +${{ matrix.os }} +``` + +**`secrets` context** — encrypted secrets: + +```yaml +${{ secrets.MY_TOKEN }} +${{ secrets.GITHUB_TOKEN }} # auto-provided, no setup needed +``` + +**`inputs` context** — `workflow_dispatch` or `workflow_call` inputs: + +```yaml +${{ inputs.environment }} +${{ inputs.debug }} +``` + +### 10.3 Setting Dynamic Variables Mid-Job + +```yaml +- name: Set computed variable + run: | + echo "DEPLOY_ENV=production" >> $GITHUB_ENV + echo "BUILD_TIME=$(date -u +%Y%m%d%H%M%S)" >> $GITHUB_ENV + +- name: Use the variable + run: echo "Deploying to $DEPLOY_ENV at $BUILD_TIME" + # Variables set via GITHUB_ENV are available to all subsequent steps in the job +``` + +--- + +## 11. Secrets and Environment Variables + +### 11.1 Secrets + +Secrets are encrypted values stored in repository or organization settings. They are never printed in logs. + +```yaml +steps: + - name: Deploy + env: + API_TOKEN: ${{ secrets.DEPLOY_TOKEN }} + DB_PASSWORD: ${{ secrets.DB_PASSWORD }} + run: ./deploy.sh +``` + +Secrets can be scoped at three levels: + +``` +Organization secrets → available to all repos in the org +Repository secrets → available to this repo only +Environment secrets → available only when deploying to a specific environment +``` + +**`GITHUB_TOKEN`** is a special secret automatically generated for every workflow run. It authenticates as the repository's GitHub App and expires when the workflow ends. No setup required. + +```yaml +- name: Comment on PR + uses: actions/github-script@v7 + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + script: | + github.rest.issues.createComment({ ... }) +``` + +### 11.2 Variable Scoping + +```yaml +env: + WORKFLOW_VAR: "visible to all jobs" # workflow-level + +jobs: + build: + env: + JOB_VAR: "visible to all steps in this job" # job-level + + steps: + - name: Deploy + env: + STEP_VAR: "visible only to this step" # step-level + run: echo "$WORKFLOW_VAR $JOB_VAR $STEP_VAR" +``` + +### 11.3 Repository Variables (non-secret) + +For non-sensitive configuration values, use GitHub repository variables (Settings → Secrets and variables → Variables): + +```yaml +run: echo "Deploying to ${{ vars.DEPLOY_HOST }}" +``` + +--- + +## 12. Artifacts and Data Passing Between Jobs + +Each job runs on a separate, isolated VM. To share files between jobs, upload them as artifacts from one job and download them in another. + +### 12.1 Upload an Artifact + +```yaml +jobs: + build: + runs-on: ubuntu-latest + steps: + - run: npm run build # produces ./dist/ + + - name: Upload build output + uses: actions/upload-artifact@v4 + with: + name: build-output # artifact name (used to download later) + path: dist/ # what to upload + retention-days: 1 # how long to keep it (default: 90) +``` + +### 12.2 Download an Artifact + +```yaml + deploy: + needs: build + runs-on: ubuntu-latest + steps: + - name: Download build output + uses: actions/download-artifact@v4 + with: + name: build-output # must match the upload name + path: dist/ # where to put it on this runner + + - run: ls dist/ # files from the build job are now here +``` + +### 12.3 Upload Test Reports + +```yaml + - name: Upload test results + if: always() # upload even if tests fail + uses: actions/upload-artifact@v4 + with: + name: test-results + path: | + coverage/ + junit.xml +``` + +--- + +## 13. Reusable Workflows and Composite Actions + +### 13.1 Reusable Workflows + +A workflow triggered by `workflow_call` can be invoked from another workflow. This is the primary way to avoid duplicating entire pipeline definitions. + +**Defining the reusable workflow** (`.github/workflows/deploy-shared.yml`): + +```yaml +on: + workflow_call: + inputs: + environment: + type: string + required: true + secrets: + deploy-token: + required: true + +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - run: echo "Deploying to ${{ inputs.environment }}" + env: + TOKEN: ${{ secrets.deploy-token }} +``` + +**Calling the reusable workflow:** + +```yaml +jobs: + deploy-staging: + uses: ./.github/workflows/deploy-shared.yml + with: + environment: staging + secrets: + deploy-token: ${{ secrets.STAGING_TOKEN }} + + deploy-production: + needs: deploy-staging + uses: ./.github/workflows/deploy-shared.yml + with: + environment: production + secrets: + deploy-token: ${{ secrets.PROD_TOKEN }} +``` + +### 13.2 Composite Actions + +A composite action bundles multiple steps into a single `uses:` reference. Unlike reusable workflows, composite actions run within the calling job (no separate VM). + +**Define it** (`.github/actions/setup-project/action.yml`): + +```yaml +name: Setup Project +description: Install Node.js and cache dependencies + +inputs: + node-version: + description: Node.js version + default: '20' + +runs: + using: composite + steps: + - uses: actions/setup-node@v4 + with: + node-version: ${{ inputs.node-version }} + cache: npm + - run: npm ci + shell: bash +``` + +**Use it in a workflow:** + +```yaml +steps: + - uses: actions/checkout@v4 + - uses: ./.github/actions/setup-project + with: + node-version: '20' + - run: npm test +``` + +--- + +## 14. Permissions + +By default, each workflow run is granted `GITHUB_TOKEN` with permissions scoped to the repository. You should always restrict permissions to the minimum required. + +```yaml +permissions: + contents: read # read files from the repo + packages: write # push to GHCR + pull-requests: write # comment on PRs + issues: write # create/modify issues + security-events: write # upload SARIF results to Security tab + id-token: write # request OIDC token (for keyless cloud auth) + actions: read # read workflow run info +``` + +Permissions can be set at workflow level (applies to all jobs) or per-job: + +```yaml +jobs: + scan: + permissions: + security-events: write # only this job gets this permission + contents: read + steps: ... +``` + +**To disable all permissions** (run with zero access): + +```yaml +permissions: {} +``` + +--- + +## 15. Concurrency Control + +By default, multiple runs of the same workflow can execute simultaneously. Use `concurrency` to limit this. + +```yaml +# Cancel any in-progress run for the same branch when a new push arrives +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true +``` + +```yaml +# Queue deployments — don't cancel, let each finish +concurrency: + group: deploy-${{ github.ref }} + cancel-in-progress: false +``` + +```yaml +# Per-job concurrency (protects a shared resource) +jobs: + deploy: + concurrency: + group: production-deploy + cancel-in-progress: false +``` + +--- + +## 16. Timeouts and Failure Strategies + +### 16.1 Timeout + +```yaml +jobs: + build: + timeout-minutes: 30 # job-level timeout (default: 360 min / 6 hours) + steps: + - name: Long operation + timeout-minutes: 10 # step-level timeout + run: ./long-script.sh +``` + +### 16.2 Continue on Error + +```yaml +steps: + - name: Optional lint check + continue-on-error: true # step failure won't fail the job + run: npm run lint + + - name: This always runs + run: echo "Even if lint failed" +``` + +At job level, use `if: always()` rather than `continue-on-error` to run cleanup steps unconditionally. + +### 16.3 Retry Logic + +GitHub Actions does not have built-in retry, but a common workaround: + +```yaml +- name: Flaky API call with retry + uses: nick-fields/retry@v3 + with: + timeout_minutes: 5 + max_attempts: 3 + command: ./scripts/call-api.sh +``` + +--- + +## 17. Caching Dependencies + +Caching re-uses files from previous runs, dramatically cutting install times. + +### 17.1 Built-in Cache via `setup-*` Actions + +The easiest approach — `actions/setup-node`, `setup-python`, `setup-java` etc. have a built-in `cache` option: + +```yaml +- uses: actions/setup-node@v4 + with: + node-version: '20' + cache: 'npm' # caches ~/.npm based on package-lock.json hash +``` + +### 17.2 Manual Cache with `actions/cache` + +```yaml +- name: Cache node_modules + uses: actions/cache@v4 + with: + path: node_modules + key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }} + # └── vary by OS └── vary by lock file contents + restore-keys: | + ${{ runner.os }}-node- # fallback: any cache for this OS +``` + +**Cache key strategy:** + +``` +Exact hit: ubuntu-node-abc123 → restore fully, skip install +Partial hit: ubuntu-node- → restore older cache, npm ci updates delta +Miss: (nothing) → full npm ci, save new cache at end +``` + +### 17.3 Docker Layer Cache + +```yaml +- uses: docker/setup-buildx-action@v3 + +- uses: docker/build-push-action@v5 + with: + context: . + push: true + cache-from: type=gha # read from GitHub Actions cache + cache-to: type=gha,mode=max # write layers to GitHub Actions cache +``` + +--- + +## 18. Complete Reference Pipeline + +This pipeline demonstrates all major concepts together: triggers, parallel jobs, dependencies, matrix, conditionals, artifacts, and notifications. + +```yaml +# .github/workflows/ci-complete.yml +name: Complete CI Pipeline + +on: + push: + branches: [main, develop] + tags: ['v*'] + pull_request: + branches: [main] + workflow_dispatch: + inputs: + skip-tests: + type: boolean + default: false + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +env: + NODE_ENV: test + REGISTRY: ghcr.io + +jobs: + # ── 1. LINT ───────────────────────────────────────────────────────────── + lint: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '20' + cache: npm + - run: npm ci + - run: npm run lint + + # ── 2. TEST (matrix over Node versions) ───────────────────────────────── + test: + runs-on: ubuntu-latest + if: ${{ !inputs.skip-tests }} + strategy: + fail-fast: false + matrix: + node: [18, 20, 22] + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: ${{ matrix.node }} + cache: npm + - run: npm ci + - run: npm test -- --coverage + - name: Upload coverage + if: matrix.node == 20 # only upload once + uses: actions/upload-artifact@v4 + with: + name: coverage + path: coverage/ + + # ── 3. BUILD IMAGE ─────────────────────────────────────────────────────── + docker: + needs: [lint, test] # wait for lint AND all test matrix cells + runs-on: ubuntu-latest + permissions: + contents: read + packages: write + outputs: + image-tag: ${{ steps.meta.outputs.tags }} + + steps: + - uses: actions/checkout@v4 + - uses: docker/setup-buildx-action@v3 + - uses: docker/login-action@v3 + with: + registry: ${{ env.REGISTRY }} + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + - id: meta + uses: docker/metadata-action@v5 + with: + images: ${{ env.REGISTRY }}/${{ github.repository }} + tags: | + type=ref,event=branch + type=semver,pattern={{version}} + type=sha,prefix=sha- + - uses: docker/build-push-action@v5 + with: + context: . + push: ${{ github.event_name != 'pull_request' }} + tags: ${{ steps.meta.outputs.tags }} + labels: ${{ steps.meta.outputs.labels }} + cache-from: type=gha + cache-to: type=gha,mode=max + + # ── 4. DEPLOY STAGING ──────────────────────────────────────────────────── + deploy-staging: + needs: docker + runs-on: ubuntu-latest + if: github.ref == 'refs/heads/develop' + environment: staging + steps: + - run: echo "Deploying ${{ needs.docker.outputs.image-tag }} to staging" + + # ── 5. DEPLOY PRODUCTION ───────────────────────────────────────────────── + deploy-production: + needs: docker + runs-on: ubuntu-latest + if: startsWith(github.ref, 'refs/tags/v') + environment: production + steps: + - run: echo "Deploying ${{ needs.docker.outputs.image-tag }} to production" + + # ── 6. NOTIFY ───────────────────────────────────────────────────────────── + notify: + needs: [lint, test, docker] + if: always() && github.event_name != 'pull_request' + runs-on: ubuntu-latest + steps: + - name: Notify Slack on failure + if: failure() + uses: slackapi/slack-github-action@v1.26.0 + with: + payload: | + { + "text": "❌ CI failed on `${{ github.ref_name }}` by ${{ github.actor }}\n${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" + } + env: + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} + SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK +``` + +--- + +## Quick Reference Card + +### Trigger → Branch Mapping + +| Goal | Trigger config | +|---|---| +| Run on every push | `on: push` | +| Run only on main | `on: push: branches: [main]` | +| Run on feature branches | `on: push: branches: ['feature/**']` | +| Run on version tags | `on: push: tags: ['v*']` | +| Run on PRs to main | `on: pull_request: branches: [main]` | +| Run on schedule | `on: schedule: - cron: '0 6 * * 1'` | +| Run manually | `on: workflow_dispatch` | + +### Job Execution Patterns + +| Pattern | Syntax | +|---|---| +| Parallel (default) | Define multiple jobs with no `needs` | +| Sequential A → B | `needs: a` on job B | +| Fan-out A → B, C, D | `needs: a` on B, C, and D each | +| Fan-in B, C, D → E | `needs: [b, c, d]` on job E | +| Loop over values | `strategy: matrix: values: [...]` | +| Skip on failure | `if: success()` (default) | +| Run only on failure | `if: failure()` | +| Always run | `if: always()` | + +### Useful Expressions Cheatsheet + +```yaml +github.ref == 'refs/heads/main' +startsWith(github.ref, 'refs/tags/v') +contains(github.event.head_commit.message, '[skip ci]') +github.event_name == 'pull_request' +github.actor == 'dependabot[bot]' +matrix.os == 'ubuntu-latest' +needs.build.result == 'success' +inputs.environment == 'production' +``` + +--- + +*This guide is part of the CI Learning Path. See the [main index](index.md) for the full table of contents.* From 67530089679422ff4aead1013dd7b5d30a04dea5 Mon Sep 17 00:00:00 2001 From: Ivan Buttinoni Date: Wed, 11 Mar 2026 15:36:21 +0100 Subject: [PATCH 5/5] Update code block syntax in markdown files --- Lessons/03-intermediate-ci.md | 17 +++++++++-------- Lessons/60-github-workflow-guide.md | 21 +++++++++++---------- 2 files changed, 20 insertions(+), 18 deletions(-) diff --git a/Lessons/03-intermediate-ci.md b/Lessons/03-intermediate-ci.md index 604720a..87dbb6d 100644 --- a/Lessons/03-intermediate-ci.md +++ b/Lessons/03-intermediate-ci.md @@ -6,7 +6,7 @@ Without Docker, a CI pipeline runs directly on the host runner — a shared machine with a pre-installed OS, language runtimes, and system libraries. This creates a class of problems that become more painful as a project grows: -``` +```text Without Docker: Runner OS: Ubuntu 22.04 @@ -33,7 +33,7 @@ Docker solves this by packaging the application **together with its exact enviro The end goal of the Docker section of a CI pipeline is to produce a **tagged, versioned image stored in a container registry** (in this guide, GHCR — GitHub Container Registry). Once the image is there, any downstream system — a deployment pipeline, a Kubernetes cluster, a developer's local machine — can pull and run it without needing the source code, build tools, or any knowledge of how it was built. -``` +```text Source code Container Registry (GHCR) ───────────── ────────────────────────── src/ @@ -55,7 +55,7 @@ Building an image takes time and requires build tools, source code, and network The `sha-a3f2c1d` tag is immutable — it always refers to the exact image built from that specific commit. Tags like `:latest` or `:main` are mutable and move with every push, making it impossible to know which version is actually running. In production deployments, always pin to the SHA tag. -``` +```text Mutable tag (risky in production): image: ghcr.io/org/repo:latest ← changes with every push, no audit trail @@ -67,7 +67,7 @@ Immutable tag (safe): A complete Docker-in-CI setup involves three files working together: -``` +```text Repository: ├── Dockerfile (1) defines the image build ├── .dockerignore (2) controls what enters the build context @@ -127,7 +127,7 @@ CMD ["node", "dist/index.js"] The workflow file and the Dockerfile are two separate files with distinct responsibilities that work together as a pipeline: -``` +```text Repository on disk: ├── Dockerfile ← defines HOW to build the image └── .github/ @@ -139,7 +139,7 @@ The Dockerfile is not referenced by name anywhere in the workflow YAML. Instead, **End-to-end flow from a push to a pullable image:** -``` +```text 1. Developer pushes to main (or creates a tag like v1.2.0) │ ▼ @@ -272,7 +272,8 @@ jobs: **Where the image lands in GHCR and how to pull it:** Once the workflow completes, the image is visible at: -``` + +```text https://github.com/ORG/REPO/pkgs/container/REPO ``` @@ -297,7 +298,7 @@ docker pull ghcr.io/org/repo:sha-a3f2c1d Docker builds images as a stack of layers. Each `COPY` and `RUN` instruction creates a new layer. When a layer changes, Docker invalidates that layer **and every layer above it** — forcing all subsequent instructions to re-run from scratch. Understanding this cascade is the key to fast, cache-efficient builds in CI. -``` +```text Layer stack (top = last instruction): [5] RUN npm run build ← rebuilt if layer 4 changes diff --git a/Lessons/60-github-workflow-guide.md b/Lessons/60-github-workflow-guide.md index 237a64e..888128b 100644 --- a/Lessons/60-github-workflow-guide.md +++ b/Lessons/60-github-workflow-guide.md @@ -32,7 +32,7 @@ A **workflow** is an automated process defined in a YAML file stored under `.github/workflows/` in your repository. GitHub Actions detects and runs every `.yml` file in that directory according to its trigger conditions. -``` +```text Repository └── .github/ └── workflows/ @@ -70,7 +70,7 @@ jobs: # (4) The work to be done — one or more named jobs **How the keys relate to each other:** -``` +```text Workflow │ ├── on → defines WHEN the workflow runs @@ -117,7 +117,7 @@ on: **Branch filter logic:** -``` +```text Push to main → workflow runs (matches 'main') Push to release/1.0 → workflow runs (matches 'release/**') Push to develop → workflow skipped (no matching rule) @@ -154,7 +154,7 @@ on: Cron format: `minute hour day-of-month month day-of-week` -``` +```text '0 6 * * 1' │ │ │ │ └── day of week: 1 = Monday (0=Sun, 7=Sun) │ │ │ └──── month: * = every month @@ -333,7 +333,7 @@ jobs: steps: ... ``` -``` +```text Time → lint ████████████ @@ -364,7 +364,7 @@ jobs: steps: ... ``` -``` +```text Time → build ████████ @@ -398,7 +398,7 @@ jobs: ... ``` -``` +```text Time → build ████████ @@ -564,6 +564,7 @@ jobs: ``` This creates 3 parallel jobs: + - `test (18)` - `test (20)` - `test (22)` @@ -579,7 +580,7 @@ strategy: This creates **3 × 2 = 6** parallel jobs, one for every combination: -``` +```text ubuntu-latest + node 18 ubuntu-latest + node 20 macos-latest + node 18 @@ -732,7 +733,7 @@ steps: Secrets can be scoped at three levels: -``` +```text Organization secrets → available to all repos in the org Repository secrets → available to this repo only Environment secrets → available only when deploying to a specific environment @@ -1052,7 +1053,7 @@ The easiest approach — `actions/setup-node`, `setup-python`, `setup-java` etc. **Cache key strategy:** -``` +```text Exact hit: ubuntu-node-abc123 → restore fully, skip install Partial hit: ubuntu-node- → restore older cache, npm ci updates delta Miss: (nothing) → full npm ci, save new cache at end