Skip to content

bootstrap_guide.md

Latest commit

 

History

History
436 lines (326 loc) · 14.6 KB

bootstrap_guide.md

File metadata and controls

436 lines (326 loc) · 14.6 KB

Bootstrap Guide

This guide explains how to bootstrap a complete GitOps environment using Cluster-Forge's five-step deployment model. The bootstrap process establishes ArgoCD, OpenBao, and Gitea as foundation components, then creates the cluster-forge Application which manages all remaining components via ArgoCD.

Prerequisites

  • Kubernetes cluster (1.33+ recommended, running and accessible via kubectl)
  • Tools installed:
    • kubectl with cluster-admin access
    • helm (3.0+)
    • openssl (for password generation)
    • yq (v4+)

Usage

./scripts/bootstrap.sh <domain> [--cluster-size=small|medium|large]

Arguments

  • domain (required): Cluster domain for all services (e.g., example.com, 192.168.1.100.nip.io)

Options

  • --apps=APP1,APP2: Deploy only specified components (default: applies to cluster)
    • Bootstrap apps: namespaces, argocd, openbao, gitea, cluster-forge
    • Child apps: Any app from enabledApps list (see values_{size}.yaml for app names)
    • Use with --template-only to render instead of applying
  • --cluster-size [small|medium|large]: Cluster size configuration (default: medium)
  • --template-only, -t: Output YAML manifests to stdout instead of applying to cluster
  • --target-revision, -r: cluster-forge git revision for ArgoCD to sync from
  • --skip-deps: Skip dependency checking (for advanced users)
  • --airm-image-repository=REPO: Custom AIRM container image repository for air-gapped deployments
  • --help, -h: Show usage information

Examples

# Basic usage with default medium cluster size
./scripts/bootstrap.sh 192.168.1.100.nip.io

# Large cluster
./scripts/bootstrap.sh example.com --cluster-size=large

# Deploy only specific components
./scripts/bootstrap.sh example.com --apps=openbao,openbao-init

# Render templates for debugging (doesn't apply)
./scripts/bootstrap.sh example.com --apps=gitea --template-only

# Deploy from specific git branch
./scripts/bootstrap.sh example.com --target-revision=feature-branch
./scripts/bootstrap.sh example.com --CLUSTER_SIZE=large

# Custom AIRM image repository
./scripts/bootstrap.sh example.com --airm-image-repository=ghcr.io/mycompany

# Air-gapped deployment with local registry
./scripts/bootstrap.sh 192.168.1.100.nip.io --cluster-size=small --airm-image-repository=harbor.internal.com/airm

How It Works

The bootstrap script uses a five-step deployment model:

Phase 1: Pre-Cleanup

  • The pre_cleanup function performs selective cleanup, only affects cf-gitea and cf-openbao namespaces
  • Detects previous installations by checking for completed gitea-init-job
  • Removes Gitea resources to enable fresh deployment
  • Deletes OpenBao initialization jobs and temporary files
  • Ensures clean state for new bootstrap

Phase 2: GitOps Foundation Bootstrap (Manual Helm Templates)

1. Configuration Preparation

  • Validates required domain argument
  • Validates cluster size (small, medium, or large)
  • Merges base values.yaml with size-specific overrides values_<size>.yaml
  • Sets global.domain and global.clusterSize in merged configuration

2. Namespace Creation Creates three namespaces for bootstrap components:

  • argocd - GitOps controller
  • cf-gitea - Git repository server
  • cf-openbao - Secrets management system

3. ArgoCD Bootstrap

  • Extracts ArgoCD values from merged configuration
  • Deploys ArgoCD using helm template with server-side apply
  • Uses --field-manager=argocd-controller to match ArgoCD's self-management
  • Waits for all ArgoCD components to be ready:
    • application-controller StatefulSet
    • applicationset-controller Deployment
    • redis Deployment
    • repo-server Deployment

4. Gitea Bootstrap

  • Generates random admin password using openssl rand -hex 16
  • Creates initial-cf-values ConfigMap with merged configuration
  • Creates gitea-admin-credentials secret
  • Extracts Gitea values from merged configuration
  • Deploys Gitea using helm template
  • Waits for Gitea deployment to be ready
  • Runs initialization job (gitea-init-job) which:
    • Creates admin API token
    • Creates cluster-org organization
    • Clones and pushes cluster-forge repository from initial-cf-values ConfigMap
    • Creates cluster-values repository with configuration

Phase 3: App-of-Apps Deployment (ArgoCD-Managed)

5. ClusterForge Application Deployment

  • Renders root helm chart with merged configuration
  • Creates cluster-forge Application resource in ArgoCD
  • ArgoCD syncs all remaining components in wave order:
    • Wave -70: OpenBao (secrets management)
    • Wave -60: OpenBao configuration
    • Wave -50: OpenBao initialization job
    • Wave -40: External Secrets, Cert-Manager
    • Wave -30 to 0: All other applications

Key Improvement: OpenBao is now managed by ArgoCD rather than bootstrapped separately, simplifying the bootstrap process while maintaining proper dependency ordering through sync waves.

  • When externalValues.enabled: true, uses multi-source feature:
    • Source 1: cluster-forge repo (root/ helm chart)
    • Source 2: cluster-values repo (custom values.yaml)
  • ArgoCD manages the complete application lifecycle
  • Proper dependency ordering ensures OpenBao is ready before applications that depend on secrets

7. Cleanup

  • Removes temporary merged values files from /tmp/

Cluster Configuration

Values Files Structure

ClusterForge uses a layered configuration approach with YAML merge semantics:

  1. Base values (root/values.yaml):

    • Contains all app definitions
    • Defines default configuration for all apps
    • Specifies enabledApps list (alpha-sorted)
    • Configured with:
      • clusterForge.repoUrl - Points to Gitea service URL (local mode) or GitHub (external mode)
      • clusterForge.targetRevision - Version/branch to deploy
      • externalValues.enabled: true - Enables dual-repository pattern
      • externalValues.repoUrl - Points to cluster-values repo in Gitea
      • global.domain - Set by bootstrap script
      • global.clusterSize - Set by bootstrap script

  2. Size-specific values (root/values_<size>.yaml):

    • Override base values for specific cluster sizes
    • Define resource limits and requests
    • Single node (small and medium) RWO local-path storage
    • Multinode (large) RWX storage
    • Modify replica counts and HA settings
    • Add size-specific enabled apps (e.g., kyverno-policies-storage-local-path for small/medium)
    • Available sizes: small, medium, large
    • Uses DRY principle - only contains differences from base

  3. External values (cluster-values/values.yaml in Gitea):

    • Created during bootstrap in the cluster-values repository
    • Contains cluster-specific overrides
    • Can be modified post-bootstrap for customizations
    • Structure: 
      clusterForge:
  repoURL: http://gitea-http.cf-gitea.svc:3000/cluster-org/cluster-forge.git
  path: root
  targetRevision: main

global:
  clusterSize: medium  # Set by --cluster-size flag
  domain: example.com  # Set by domain argument

# AIRM Image Repository Configuration (optional, only when AIRM_IMAGE_REPOSITORY is set)
airm-api:
  airm:
    backend:
      image:
        repository: ghcr.io/mycompany/airm-api
    frontend:
      image:
        repository: ghcr.io/mycompany/airm-ui

airm-dispatcher:
  airm:
    dispatcher:
      image:
        repository: ghcr.io/mycompany/airm-dispatcher

Value Merging Order

When ArgoCD renders the cluster-forge application, values are merged in this order (later values override earlier):

  1. Base values.yaml
  2. Size-specific values_<size>.yaml
  3. External cluster-values/values.yaml from Gitea

Cluster Sizes

Each cluster size is optimized for different resource constraints:

  • Small: Development/testing environments, minimal resources
  • Medium (default): Production-ready, balanced configuration
  • Large: High-availability, maximum performance

Size-specific configurations typically adjust:

  • Component replica counts (ArgoCD, PostgreSQL, etc.)
  • Resource limits and requests (CPU, memory)
  • Storage sizes (PVC, retention periods)
  • High-availability features (Redis HA, multiple replicas)

ClusterForge App-of-Apps Model

The bootstrap script creates the root cluster-forge Application in ArgoCD, which implements an app-of-apps pattern.

Application Structure

The cluster-forge Application is defined in root/templates/cluster-forge.yaml:

Child Applications

The root chart renders individual Application resources for each app listed in enabledApps using the template in root/templates/cluster-apps.yaml.

Each child application includes:

  • Namespace: Target namespace for the application
  • Path: Location of helm chart or manifests in sources/<path>
  • Values: Configuration from apps.<name>.valuesObject or valuesFile
  • Sync wave: Deployment order (lower numbers deploy first)
  • Sync policy: Automated with prune and self-heal enabled
  • Ignore differences: Optional resource-specific ignore rules

Example child application configuration in values:

apps:
  argocd:
    path: argocd/8.3.5
    namespace: argocd
    syncWave: -3
    valuesObject:
      # ArgoCD-specific values
    helmParameters:
      - name: global.domain
        value: "argocd.{{ .Values.global.domain }}"

Access to Main Components

  1. ArgoCD:

    # Initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

# Access URL (replace with your domain)
echo "https://argocd.${DOMAIN}"

  2. Gitea:

    # Admin username
kubectl -n cf-gitea get secret gitea-admin-credentials -o jsonpath="{.data.username}" | base64 -d

# Admin password
kubectl -n cf-gitea get secret gitea-admin-credentials -o jsonpath="{.data.password}" | base64 -d

# API token (created by init job)
kubectl -n cf-gitea get secret gitea-admin-token -o jsonpath="{.data.token}" | base64 -d

# Access URL (replace with your domain)
echo "https://gitea.${DOMAIN}"

  3. OpenBao:

    # Root token
kubectl -n cf-openbao get secret openbao-keys -o jsonpath='{.data.root_token}' | base64 -d

# Unseal keys (stored in openbao-keys secret)
kubectl -n cf-openbao get secret openbao-keys -o jsonpath='{.data.unseal_keys_b64}' | base64 -d

  4. Keycloak (deployed by ArgoCD):

    # Admin password
kubectl -n keycloak get secret keycloak-credentials -o jsonpath="{.data.KEYCLOAK_INITIAL_ADMIN_PASSWORD}" | base64 -d

# Dev user password
kubectl -n keycloak get secret airm-devuser-credentials -o jsonpath="{.data.KEYCLOAK_INITIAL_DEVUSER_PASSWORD}" | base64 -d

Troubleshooting

Bootstrap Fails at Gitea Init

If the Gitea initialization job fails during repository migration:

# Check job logs
kubectl logs -n cf-gitea job/gitea-init-job

# The job automatically retries migration up to 5 times
# If it continues failing, check Gitea pod logs
kubectl logs -n cf-gitea deploy/gitea -c gitea

OpenBao Init Job Fails

Production mode only

If OpenBao initialization fails:

# Check init job logs
kubectl logs -n cf-openbao job/openbao-init-job

# Verify OpenBao is running
kubectl get pods -n cf-openbao

# Re-run bootstrap (pre-cleanup will handle the retry)
./bootstrap.sh your-domain.com

ArgoCD Applications Not Syncing

If applications aren't deploying:

# Check cluster-forge app status
kubectl get application cluster-forge -n argocd -o yaml

# Check individual app status
kubectl get applications -n argocd

# View app details in ArgoCD UI
# https://argocd.your-domain.com

Merged Values Inspection

The bootstrap script creates temporary merged values at /tmp/merged_values.yaml for debugging. You can inspect this file during bootstrap to see the final merged configuration.

Post-Bootstrap Customization

Production Mode (with Gitea)

After bootstrap completes in production mode, you can customize the cluster by modifying the cluster-values repository in Gitea:

  1. Access Gitea at https://gitea.${DOMAIN}
  2. Navigate to cluster-org/cluster-values repository
  3. Edit values.yaml to add/override configuration
  4. Commit changes
  5. ArgoCD will automatically detect and apply changes

Example customizations in cluster-values/values.yaml:

# Override app-specific values
apps:
  keycloak:
    valuesObject:
      replicas: 2
      resources:
        requests:
          memory: "1Gi"

# Disable specific apps
enabledApps:
  - argocd
  - gitea
  # ... list only apps you want enabled

# Add custom global values
global:
  myCustomValue: "something"

Selective Component Deployment

The --apps flag allows you to deploy only specific components instead of the full stack. This is useful for:

  • Development workflows: Deploy only the components you're working on
  • Troubleshooting: Deploy components individually to isolate issues
  • Testing: Validate specific component configurations
  • Incremental deployment: Add components to an existing cluster

Bootstrap Components

These are the core infrastructure components deployed manually via helm template:

  • namespaces - Creates required namespaces (argocd, cf-gitea, cf-openbao)
  • argocd - GitOps controller for managing all other components
  • gitea - Self-hosted Git server for cluster-forge and cluster-values repositories
  • cluster-forge - ArgoCD parent application that manages all child apps

Cluster-Forge Child Apps

Any application listed in enabledApps from values.yaml can be deployed individually:

# Deploy only OpenBao components
./scripts/bootstrap.sh example.com --apps=openbao,openbao-init,openbao-config

# Deploy only monitoring stack
./scripts/bootstrap.sh example.com --apps=prometheus-crds,otel-lgtm-stack,opentelemetry-operator

# Deploy identity management
./scripts/bootstrap.sh example.com --apps=keycloak,cluster-auth,cluster-auth-config

Template-Only Mode

Combine with --template-only to render manifests without applying:

# Generate YAML for debugging
./scripts/bootstrap.sh example.com --apps=keycloak --template-only > keycloak-manifests.yaml

# View what would be deployed
./scripts/bootstrap.sh example.com --apps=openbao,openbao-init --template-only | kubectl diff -f -

File Cleanup

The bootstrap script automatically cleans up temporary files at the end:

  • /tmp/merged_values.yaml
  • /tmp/argocd_values.yaml
  • /tmp/argocd_size_values.yaml
  • /tmp/openbao_values.yaml
  • /tmp/openbao_size_values.yaml
  • /tmp/gitea_values.yaml
  • /tmp/gitea_size_values.yaml