Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions infra/docker/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,11 @@ COMPOSE_FILE=compose.yml
COMPOSE_PATH_SEPARATOR=:

# ── Container images ────────────────────────────────────────────────
# Backend / frontend default to locally-built holdfast-* images. Leave
# unset (or set to empty) to let compose use the in-tree build.
# BACKEND_IMAGE_NAME=
# FRONTEND_IMAGE_NAME=
# Backend image defaults to a locally-built holdfast-backend-dotnet:latest.
# Leave unset (or set to empty) to let compose use the in-tree build.
# HOL-20: the frontend nginx image was retired — the SPA bundle now ships
# inside the backend image at /app/wwwroot and is served by Kestrel.
# BACKEND_DOTNET_IMAGE_NAME=

# Infra dependencies — pinned for reproducibility.
CLICKHOUSE_IMAGE_NAME=clickhouse/clickhouse-server:24.3.15.72-alpine
Expand Down Expand Up @@ -45,9 +46,14 @@ PSQL_USER=postgres
# Removed in HOL-22 — backend uses IMemoryCache + Postgres instead.

# ── Frontend / API URLs ─────────────────────────────────────────────
# HOL-20: backend serves the SPA bundle, so all URLs collapse to :8082.
# These values are baked into the SPA bundle at backend image build time
# (see ARG REACT_APP_* in backend-dotnet.Dockerfile). To deploy behind a
# reverse proxy / custom domain, override the build args at image build
# time and rebuild the backend image — runtime substitution was dropped.
REACT_APP_DISABLE_ANALYTICS=false
REACT_APP_FRONTEND_ORG=1
REACT_APP_FRONTEND_URI=http://localhost:3000
REACT_APP_FRONTEND_URI=http://localhost:8082
REACT_APP_IN_DOCKER=true
REACT_APP_OTLP_ENDPOINT=http://localhost:8082/otel
REACT_APP_PRIVATE_GRAPH_URI=http://localhost:8082/private
Expand Down
55 changes: 25 additions & 30 deletions infra/docker/README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
# HoldFast — Local Docker Stack

Quickstart for running HoldFast end-to-end on a single machine. Works on Linux,
macOS, and Windows (via Docker Desktop). The stack runs as 8 services in a
single `docker compose` project: backend, frontend, collector,
postgres, clickhouse, kafka, zookeeper, redis.
macOS, and Windows (via Docker Desktop). The hobby stack runs as **3 services**
in a single `docker compose` project: `backend`, `postgres`, `clickhouse`.

(See [HOL-17](https://yt.brewingcoder.com/issue/HOL-17) for the in-flight
work to reduce this further.)
The backend image bundles the SPA frontend, embeds an OTLP receiver, and uses
an in-process message bus, so kafka / zookeeper / redis / collector / nginx
have all been retired (see [HOL-17](https://yt.brewingcoder.com/issue/HOL-17)
and its subtasks).

> Production deployments should use a Helm chart (TODO) or a managed-service
> compose. This compose file is for development and demos only — secrets are
Expand All @@ -15,8 +16,8 @@ work to reduce this further.)
## Prerequisites

- Docker 24+ with Docker Compose v2
- ~6 GB free RAM (kafka + clickhouse + jvm overhead)
- Ports free on the host: 3000, 5432, 6379, 8082, 8123, 8889, 9000, 9092, 4317-4319
- ~3 GB free RAM (clickhouse is the dominant consumer; tuned by HOL-18)
- Ports free on the host: 5432 (postgres), 8082 (backend + UI), 8123 / 9000 (clickhouse)

If you're on Windows, also ensure `*.sh` files in your checkout have LF line
endings. The `.gitattributes` rule should handle this for fresh clones; if
Expand All @@ -30,8 +31,9 @@ cp .env.example .env # copy the template — .env itself is gitignored
docker compose -f compose.yml -f compose.hobby-dotnet.yml up -d
```

The first build takes 10–15 minutes (frontend has the largest layer). Subsequent
runs reuse cached layers and complete in seconds.
The first build takes 10–15 minutes (the frontend bundle stage inside
`backend-dotnet.Dockerfile` has the largest layer). Subsequent runs reuse
cached layers and complete in seconds.

## What auto-runs on startup

Expand All @@ -42,22 +44,18 @@ A handful of bootstrap services run inside the backend on first boot:
idempotently. Tracks applied versions in `default.schema_migrations`. Set
`ClickHouse__Migrations__Disabled=true` to skip (for environments where
the schema is managed externally).
3. **`KafkaTopicBootstrapService`** — creates the topics consumers will subscribe to
(`session-events`, `backend-errors`, `frontend-errors`, `metrics`, `logs`,
`traces`). Idempotent. Disable via `Kafka__TopicBootstrap__Disabled=true`.
4. **`DevSeedService`** — creates an admin user and four demo workspaces with
3. **`DevSeedService`** — creates an admin user and four demo workspaces with
one default project each. **Hobby/dev only** — production sets
`DevSeed__Enabled=false`.

After ~30 seconds you should see in `docker compose logs backend`:
- `ClickHouse migrations: 146 applied, 146 total`
- `Kafka topic bootstrap: created N of 6 topics`
- `DevSeed: complete — admin=dev@holdfast.local, workspaces=4`

## Logging in

```
URL: http://localhost:3000
URL: http://localhost:8082
Email: dev@holdfast.local
Password: $ADMIN_PASSWORD from .env (default "password")
```
Expand Down Expand Up @@ -108,15 +106,10 @@ Smoke test passed. The ingest pipeline is working end-to-end.
warmup. Once `Application started.` appears in the backend logs, it should
stay healthy. If it crashes after, check `docker compose logs backend |
grep fail`.
- **Port 8888 collision**. The collector's Prometheus self-metrics endpoint is
remapped to host port 8889 to dodge collisions with Jupyter, etc. (see
HOL-7). The container internally still uses 8888.
- **`Table default.X does not exist`** in worker logs after a fresh `up`.
The migration runner may have failed. Look for `ClickHouse migrations:`
lines in the backend logs. If you see `Hosting failed to start`, drop the
ClickHouse data volume and try again: `docker compose down -v`.
- **Kafka consumers crashing on subscribe**. Topics aren't created. Check for
`Kafka topic bootstrap:` lines in the backend logs.

## Stopping and restarting

Expand All @@ -133,14 +126,13 @@ docker compose -f compose.yml -f compose.hobby-dotnet.yml up -d --force-recreate

## Compose file layering

- `compose.yml` — base infra (postgres, clickhouse, kafka, redis, etc.).
Always required.
- `compose.hobby-dotnet.yml` — adds the .NET backend, frontend, and collector
with hobby/dev defaults (DevSeed on, SSL off, plaintext password).
- `compose.hobby.yml` — adds the **legacy Go** backend instead of .NET. Pick
one or the other, not both. Default for new deployments is `hobby-dotnet`.
- `compose.dev-frontend.yml` — overlay for frontend hot-reload during local
development. See `frontend.Dockerfile` and PR #66 for context.
- `compose.yml` — base infra (postgres, clickhouse). Always required.
- `compose.hobby-dotnet.yml` — adds the .NET backend (which now bundles the
SPA + OTLP receiver) with hobby/dev defaults (DevSeed on, SSL off, plaintext
password).
- `compose.hobby.yml` — adds the **legacy Go** backend + a separate nginx
frontend container. Pick one or the other, not both. Default for new
deployments is `hobby-dotnet`.
- `compose.enterprise*.yml` — production-shaped variants. Out of scope for
this README.

Expand All @@ -153,7 +145,10 @@ The full list is in `.env.example`. The most commonly tweaked ones:
| `ADMIN_PASSWORD` | `password` | Auth password for the seeded admin user |
| `DEV_SEED_ENABLED` | `true` | Auto-create demo workspaces + projects |
| `CLICKHOUSE_MIGRATIONS_DISABLED` | `false` | Skip the .NET-side migration runner |
| `KAFKA_AUTO_CREATE_TOPICS_ENABLE` | `true` | Kafka broker setting (Confluent default off) |
| `REACT_APP_FRONTEND_URI` | `http://localhost:3000` | Where the frontend is reachable |
| `REACT_APP_FRONTEND_URI` | `http://localhost:8082` | Where the dashboard is reachable |
| `REACT_APP_PRIVATE_GRAPH_URI` | `http://localhost:8082/private` | Dashboard GraphQL URL |
| `REACT_APP_PUBLIC_GRAPH_URI` | `http://localhost:8082/public` | SDK ingest URL |

The `REACT_APP_*` URLs are baked into the SPA at backend image build time
(see `backend-dotnet.Dockerfile`). Override the build args and rebuild to
deploy behind a custom domain.
85 changes: 81 additions & 4 deletions infra/docker/backend-dotnet.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,79 @@
# ── Build stage ────────────────────────────────────────────────────────
FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:10.0 AS build
# ── Frontend pruner ───────────────────────────────────────────────────
# HOL-20: the dedicated frontend nginx container was folded into the
# backend image. The frontend build stages below mirror what the now-removed
# frontend.Dockerfile did; the runtime stage at the bottom copies the
# resulting bundle into /app/wwwroot for Kestrel to serve.
FROM --platform=$BUILDPLATFORM node:lts-alpine AS frontend-pruner

RUN apk add --no-cache libc6-compat && npm install -g turbo@^2

WORKDIR /app
COPY . .
RUN turbo prune @holdfast-io/frontend --docker

# ── Frontend build ────────────────────────────────────────────────────
FROM --platform=$BUILDPLATFORM node:lts-alpine AS frontend-build

RUN apk update && apk add --no-cache build-base python3

WORKDIR /app

COPY .yarnrc.yml .
COPY .yarn/patches ./.yarn/patches
COPY .yarn/releases ./.yarn/releases

# Install only pruned workspace deps. rrweb/package.json is added explicitly
# because turbo prune doesn't pick it up (its devDeps are imported by
# rrweb/vite.config.default.ts which every rrweb package uses).
COPY --from=frontend-pruner /app/out/json/ .
COPY rrweb/package.json ./rrweb/package.json
# Use the full yarn.lock so --immutable works for rrweb's deps.
COPY yarn.lock ./yarn.lock

RUN --mount=type=cache,target=/root/.yarn/berry/cache,sharing=locked \
yarn install

COPY --from=frontend-pruner /app/out/full/ .

# Root config files turbo prune omits.
COPY rrweb/tsconfig.base.json ./rrweb/tsconfig.base.json
COPY rrweb/tsconfig.json ./rrweb/tsconfig.json
COPY rrweb/vite.config.default.ts ./rrweb/vite.config.default.ts
COPY rrweb/turbo.json ./rrweb/turbo.json
COPY tsconfig.json ./tsconfig.json
COPY graphql.config.js ./graphql.config.js

# GraphQL schemas live outside the frontend workspace; needed for codegen.
COPY src/backend/localhostssl ./src/backend/localhostssl
COPY src/backend/private-graph ./src/backend/private-graph
COPY src/backend/public-graph ./src/backend/public-graph

# Bake URLs at build time. Defaults match the lean single-port deploy where
# the backend serves both API and frontend on :8082. Override via build args
# (e.g. for production deployments behind a custom domain). The
# frontend-entrypoint.py runtime substitution from the old frontend.Dockerfile
# was dropped — for self-hosted single-tenant deploys, rebuilding on URL
# changes is straightforward and avoids the runtime mutation step.
ARG NODE_OPTIONS="--max-old-space-size=8192"
ARG REACT_APP_AUTH_MODE=Password
ARG REACT_APP_FRONTEND_URI=http://localhost:8082
ARG REACT_APP_PRIVATE_GRAPH_URI=http://localhost:8082/private
ARG REACT_APP_PUBLIC_GRAPH_URI=http://localhost:8082/public
ARG REACT_APP_OTLP_ENDPOINT=http://localhost:8082/otel
ARG REACT_APP_IN_DOCKER=true

ENV REACT_APP_AUTH_MODE=$REACT_APP_AUTH_MODE
ENV REACT_APP_FRONTEND_URI=$REACT_APP_FRONTEND_URI
ENV REACT_APP_PRIVATE_GRAPH_URI=$REACT_APP_PRIVATE_GRAPH_URI
ENV REACT_APP_PUBLIC_GRAPH_URI=$REACT_APP_PUBLIC_GRAPH_URI
ENV REACT_APP_OTLP_ENDPOINT=$REACT_APP_OTLP_ENDPOINT
ENV REACT_APP_IN_DOCKER=$REACT_APP_IN_DOCKER

RUN --mount=type=cache,target=/root/.turbo,sharing=locked \
TURBO_CACHE_DIR=/root/.turbo npx turbo run build:fast --filter=@holdfast-io/frontend...

# ── Backend build ─────────────────────────────────────────────────────
FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:10.0 AS backend-build

WORKDIR /src

Expand Down Expand Up @@ -32,13 +106,16 @@ ARG REACT_APP_COMMIT_SHA
ENV REACT_APP_COMMIT_SHA=$REACT_APP_COMMIT_SHA

LABEL org.opencontainers.image.source=https://github.com/BrewingCoder/holdfast
LABEL org.opencontainers.image.description="HoldFast .NET Backend"
LABEL org.opencontainers.image.description="HoldFast .NET Backend (with frontend bundle)"
LABEL org.opencontainers.image.licenses="AGPL-3.0"

RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY --from=build /app .
COPY --from=backend-build /app .

# Frontend SPA bundle — served by Kestrel via UseStaticFiles + MapFallbackToFile.
COPY --from=frontend-build /app/src/frontend/build /app/wwwroot

# ClickHouse migration files — applied at startup by ClickHouseMigrationService.
# Disable via ClickHouse__Migrations__Disabled=true when the schema is managed
Expand Down
36 changes: 0 additions & 36 deletions infra/docker/compose.dev-frontend.yml

This file was deleted.

25 changes: 6 additions & 19 deletions infra/docker/compose.hobby-dotnet.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ services:
- ClickHouse__Migrations__Disabled=${CLICKHOUSE_MIGRATIONS_DISABLED:-false}
- Storage__Type=filesystem
- Storage__FilesystemRoot=/highlight-data
- Frontend__Uri=${REACT_APP_FRONTEND_URI:-http://localhost:3000}
- Frontend__Uri=${REACT_APP_FRONTEND_URI:-http://localhost:8082}
- Auth__Mode=${REACT_APP_AUTH_MODE:-Password}
- Auth__AdminPassword=${ADMIN_PASSWORD:-}
- DevSeed__Enabled=${DEV_SEED_ENABLED:-true}
Expand All @@ -40,24 +40,11 @@ services:
# Collector removed in HOL-21 — backend has its own OTLP receiver at
# /otel/v1/{logs,traces,metrics}. SDKs should target the backend's
# OTLP endpoint (default http://backend:8082/otel).

frontend:
container_name: frontend
build:
context: ../..
dockerfile: infra/docker/frontend.Dockerfile
image: ${FRONTEND_IMAGE_NAME:-holdfast-frontend:latest}
restart: on-failure
ports:
- '0.0.0.0:3000:3000'
- '0.0.0.0:8080:8080'
environment:
- SSL=false
- REACT_APP_AUTH_MODE=${REACT_APP_AUTH_MODE:-Password}
- REACT_APP_FRONTEND_URI=${REACT_APP_FRONTEND_URI:-http://localhost:3000}
- REACT_APP_PRIVATE_GRAPH_URI=${REACT_APP_PRIVATE_GRAPH_URI:-http://localhost:8082/private}
- REACT_APP_PUBLIC_GRAPH_URI=${REACT_APP_PUBLIC_GRAPH_URI:-http://localhost:8082/public}
- REACT_APP_OTLP_ENDPOINT=${REACT_APP_OTLP_ENDPOINT:-http://localhost:8082/otel}
#
# HOL-20: dedicated frontend nginx container removed. The backend image
# now bundles the SPA in /app/wwwroot and serves it via Kestrel on the
# same port (:8082). Frontend URLs are baked at backend image build time
# — set REACT_APP_* build args on the backend service to customize.

volumes:
highlight-data:
58 changes: 0 additions & 58 deletions infra/docker/frontend-entrypoint.py

This file was deleted.

Loading
Loading