✨ feat(upstream): remote Docker TCP+TLS upstreams with active/passive failover

CHANGELOG.md

@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - **`SecurityOpt` SELinux and system-paths directives are now policy-evaluable.** Three opt-in `request_body.container_create` knobs (all default off — zero behavior change): `deny_selinux_disable` denies `label=disable` and the legacy `label:disable` colon form (which turn off SELinux confinement); `deny_selinux_label_override` denies `label=user:`/`role:`/`type:`/`level:` SELinux context customization; `deny_unconfined_system_paths` denies `systempaths=unconfined` **and** requests that set `MaskedPaths`/`ReadonlyPaths` to an explicit empty array — the Docker CLI translates `--security-opt systempaths=unconfined` into `MaskedPaths: []` client-side, so direct API callers could otherwise clear the masked-path protections without ever sending the SecurityOpt string. Both vectors are covered.
 - **Swarm services gained seccomp/AppArmor confinement-mode rails**, completing `ContainerSpec.Privileges` parity with container-create. Three opt-in `request_body.service` knobs (all default off): `deny_unconfined_seccomp` (denies `Privileges.Seccomp.Mode: "unconfined"`), `deny_custom_seccomp_profiles` (denies `Mode: "custom"`, and fail-closed denies a `Seccomp` object carrying a `Profile` blob with no `Mode` — an inline profile the proxy cannot vet can encode an allow-everything policy), and `deny_unconfined_apparmor` (denies `Privileges.AppArmor.Mode: "disabled"`, swarm's equivalent of unconfined).
+- **Remote Docker TCP+TLS upstreams with active/passive failover (`upstream.endpoints[]`).** Sockguard can now dial a remote Docker daemon over standard Docker mTLS — or any mix of local `unix://`/bare-path sockets and remote `tcp://host:port` endpoints — with per-endpoint TLS config (`tls.ca_file`/`cert_file`/`key_file`/`server_name`) and insecure opt-ins (`insecure_allow_plain_tcp`, `insecure_skip_tls_verify`). Requests route to the first healthy endpoint in the ordered list; a connect or request failure instantly demotes that endpoint so the next request fails over without retry (Docker writes aren't idempotent). Active connect-level health probes run on a configurable `failover.health_interval`/`health_timeout` schedule, keeping the hot path aware of endpoint state between requests. TLS negotiation lives inside the dialer, so it works transparently across the reverse proxy, exec/attach hijack, and all inspect side-channel paths — the rest of the proxy stack is unaware of whether the upstream is local or remote. The intended topology is active/passive redundancy across equivalent daemons (a swarm VIP + managers, an HA pair) — all endpoints must address the same logical daemon so daemon-local state (container IDs, exec sessions, owner labels) stays consistent. `DOCKER_HOST`/`DOCKER_TLS_VERIFY`/`DOCKER_CERT_PATH` are auto-detected as a single endpoint when no explicit `endpoints` are configured, and `endpoints`/`failover` are reload-immutable while `request_timeout` stays mutable. Legacy `upstream.socket` continues to work as the default single-local-socket path.
 - **Three bundled presets for the Portwing Docker agent and drydock self-update (12 → 15 presets).** `portwing.yaml` covers container lifecycle, image pull/remove, `GET /containers/*/logs` streaming, and event/network/volume/Swarm-service reads with exec denied; `portwing-with-exec.yaml` adds interactive exec (`/containers/*/exec`, `/exec/*/start`, `/exec/*/resize`, `/exec/*/json`). Both Portwing presets disable response redaction so container inspect data forwards intact through the tri-tool topology (sockguard → Portwing → drydock) and set `insecure_allow_read_exfiltration: true` for the logs path. `drydock-with-selfupdate.yaml` extends the drydock preset with the exec paths drydock's self-update finalize callback needs, pinned to the finalize entrypoint argv via `allowed_commands`. A ready-to-run `examples/compose/portwing/` stack ships alongside.
 
 ### Fixed

README.md

@@ -299,7 +299,7 @@ How we stack up against other Docker socket proxies:
 | Per-client admission / policy selection | ❌ | ❌ | Partial (IP/hostname + per-container labels) | ❌ | ❌ | ✅ (CIDR + labels + cert selectors incl. SPKI + unix peer profiles) |
 | Read-side visibility / redaction | ❌ | ❌ | ❌ | Partial (blocks 7 risky GETs) | ❌ | ✅ (visibility + protected JSON redaction) |
 | Remote TCP mTLS (listener) | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (TLS 1.3) |
-| Remote daemon upstream (TLS) | ❌ | ❌ | ❌ | ❌ | ✅ | Roadmap (v1.3) |
+| Remote daemon upstream (TLS) | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ (failover) |
 | Structured access logs | ❌ | ❌ | ✅ (JSON option) | ❌ | ❌ | ✅ (request + trace correlation) |
 | Dedicated audit log schema | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (JSON schema + reason codes) |
 | Rate limits / concurrency caps | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ (per-profile token-bucket + global priority gate) |
@@ -309,7 +309,7 @@ How we stack up against other Docker socket proxies:
 | YAML config | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
 | Tecnativa env compat | N/A | ✅ | ❌ | ❌ | ❌ | ✅ |
 
-`11notes/docker-socket-proxy` takes a deliberately narrow stance: a fixed read-only proxy that allows every Docker API `GET` except seven exfiltration-prone endpoints (container `attach/ws`, `export`, `archive`, `secrets`/`configs` listing, `swarm/unlockkey`, `images/{name}/get`) and blocks all writes, shipped as a non-root distroless image — we match its read-side blocking with finer-grained per-field redaction and visibility rules, but additionally allow scoped writes instead of refusing them outright. `hectorm/cetusguard` is the closest in spirit to us: a zero-dependency, default-deny proxy with method + regex path rules and mTLS on both the frontend and backend — but it has no request-body inspection, no per-client policies, no owner isolation, no read-side filtering, no metrics, and no hot-reload. Where we go further is body inspection breadth (every body-bearing Docker write path we can safely constrain), named profiles, ownership isolation, and read-side visibility/redaction. CetusGuard, in turn, can dial a remote Docker daemon over backend TLS today — our upstream is the local socket, with remote TCP upstreams on the v1.3 roadmap.
+`11notes/docker-socket-proxy` takes a deliberately narrow stance: a fixed read-only proxy that allows every Docker API `GET` except seven exfiltration-prone endpoints (container `attach/ws`, `export`, `archive`, `secrets`/`configs` listing, `swarm/unlockkey`, `images/{name}/get`) and blocks all writes, shipped as a non-root distroless image — we match its read-side blocking with finer-grained per-field redaction and visibility rules, but additionally allow scoped writes instead of refusing them outright. `hectorm/cetusguard` is the closest in spirit to us: a zero-dependency, default-deny proxy with method + regex path rules and mTLS on both the frontend and backend — but it has no request-body inspection, no per-client policies, no owner isolation, no read-side filtering, no metrics, and no hot-reload. Where we go further is body inspection breadth (every body-bearing Docker write path we can safely constrain), named profiles, ownership isolation, and read-side visibility/redaction. CetusGuard can dial a remote Docker daemon over backend TLS, and sockguard now does too — remote `tcp://host:port` endpoints with per-endpoint mTLS, configured under `upstream.endpoints[]`. We go further with health-checked active/passive failover across redundant endpoints (a swarm VIP, an HA pair), which CetusGuard does not have.
 
 </details>
 
@@ -467,6 +467,13 @@ LinuxServer's socket-proxy env surface is already Tecnativa-compatible for the b
 | **Observability** | Prometheus `/metrics`, dedicated audit schema, trusted request IDs, deny-reason enums, W3C trace/log correlation, active upstream socket watchdog, lock-free hot path |
 | **Dynamic policy** | `POST /admin/validate` CI gate, `fsnotify` + SIGHUP hot reload with immutable-field gate, monotonic policy versioning, optional dedicated admin listener, cosign-signed policy bundles |
 
+### Shipping in v1.4
+
+| Track | Surface |
+|---|---|
+| **Remote upstreams & failover** | `upstream.endpoints[]` — ordered failover set of Docker daemons (`unix://` or `tcp://host:port`), per-endpoint mTLS (`tls.ca_file`/`cert_file`/`key_file`/`server_name`), per-endpoint insecure opt-ins; active connect-level health probes on configurable `failover.health_interval`/`health_timeout`; request-failure demotes the active endpoint for immediate failover; TLS inside the dialer so the reverse proxy, hijack, and inspect paths are all covered; designed for active/passive redundancy across equivalent daemons (swarm managers, HA pairs) — not cross-daemon fan-out; `DOCKER_HOST`/`DOCKER_TLS_VERIFY`/`DOCKER_CERT_PATH` auto-detected when no endpoints are set |
+| **SecurityOpt policy rails** | `deny_selinux_disable`, `deny_selinux_label_override`, `deny_unconfined_system_paths` for `containers/create`; `deny_unconfined_seccomp`, `deny_custom_seccomp_profiles`, `deny_unconfined_apparmor` for `services/create/update`; swarm `ContainerSpec.Privileges` confinement parity with container create |
+
 ### Post-1.0 preview
 
 | Tier | Theme |
@@ -475,7 +482,6 @@ LinuxServer's socket-proxy env surface is already Tecnativa-compatible for the b
 | Policy refinement (v1.x) | Multiple frontend listeners on the main proxy, named rule path aliases |
 | Internals (v1.x) | Code-review backlog: collapse the config → filter-options → policy translation layers behind a single source of truth (generated Viper defaults); profiling-gated JSON redaction fast path |
 | Compliance (v1.x) | CIS Docker Benchmark control mapping, audit-ready policy templates |
-| Multi-host (v1.3) | Remote Docker TCP upstreams, multi-upstream fan-out, remote daemon health checking, connection pooling, automatic failover |
 | Extensibility (v1.x+) | Optional plugin extension points (WASM or Go plugins), OPA/Rego policy integration |
 
 </details>

app/internal/clientacl/middleware.go

@@ -219,6 +219,13 @@ func Middleware(upstreamSocket string, logger *slog.Logger, opts Options) func(h
 	return middlewareWithDeps(logger, opts, newACLResolveClient(upstreamSocket, resolvedLabelPrefix(opts)))
 }
 
+// MiddlewareWithRoundTripper is Middleware over the shared upstream RoundTripper
+// (typically an *upstream.Resolver) so container-label ACL resolution follows
+// the same active endpoint as the proxied request under failover.
+func MiddlewareWithRoundTripper(rt http.RoundTripper, logger *slog.Logger, opts Options) func(http.Handler) http.Handler {
+	return middlewareWithDeps(logger, opts, newACLResolveClientForClient(dockerclient.NewWithRoundTripper(rt), resolvedLabelPrefix(opts)))
+}
+
 // resolvedLabelPrefix replicates the label-prefix resolution from
 // compileOptions so newACLResolveClient can pre-bind a compile hook on the
 // cache without standing up the full compiled options pipeline first.
@@ -374,8 +381,12 @@ func resolveLabelACLRules(client resolvedClient, labelPrefix string) ([]*filter.
 }
 
 func newACLResolveClient(upstreamSocket, labelPrefix string) func(context.Context, netip.Addr) (resolvedClient, bool, error) {
+	return newACLResolveClientForClient(dockerclient.New(upstreamSocket), labelPrefix)
+}
+
+func newACLResolveClientForClient(client *http.Client, labelPrefix string) func(context.Context, netip.Addr) (resolvedClient, bool, error) {
 	resolver := upstreamResolver{
-		client: dockerclient.New(upstreamSocket),
+		client: client,
 	}
 	cache := newClientCache(clientCacheTTL, clientCacheMaxSize, time.Now, resolver.resolveClient)
 	if labelPrefix != "" {

app/internal/cmd/coverage_gaps_test.go

@@ -117,7 +117,7 @@ func TestBuildServeClientProfiles_Error(t *testing.T) {
 		},
 	}
 
-	_, err := buildServeClientProfiles(&cfg)
+	_, err := buildServeClientProfiles(&cfg, nil)
 	if err == nil {
 		t.Fatal("expected buildServeClientProfiles() to fail")
 	}