exec & log session improvement, and console version bump

.env.example

@@ -6,7 +6,7 @@ NODE_ENV=development
 
 # EdgeOps Console static embed (npm run build:console → dev/console/build)
 EDGEOPS_CONSOLE_PATH=dev/console/build  # must be absolute path
-EDGEOPS_CONSOLE_VERSION=v1.0.2
+EDGEOPS_CONSOLE_VERSION=v1.0.3
 # EDGEOPS_CONSOLE_REPO=https://github.com/Datasance/edgeops-console
 # EDGEOPS_CONSOLE_FLAVOR=datasance
 

.github/actions/set-build-env/action.yml

@@ -8,7 +8,7 @@ runs:
       shell: bash
       run: |
         VERSION="${{ env.EDGEOPS_CONSOLE_VERSION }}"
-        if [ -z "$VERSION" ]; then VERSION="1.0.2"; fi
+        if [ -z "$VERSION" ]; then VERSION="1.0.3"; fi
         echo "EDGEOPS_CONSOLE_VERSION=$VERSION" >> "${GITHUB_ENV}"
 
         REPO="${{ env.EDGEOPS_CONSOLE_REPO }}"

CHANGELOG.md

@@ -16,7 +16,7 @@ Controller v3.8 is a **greenfield** release aligned with **Edgelet**. There is *
 - Agent status: removed **`processedMessages`**, **`messageSpeed`**; added **`availableRuntimes`**, optional **`runtimeAgentPhase`**, **`controlPlaneQuiesced`**.
 - Default container registry: **`docker.io`** (was `registry.hub.docker.com`).
 - Reserved ports: **54321**, **54322**, **53**.
-- New field-agent endpoint: **`POST /api/v3/agent/controller/register`** (system fogs only; Edgelet rc.1+).
+- New field-agent endpoint: **`POST /api/v3/agent/controller/register`** (system fogs only; Edgelet rc.1+). Register body accepts full container workload fields (`cmd`, `isPrivileged`, `healthCheck`, `capAdd`/`capDrop`, `extraHosts`, resources, etc.) with the same semantics as user microservice deploy; excludes `serviceAccount`, `natsConfig`, and ownership fields.
 
 #### API — architectures and applications
 
@@ -42,7 +42,7 @@ Controller v3.8 is a **greenfield** release aligned with **Edgelet**. There is *
 #### WebSocket exec — multi-session
 
 - **Microservice exec REST removed** — `POST/DELETE /api/v3/microservices/:uuid/exec` and `…/system/:uuid/exec` no longer exist. Open exec with **direct WebSocket** only: `WS /api/v3/microservices/exec/:uuid` (or `…/system/exec/:uuid`).
-- **3 concurrent exec sessions** per microservice (was 1 user exec WS per MS in Plan 16).
+- **3 concurrent exec sessions** per microservice (was 1 user exec WS per MS).
 - **Per-session lifecycle** — closing one exec session deletes only that session row only (no microservice-level exec flag).
 - **`execEnabled` removed** — dropped `microservices.exec_enabled` column and agent MS list field; exec attach is poll-driven only (`GET /agent/exec/sessions`).
 - **Agent exec discovery** — new `GET /api/v3/agent/exec/sessions` when change tracking reports `execSessions: true`.
@@ -108,11 +108,15 @@ Controller v3.8 is a **greenfield** release aligned with **Edgelet**. There is *
 - **K8s control plane:** hub **`iofog-router`** ConfigMap patches serialized via DB lock; K8s Service create/update/delete with LoadBalancer watch timeout.
 - **`service-bridge-config.js`** — full recompute of service-derived TCP bridge config per fog on reconcile (preserves router base config).
 - **SQLite single-node production hardening** — WAL + `busy_timeout` pragmas, reconcile task claim retry on `SQLITE_BUSY`, staggered startup for reconcile-heavy background jobs (`settings.jobStartupDelaySeconds`).
-- **WebSocket exec & log session hardening** — quotas (**3 exec** / 3 log WS per resource), per-session exec lifecycle (Plan 17), 60s/120s pending timeouts, 8h exec max, 30s graceful drain, OTEL metrics, HA AMQP fail-fast, integration tests, swagger WS protocol docs, operator guide (`docs/operations/ws-sessions.md`).
-- **Multi exec sessions (Plan 17)** — `GET /api/v3/agent/exec/sessions`; agent exec WS `…/agent/exec/microservice/:uuid/:sessionId`; user ACTIVATION with `sessionId`; `MicroserviceExecSessions` table; `execMaxConcurrentPerResource` config (default **3**).
+- **WebSocket exec & log session hardening** — quotas (**3 exec** / 3 log WS per resource), per-session exec lifecycle, 60s/120s pending timeouts, 8h exec max, 30s graceful drain, OTEL metrics, HA AMQP fail-fast, integration tests, swagger WS protocol docs, operator guide (`docs/operations/ws-sessions.md`).
+- **Multi exec sessions** — `GET /api/v3/agent/exec/sessions`; agent exec WS `…/agent/exec/microservice/:uuid/:sessionId`; user ACTIVATION with `sessionId`; `MicroserviceExecSessions` table; `execMaxConcurrentPerResource` config (default **3**).
+- **WebSocket relay production** — unified **`WsRelayTransport`** abstraction; cross-replica exec/log relay backend selected at startup by **`nats.enabled`** (`NATS_ENABLED`): **AMQP** router pool (8 connections per replica, overflow recovery, sendable gating) when `false`, **NATS Core** pub/sub on platform hub (`controller-relay` account) when `true`. Fail-fast activation on both transports; log backpressure drops `LOG_LINE` under pressure. Config: `server.webSocket.relay.amqp.*`, `server.webSocket.relay.nats.*`. No new relay env var; HA swagger/docs updated per R112.
 
 ### Fixed
 
+- NATS relay and AMQP router connection resolvers — **Remote CP** uses Edgelet bridge DNS then DB host only (no `*.svc.cluster.local`); **Kubernetes CP** uses `nats-server.{namespace}.svc.cluster.local` / `router.{namespace}.svc.cluster.local` with DB host fallback; connect failures log and throw aggregate errors for all attempts; relay log messages are transport-aware.
+- NATS relay **`controller-relay` creds loading** — read Opaque secret values as plain UTF-8 `.creds` text (matches `nats-service.js` and DB storage); fixes **`unable to parse credentials`** on hub connect when `NATS_ENABLED=true`.
+- NATS platform relay identity renamed to account/user **`controller`** with rules **`controller-account`** / **`controller-user`**; **`GET /nats/accounts/controller/users/controller/creds`** supported for operator cred export.
 - Exec AMQP relay re-attaches queue receivers whenever user or agent WebSocket connects (fixes user-first sessions where ACTIVATION and STDIN never reached Edgelet); ACTIVATION is resent on agent WS reconnect.
 - Controller register accepts optional **`schedule: 0`**; server always enforces schedule **0** on create, re-register, and **`PATCH /api/v3/microservices/system/:uuid`** for controller workloads.
 - Agent version command (**`GET /api/v3/agent/version`**) refreshes the provision key on each pull instead of returning a stale or deleted key.

Dockerfile

@@ -2,10 +2,10 @@
 # ioFog overrides: EDGEOPS_CONSOLE_REPO=https://github.com/eclipse-iofog/edgeops-console
 #                   EDGEOPS_CONSOLE_FLAVOR=iofog
 # node:24-bookworm — pin manifest list digest for reproducible multi-arch builds
-FROM node:24-bookworm@sha256:40ad9f3064e67d6860b4bc3fe1880b2953934fd6320ada990e45fe0efa6badd7 AS console-builder
+FROM node:24-bookworm@sha256:fdddfb3e688158251943d52eba361de991548f6814007acba4917ae6b512d6be AS console-builder
 
 ARG EDGEOPS_CONSOLE_REPO=https://github.com/Datasance/edgeops-console
-ARG EDGEOPS_CONSOLE_VERSION=v1.0.2
+ARG EDGEOPS_CONSOLE_VERSION=v1.0.3
 ARG EDGEOPS_CONSOLE_FLAVOR=datasance
 
 RUN apt-get update \
@@ -28,7 +28,7 @@ RUN test -f build/index.html \
     && cp -a build /tmp/console/build
 
 
-FROM node:24-bookworm@sha256:40ad9f3064e67d6860b4bc3fe1880b2953934fd6320ada990e45fe0efa6badd7 AS builder
+FROM node:24-bookworm@sha256:fdddfb3e688158251943d52eba361de991548f6814007acba4917ae6b512d6be AS builder
 
 ARG PKG_VERSION
 
@@ -48,9 +48,9 @@ RUN npm pack
 
 
 # ubi9/nodejs-24-minimal:latest — pin manifest list digest for reproducible multi-arch builds
-FROM registry.access.redhat.com/ubi9/nodejs-24-minimal@sha256:e76548c58c4a29907cef1e01cb6a4cab426eb071ffe28ac1f25dd58d14a89569
+FROM registry.access.redhat.com/ubi9/nodejs-24-minimal@sha256:cc7648f8e1c7d628e4334328a712f30ea0820787bb92836cc93e349674c689bf
 
-ARG EDGEOPS_CONSOLE_VERSION=v1.0.2
+ARG EDGEOPS_CONSOLE_VERSION=v1.0.3
 ARG IMAGE_REGISTRY
 ARG OCI_SOURCE_REPO
 ARG CONTROLLER_DISTRIBUTION=iofog

Makefile

@@ -1,5 +1,5 @@
 # Local Docker build — mirrors CI/release build-args (see .github/actions/set-build-env).
-# Override any variable: make build FLAVOR=iofog EDGEOPS_CONSOLE_VERSION=v1.0.2
+# Override any variable: make build FLAVOR=iofog EDGEOPS_CONSOLE_VERSION=v1.0.3
 
 FLAVOR ?= datasance
 IMAGE_NAME ?= controller
@@ -25,7 +25,7 @@ else
   $(error FLAVOR must be "datasance" or "iofog", got "$(FLAVOR)")
 endif
 
-EDGEOPS_CONSOLE_VERSION ?= v1.0.2
+EDGEOPS_CONSOLE_VERSION ?= v1.0.3
 
 IMAGE_REF = $(IMAGE_REGISTRY)/$(IMAGE_NAME):$(DOCKER_TAG)
 

docs/architecture.md

@@ -206,15 +206,45 @@ Full spec: [`.cursor/controllerv3.8/docs/15-fog-platform-reconcile.md`](../.curs
 
 ## WebSocket exec & log sessions
 
-Interactive **exec** and **log streaming** use paired WebSocket sessions between operators (Bearer JWT), Controller, and Edgelet agents (fog token). Plan 16 hardens log sessions and shared WS infra (HA, drain, OTEL). **Plan 17** redesigns **microservice exec** to log-style multi-session flow (3 concurrent per MS, agent poll + session-scoped WS) — **Edgelet agent wire change required** for exec (see [edgelet-invariants.md §10.1](../.cursor/controllerv3.8/docs/edgelet-invariants.md)).
+Interactive **exec** and **log streaming** use paired WebSocket sessions between operators (Bearer JWT), Controller, and Edgelet agents (fog token). Plan 16 hardens log sessions and shared WS infra (HA, drain, OTEL). **Plan 17** redesigns **microservice exec** to log-style multi-session flow (3 concurrent per MS, agent poll + session-scoped WS). **Plan 18** production-hardens cross-replica relay via **`WsRelayTransport`** — AMQP pool + recovery when `nats.enabled=false`, NATS Core when `nats.enabled=true` (R102–R113). **Edgelet agent wire change required** for exec only (see [edgelet-invariants.md §10.1](../.cursor/controllerv3.8/docs/edgelet-invariants.md)).
+
+```mermaid
+flowchart TB
+  subgraph ws [WebSocketServer]
+    U[User WS]
+    A[Agent WS]
+  end
+
+  subgraph factory [WsRelayTransportFactory]
+    SEL{nats.enabled?}
+  end
+
+  subgraph amqp [AmqpRelayTransport]
+    POOL[RouterConnectionManager pool x8]
+    Q[agent/user queues per sessionId]
+  end
+
+  subgraph nats [NatsRelayTransport]
+    NC[NatsRelayConnectionManager]
+    SUB[Core pub/sub per sessionId]
+  end
+
+  U --> ws
+  A --> ws
+  ws --> factory
+  SEL -->|false| amqp
+  SEL -->|true| nats
+  POOL --> Q
+  NC --> SUB
+```
 
 ```mermaid
 sequenceDiagram
   participant U as User WS
   participant C as Controller
   participant DB as MicroserviceExecSessions
   participant CT as Change tracking
-  participant Q as AMQP
+  participant R as WsRelayTransport
   participant A as Edgelet agent
 
   Note over U,A: MS exec (R92–R98) — log-style
@@ -228,11 +258,11 @@ sequenceDiagram
   A->>C: GET /agent/exec/sessions
   A->>C: WS /agent/exec/microservice/:uuid/:sessionId
   C->>DB: ACTIVE agentConnected
-  C->>Q: agent-{sessionId} user-{sessionId}
+  C->>R: enable bridge if cross-replica
   U->>C: STDIN
-  C->>Q->>A: relay
+  C->>R->>A: relay
   A->>C: STDOUT
-  C->>Q->>U: relay
+  C->>R->>U: relay
   U-->>C: close
   C->>DB: DELETE row
   C->>CT: execSessions if needed
@@ -242,12 +272,12 @@ sequenceDiagram
   C->>C: provision debug system MS
   U->>C: WS /microservices/system/exec/:debugMsUuid
 
-  Note over U,A: Logs (R82–R83, R84) — unchanged Plan 16
+  Note over U,A: Logs (R82–R83, R84/R112)
   U->>C: WS logs + tail params
   C->>DB: PENDING sessionId
   A->>C: WS agent/logs/:sessionId
   A->>C: LOG_LINE
-  C->>Q->>U: logs-user-{sessionId}
+  C->>R->>U: relay
 ```
 
 | Topic | Normative value |
@@ -263,21 +293,23 @@ sequenceDiagram
 | Log concurrency | **3** user log WS per microservice (or per fog for node logs) |
 | Log limits | Tail max **5,000** lines; **120s** pending; **2h** idle |
 | Log content | Live relay only — no log line persistence; audit connect/disconnect |
-| HA relay | Cross-replica sessions **require** AMQP (`WebSocketQueueService`); same-replica may use direct WS; **fail fast** when router down |
+| HA relay | Cross-replica sessions **require** a **relay backend** (R112): **AMQP** router queues when `nats.enabled=false`; **NATS Core** subjects on hub when `nats.enabled=true`. Same-replica may use direct WS; **fail fast** close **1013** when active backend unavailable |
 | Graceful drain | **30s** on SIGTERM / k8s `preStop` — CLOSE frames, queue cleanup, session row delete |
 | Security | Agent handlers validate fog token **before** message processing; **50** upgrades/min/IP; **100** active WS/IP; JWT in `?token=` (ingress log redaction required) |
 | Scale SLO | **500** concurrent WS per replica; **p99 pairing < 5s** |
 | Observability | OpenTelemetry: active/pending sessions, pairing latency, AMQP failures, router connectivity |
 
 **OTEL metric names (R87):** `ws_exec_sessions_active`, `ws_log_sessions_active`, `ws_pending_pairings`, `ws_pairing_duration_ms` (histogram), `ws_amqp_publish_errors`, `ws_router_connected` (gauge). Emitted when `ENABLE_TELEMETRY=true`; see `src/websocket/ws-metrics.js`.
 
-**HA config (`server.webSocket.ha`):** `crossReplicaRequiresAmqp` (default `true`), `failFastOnRouterUnavailable` (default `true`). Env: `WS_HA_CROSS_REPLICA_REQUIRES_AMQP`, `WS_HA_FAIL_FAST_ON_ROUTER_UNAVAILABLE`. Graceful drain timeout: `server.webSocket.session.drainTimeoutMs` (default **30s**, env `WS_DRAIN_TIMEOUT_MS`).
+**Relay transport (Plan 18, R102):** Selected once at startup from existing `nats.enabled` / `NATS_ENABLED` — **no new relay env var**. `false` → AMQP pool (8 connections, sticky by `sessionId`); `true` → NATS Core on hub with **`controller`** NATS account. Relay connect is **lazy** — does not block Controller startup.
+
+**HA config (`server.webSocket.ha`):** `crossReplicaRequiresAmqp` (default `true`; semantics: cross-replica requires **active relay backend** per R112), `failFastOnRouterUnavailable` (default `true`; applies to selected backend). Env: `WS_HA_CROSS_REPLICA_REQUIRES_AMQP`, `WS_HA_FAIL_FAST_ON_ROUTER_UNAVAILABLE`. Graceful drain timeout: `server.webSocket.session.drainTimeoutMs` (default **30s**, env `WS_DRAIN_TIMEOUT_MS`).
 
-**Core modules:** `src/websocket/server.js`, `exec-session-manager.js` (Plan 17), `log-session-manager.js`, `src/services/websocket-queue-service.js`, `src/services/router-connection-service.js`.
+**Core modules:** `src/websocket/server.js`, `exec-session-manager.js` (Plan 17), `log-session-manager.js`, `ws-relay-transport-factory.js` / `amqp-relay-transport.js` / `nats-relay-transport.js` (Plan 18), `src/services/websocket-queue-service.js`, `src/services/router-connection-manager.js` (Plan 18), `src/services/nats-relay-connection-manager.js` (Plan 18).
 
-**Operator guide:** [operations/ws-sessions.md](operations/ws-sessions.md) — ingress `?token=` log redaction, HTTPS/WSS, multi-replica AMQP requirement, k8s preStop drain, load SLO probe.
+**Operator guide:** [operations/ws-sessions.md](operations/ws-sessions.md) — ingress `?token=` log redaction, HTTPS/WSS, multi-replica relay backend (`nats.enabled`), k8s preStop drain, load SLO probe.
 
-Full spec: Plan 16 [logs + shared infra](../.cursor/controllerv3.8/docs/16-ws-exec-log-hardening.md) · Plan 17 [MS exec](../.cursor/controllerv3.8/docs/17-multi-exec-sessions.md) · RFC R80–R91, R92–R101 · Edgelet contract: [edgelet-invariants.md §10–§10.1](../.cursor/controllerv3.8/docs/edgelet-invariants.md).
+Full spec: Plan 16 [logs + shared infra](../.cursor/controllerv3.8/docs/16-ws-exec-log-hardening.md) · Plan 17 [MS exec](../.cursor/controllerv3.8/docs/17-multi-exec-sessions.md) · Plan 18 [WS relay production](../.cursor/controllerv3.8/docs/18-ws-relay-production.md) · RFC R80–R91, R92–R101, R102–R113 · Edgelet contract: [edgelet-invariants.md §10–§10.1](../.cursor/controllerv3.8/docs/edgelet-invariants.md).
 
 ---