multi exec per resources

.env.example

@@ -6,7 +6,7 @@ NODE_ENV=development
 
 # EdgeOps Console static embed (npm run build:console → dev/console/build)
 EDGEOPS_CONSOLE_PATH=dev/console/build  # must be absolute path
-EDGEOPS_CONSOLE_VERSION=v1.0.1
+EDGEOPS_CONSOLE_VERSION=v1.0.2
 # EDGEOPS_CONSOLE_REPO=https://github.com/Datasance/edgeops-console
 # EDGEOPS_CONSOLE_FLAVOR=datasance
 

.github/actions/set-build-env/action.yml

@@ -8,7 +8,7 @@ runs:
       shell: bash
       run: |
         VERSION="${{ env.EDGEOPS_CONSOLE_VERSION }}"
-        if [ -z "$VERSION" ]; then VERSION="1.0.1"; fi
+        if [ -z "$VERSION" ]; then VERSION="1.0.2"; fi
         echo "EDGEOPS_CONSOLE_VERSION=$VERSION" >> "${GITHUB_ENV}"
 
         REPO="${{ env.EDGEOPS_CONSOLE_REPO }}"

CHANGELOG.md

@@ -39,6 +39,17 @@ Controller v3.8 is a **greenfield** release aligned with **Edgelet**. There is *
 - **Diagnostics**, **strace**, image **snapshot** / **download** APIs.
 - All **`/api/v3/flow`** routes.
 
+#### WebSocket exec — multi-session
+
+- **Microservice exec REST removed** — `POST/DELETE /api/v3/microservices/:uuid/exec` and `…/system/:uuid/exec` no longer exist. Open exec with **direct WebSocket** only: `WS /api/v3/microservices/exec/:uuid` (or `…/system/exec/:uuid`).
+- **3 concurrent exec sessions** per microservice (was 1 user exec WS per MS in Plan 16).
+- **Per-session lifecycle** — closing one exec session deletes only that session row only (no microservice-level exec flag).
+- **`execEnabled` removed** — dropped `microservices.exec_enabled` column and agent MS list field; exec attach is poll-driven only (`GET /agent/exec/sessions`).
+- **Agent exec discovery** — new `GET /api/v3/agent/exec/sessions` when change tracking reports `execSessions: true`.
+- **Agent exec WebSocket** — `WS /api/v3/agent/exec/microservice/:microserviceUuid/:sessionId` only; legacy `WS /api/v3/agent/exec/:microserviceUuid` with initial MessagePack pairing frame **removed**.
+- **User session announce** — Controller sends **ACTIVATION** (type 5) to user with `{ sessionId, microserviceUuid }` on connect.
+- **Fog debug** — `POST/DELETE /api/v3/iofog/:uuid/exec` unchanged (provisions debug system MS); interactive shell via `WS /api/v3/microservices/system/exec/:debugMsUuid` (system path — debug MS is a system microservice).
+
 #### Authentication
 
 - **`keycloak-connect` removed** — generic OIDC (`openid-client` + JWKS discovery).
@@ -97,10 +108,12 @@ Controller v3.8 is a **greenfield** release aligned with **Edgelet**. There is *
 - **K8s control plane:** hub **`iofog-router`** ConfigMap patches serialized via DB lock; K8s Service create/update/delete with LoadBalancer watch timeout.
 - **`service-bridge-config.js`** — full recompute of service-derived TCP bridge config per fog on reconcile (preserves router base config).
 - **SQLite single-node production hardening** — WAL + `busy_timeout` pragmas, reconcile task claim retry on `SQLITE_BUSY`, staggered startup for reconcile-heavy background jobs (`settings.jobStartupDelaySeconds`).
-- **WebSocket exec & log session hardening** — quotas (1 exec / 3 log WS per resource), exec_b lifecycle, 60s/120s pending timeouts, 8h exec max, 30s graceful drain, OTEL metrics, HA AMQP fail-fast, integration tests, swagger WS protocol docs, operator guide (`docs/operations/ws-sessions.md`).
+- **WebSocket exec & log session hardening** — quotas (**3 exec** / 3 log WS per resource), per-session exec lifecycle (Plan 17), 60s/120s pending timeouts, 8h exec max, 30s graceful drain, OTEL metrics, HA AMQP fail-fast, integration tests, swagger WS protocol docs, operator guide (`docs/operations/ws-sessions.md`).
+- **Multi exec sessions (Plan 17)** — `GET /api/v3/agent/exec/sessions`; agent exec WS `…/agent/exec/microservice/:uuid/:sessionId`; user ACTIVATION with `sessionId`; `MicroserviceExecSessions` table; `execMaxConcurrentPerResource` config (default **3**).
 
 ### Fixed
 
+- Exec AMQP relay re-attaches queue receivers whenever user or agent WebSocket connects (fixes user-first sessions where ACTIVATION and STDIN never reached Edgelet); ACTIVATION is resent on agent WS reconnect.
 - Controller register accepts optional **`schedule: 0`**; server always enforces schedule **0** on create, re-register, and **`PATCH /api/v3/microservices/system/:uuid`** for controller workloads.
 - Agent version command (**`GET /api/v3/agent/version`**) refreshes the provision key on each pull instead of returning a stale or deleted key.
 - Controller AMQP certificate provisioning uses shared **`default-router-local-ca`** instead of per-fog router local CA secret names.

Dockerfile

@@ -5,7 +5,7 @@
 FROM node:24-bookworm@sha256:40ad9f3064e67d6860b4bc3fe1880b2953934fd6320ada990e45fe0efa6badd7 AS console-builder
 
 ARG EDGEOPS_CONSOLE_REPO=https://github.com/Datasance/edgeops-console
-ARG EDGEOPS_CONSOLE_VERSION=v1.0.1
+ARG EDGEOPS_CONSOLE_VERSION=v1.0.2
 ARG EDGEOPS_CONSOLE_FLAVOR=datasance
 
 RUN apt-get update \
@@ -50,7 +50,7 @@ RUN npm pack
 # ubi9/nodejs-24-minimal:latest — pin manifest list digest for reproducible multi-arch builds
 FROM registry.access.redhat.com/ubi9/nodejs-24-minimal@sha256:e76548c58c4a29907cef1e01cb6a4cab426eb071ffe28ac1f25dd58d14a89569
 
-ARG EDGEOPS_CONSOLE_VERSION=v1.0.1
+ARG EDGEOPS_CONSOLE_VERSION=v1.0.2
 ARG IMAGE_REGISTRY
 ARG OCI_SOURCE_REPO
 ARG CONTROLLER_DISTRIBUTION=iofog

Makefile

@@ -1,5 +1,5 @@
 # Local Docker build — mirrors CI/release build-args (see .github/actions/set-build-env).
-# Override any variable: make build FLAVOR=iofog EDGEOPS_CONSOLE_VERSION=v1.0.1
+# Override any variable: make build FLAVOR=iofog EDGEOPS_CONSOLE_VERSION=v1.0.2
 
 FLAVOR ?= datasance
 IMAGE_NAME ?= controller
@@ -25,7 +25,7 @@ else
   $(error FLAVOR must be "datasance" or "iofog", got "$(FLAVOR)")
 endif
 
-EDGEOPS_CONSOLE_VERSION ?= v1.0.1
+EDGEOPS_CONSOLE_VERSION ?= v1.0.2
 
 IMAGE_REF = $(IMAGE_REGISTRY)/$(IMAGE_NAME):$(DOCKER_TAG)
 

docs/architecture.md

@@ -206,45 +206,65 @@ Full spec: [`.cursor/controllerv3.8/docs/15-fog-platform-reconcile.md`](../.curs
 
 ## WebSocket exec & log sessions
 
-Interactive **exec** and **log streaming** use paired WebSocket sessions between operators (Bearer JWT), Controller, and Edgelet agents (fog token). Plan 16 hardens lifecycle, quotas, multi-replica HA relay, and observability — **without changing the Edgelet wire protocol**.
+Interactive **exec** and **log streaming** use paired WebSocket sessions between operators (Bearer JWT), Controller, and Edgelet agents (fog token). Plan 16 hardens log sessions and shared WS infra (HA, drain, OTEL). **Plan 17** redesigns **microservice exec** to log-style multi-session flow (3 concurrent per MS, agent poll + session-scoped WS) — **Edgelet agent wire change required** for exec (see [edgelet-invariants.md §10.1](../.cursor/controllerv3.8/docs/edgelet-invariants.md)).
 
 ```mermaid
 sequenceDiagram
   participant U as User WS
-  participant C as Controller replica
-  participant DB as Database
-  participant Q as AMQP Router
-  participant A as Agent WS
-
-  Note over U,A: Exec (R80–R81, R84)
-  U->>C: WS exec (RBAC)
-  A->>C: WS agent/exec + execId
-  C->>C: Pair SessionManager
-  C->>Q: Enable agent-{execId} user-{execId}
+  participant C as Controller
+  participant DB as MicroserviceExecSessions
+  participant CT as Change tracking
+  participant Q as AMQP
+  participant A as Edgelet agent
+
+  Note over U,A: MS exec (R92–R98) — log-style
+  U->>C: WS /microservices/exec/:uuid
+  C->>DB: INSERT sessionId PENDING
+  C->>CT: execSessions=true
+  C->>U: ACTIVATION { sessionId }
+  C->>U: STDERR waiting…
+
+  A->>C: GET /changes execSessions
+  A->>C: GET /agent/exec/sessions
+  A->>C: WS /agent/exec/microservice/:uuid/:sessionId
+  C->>DB: ACTIVE agentConnected
+  C->>Q: agent-{sessionId} user-{sessionId}
   U->>C: STDIN
-  C->>Q->>A: or direct WS same replica
+  C->>Q->>A: relay
   A->>C: STDOUT
   C->>Q->>U: relay
   U-->>C: close
-  C->>DB: execEnabled=false INACTIVE
+  C->>DB: DELETE row
+  C->>CT: execSessions if needed
 
-  Note over U,A: Logs (R82–R83, R84)
+  Note over U,A: Fog debug (R99)
+  U->>C: POST /iofog/:uuid/exec
+  C->>C: provision debug system MS
+  U->>C: WS /microservices/system/exec/:debugMsUuid
+
+  Note over U,A: Logs (R82–R83, R84) — unchanged Plan 16
   U->>C: WS logs + tail params
   C->>DB: PENDING sessionId
   A->>C: WS agent/logs/:sessionId
   A->>C: LOG_LINE
   C->>Q->>U: logs-user-{sessionId}
 ```
 
-| Topic | Normative value (RFC R80–R91) |
-|-------|-------------------------------|
-| Exec lifecycle | **exec_b** — WS close sets `execEnabled=false`; **1** user exec WS per microservice |
-| Exec timeouts | **60s** pending for agent; **8h** max active session |
+| Topic | Normative value |
+|-------|-----------------|
+| MS exec entry | **Direct user WS** — no `POST …/microservices/…/exec` (R92, R94) |
+| MS exec concurrency | **3** user exec WS per microservice (R93) |
+| MS exec lifecycle | **Per-session** — close deletes session row only; **no** `execEnabled=false` (R98) |
+| MS exec pending / max | **60s** pending for agent; **8h** max active session (Plan 16 carry-over) |
+| Agent exec discovery | `GET /agent/exec/sessions` on `execSessions` change flag (R95, R100) |
+| Agent exec WS | `/agent/exec/microservice/:uuid/:sessionId` only — legacy `/agent/exec/:uuid` removed (R96) |
+| User session notify | **ACTIVATION** (type 5) with `{ sessionId, microserviceUuid }` (R97) |
+| Fog debug provision | `POST/DELETE /iofog/:uuid/exec` unchanged; shell via `WS /microservices/system/exec/:debugMsUuid` (R99) |
 | Log concurrency | **3** user log WS per microservice (or per fog for node logs) |
 | Log limits | Tail max **5,000** lines; **120s** pending; **2h** idle |
 | Log content | Live relay only — no log line persistence; audit connect/disconnect |
 | HA relay | Cross-replica sessions **require** AMQP (`WebSocketQueueService`); same-replica may use direct WS; **fail fast** when router down |
-| Graceful drain | **30s** on SIGTERM / k8s `preStop` — CLOSE frames, queue cleanup, DB status update |
+| Graceful drain | **30s** on SIGTERM / k8s `preStop` — CLOSE frames, queue cleanup, session row delete |
 | Security | Agent handlers validate fog token **before** message processing; **50** upgrades/min/IP; **100** active WS/IP; JWT in `?token=` (ingress log redaction required) |
 | Scale SLO | **500** concurrent WS per replica; **p99 pairing < 5s** |
 | Observability | OpenTelemetry: active/pending sessions, pairing latency, AMQP failures, router connectivity |
@@ -253,11 +273,11 @@ sequenceDiagram
 
 **HA config (`server.webSocket.ha`):** `crossReplicaRequiresAmqp` (default `true`), `failFastOnRouterUnavailable` (default `true`). Env: `WS_HA_CROSS_REPLICA_REQUIRES_AMQP`, `WS_HA_FAIL_FAST_ON_ROUTER_UNAVAILABLE`. Graceful drain timeout: `server.webSocket.session.drainTimeoutMs` (default **30s**, env `WS_DRAIN_TIMEOUT_MS`).
 
-**Core modules:** `src/websocket/server.js`, `session-manager.js`, `log-session-manager.js`, `src/services/websocket-queue-service.js`, `src/services/router-connection-service.js`.
+**Core modules:** `src/websocket/server.js`, `exec-session-manager.js` (Plan 17), `log-session-manager.js`, `src/services/websocket-queue-service.js`, `src/services/router-connection-service.js`.
 
 **Operator guide:** [operations/ws-sessions.md](operations/ws-sessions.md) — ingress `?token=` log redaction, HTTPS/WSS, multi-replica AMQP requirement, k8s preStop drain, load SLO probe.
 
-Full spec: [`.cursor/controllerv3.8/docs/16-ws-exec-log-hardening.md`](../.cursor/controllerv3.8/docs/16-ws-exec-log-hardening.md) · RFC R80–R91 · Edgelet contract: [edgelet-invariants.md §10](../.cursor/controllerv3.8/docs/edgelet-invariants.md).
+Full spec: Plan 16 [logs + shared infra](../.cursor/controllerv3.8/docs/16-ws-exec-log-hardening.md) · Plan 17 [MS exec](../.cursor/controllerv3.8/docs/17-multi-exec-sessions.md) · RFC R80–R91, R92–R101 · Edgelet contract: [edgelet-invariants.md §10–§10.1](../.cursor/controllerv3.8/docs/edgelet-invariants.md).
 
 ---
 

docs/operations/ws-sessions.md

@@ -18,6 +18,8 @@ Controller exposes **interactive exec** and **log streaming** over WebSocket on
 | **User auth** | Bearer JWT via `Authorization` header or `?token=` query param (browser Console). RBAC: `execSessions`, `logs`, `systemExecSessions`, `systemLogs`. |
 | **Agent auth** | Fog token on `/api/v3/agent/exec/*` and `/api/v3/agent/logs/*` — OIDC does **not** apply to agent routes. |
 
+> **Plan 17 (MS exec):** Open exec with **direct WebSocket** — `wss://…/api/v3/microservices/exec/:uuid` (app MS) or `…/system/exec/:uuid` (system MS). **No** `POST …/exec` before connect. Up to **3** concurrent exec sessions per microservice. Agent discovers sessions via `GET /api/v3/agent/exec/sessions` and connects `WS /api/v3/agent/exec/microservice/:uuid/:sessionId`. Fog node debug: `POST/DELETE /api/v3/iofog/:uuid/exec` provisions the debug system MS, then **`WS …/microservices/system/exec/:debugMsUuid`** (not the app exec path). Full spec: [17-multi-exec-sessions.md](../.cursor/controllerv3.8/docs/17-multi-exec-sessions.md).
+
 ### Ingress log redaction (required)
 
 Browser clients pass JWT in the query string: `wss://controller.example.com/api/v3/microservices/{uuid}/logs?token=…`
@@ -44,7 +46,7 @@ Without redaction, long-lived bearer tokens may appear in load balancer logs.
 **Requirements:**
 
 1. Deploy the **router** system microservice and ensure Controller can reach AMQP (`RouterConnectionService`).
-2. Run **2+ Controller replicas** behind a load balancer with **sticky sessions optional** — cross-replica exec/log uses AMQP queues (`agent-{execId}`, `user-{execId}`, `logs-user-{sessionId}`).
+2. Run **2+ Controller replicas** behind a load balancer with **sticky sessions optional** — cross-replica exec/log uses AMQP queues (`agent-{sessionId}`, `user-{sessionId}`, `logs-user-{sessionId}`).
 3. When the router is unavailable, new cross-replica sessions close with WebSocket code **1013** (`Router unavailable for cross-replica session`).
 
 Same-replica sessions may relay directly without AMQP when both user and agent land on the same pod.
@@ -88,7 +90,7 @@ spec:
 
 1. Open an exec or log session against a running pod.
 2. `kubectl delete pod <controller-pod> --grace-period=45`
-3. Confirm the client receives close code **1001** within ~30s and exec is disabled (`execEnabled=false` for exec sessions).
+3. Confirm the client receives close code **1001** within ~30s and the session row is cleaned up (Plan 17: per-session delete; no global `execEnabled=false` for MS exec).
 4. Confirm replacement pod accepts new sessions.
 
 ---
@@ -99,14 +101,18 @@ spec:
 |--------|--------|
 | Concurrent WS per replica | **500** (`WS_REPLICA_MAX_CONCURRENT_WS`) |
 | p99 exec pairing latency | **< 5s** |
+| Exec sessions per microservice | **3** concurrent user WS (Plan 17) |
 
 Run the load probe locally:
 
 ```bash
 nvm use 24
 node test/load/ws-pairing-load.js --pairs 500
+node test/load/ws-pairing-load.js --multi-ms 100
 ```
 
+The `--multi-ms` mode creates **3 exec sessions per microservice** (100 MS × 3 = 300 pairs) to validate multi-session pairing latency under the same p99 SLO.
+
 For production validation, repeat against a staging cluster with real agent simulators and record p99 from Controller OTEL histogram `ws_pairing_duration_ms`.
 
 ---
@@ -130,7 +136,7 @@ Enable `ENABLE_TELEMETRY=true`. Key metrics (`src/websocket/ws-metrics.js`):
 
 | Session | Limit |
 |---------|-------|
-| Exec user WS per microservice | **1** |
+| Exec user WS per microservice | **3** (Plan 17 — direct WS; no POST/DELETE MS exec REST) |
 | Exec pending (user waits for agent) | **60s** |
 | Exec max duration | **8h** |
 | Log user WS per microservice/fog | **3** |