[codex] Harden staging MCP E2E credentials

README.md

@@ -208,6 +208,17 @@ discover_services -> invoke_and_pay -> get_receipt
 Use staging only for preview validation and live E2E smoke tests. Do not use staging for production Agent workflows.
 
 ```bash
+export SYNAPSE_ENV=staging
+
+npm run test:e2e:staging
+```
+
+For SynapseNetwork maintainers, staging E2E reads `SYNAPSE_AGENT_KEY` from Google Secret Manager by default using the `synapse-staging-e2e-agent-credential` secret. Set `SYNAPSE_E2E_SECRET_PROJECT` when the active `gcloud` project is not the staging secrets project.
+
+Developers who are not using SynapseNetwork's GCP project can force a local Agent Key:
+
+```bash
+export SYNAPSE_E2E_AGENT_KEY_SOURCE=env
 export SYNAPSE_AGENT_KEY=agt_xxx
 export SYNAPSE_ENV=staging
 
@@ -219,7 +230,6 @@ By default, staging E2E performs broad discovery with `sort=lowest_price`, then
 Specified staging service:
 
 ```bash
-export SYNAPSE_AGENT_KEY=agt_xxx
 export SYNAPSE_ENV=staging
 export SYNAPSE_E2E_SERVICE_ID=svc_quotes_famous_top3
 export SYNAPSE_E2E_PAYLOAD_JSON='{"topic":"agent payments"}'
@@ -231,7 +241,6 @@ npm run test:e2e:staging
 Token-metered staging service:
 
 ```bash
-export SYNAPSE_AGENT_KEY=agt_xxx
 export SYNAPSE_ENV=staging
 export SYNAPSE_E2E_SERVICE_ID=svc_deepseek_chat
 export SYNAPSE_E2E_PAYLOAD_JSON='{"messages":[{"role":"user","content":"hello"}],"max_tokens":32}'

docs/bugfix/2026-05-12-staging-e2e-credential-invalid.md

@@ -0,0 +1,83 @@
+# Staging MCP E2E Credential Blocked
+
+- Date: 2026-05-12
+- Environment profile: staging
+- Command: `SYNAPSE_ENV=staging SYNAPSE_E2E_SERVICE_ID=svc_synapse_echo SYNAPSE_E2E_PAYLOAD_JSON='{"message":"mcp staging e2e","source":"synapse-mcp-server"}' SYNAPSE_E2E_COST_USDC='0.000000' npm run test:e2e:staging`
+- Status: fixed on 2026-05-13
+
+## Sanitized Failure
+
+The MCP server started over stdio, listed tools, discovered `svc_synapse_echo`, and selected it for invocation.
+
+`invoke_and_pay` failed with:
+
+```json
+{
+  "status": 401,
+  "code": "CREDENTIAL_INVALID",
+  "message": "Credential is invalid"
+}
+```
+
+A second run mapped the locally configured legacy `SYNAPSE_API_KEY` value into `SYNAPSE_AGENT_KEY`; it failed with the same sanitized Gateway error.
+
+After updating staging E2E to read `SYNAPSE_AGENT_KEY` from Google Secret Manager secret `synapse-staging-e2e-agent-credential`, the MCP server again started over stdio, listed tools, discovered `svc_synapse_echo`, and selected it for invocation.
+
+`invoke_and_pay` then failed with:
+
+```json
+{
+  "status": 401,
+  "code": "CREDENTIAL_INACTIVE",
+  "message": "Credential is inactive"
+}
+```
+
+## Root Cause
+
+The current local Agent Key candidates are not accepted by the staging Gateway for paid invocation, and the Secret Manager-backed staging E2E credential is inactive in Gateway. The MCP adapter is still forwarding the credential as `X-Credential`; local unit tests and mock E2E cover this request construction.
+
+Secret Manager checks confirmed that `synapse-staging-e2e-agent-credential` exists and its latest version is enabled. The remaining issue is Gateway credential lifecycle state, not Secret Manager access.
+
+## Fix Summary
+
+- Hardened live MCP E2E stdio child-process env handling so owner/provider/admin/private-key environment variables are not inherited by the MCP server process.
+- Added a regression test proving sensitive Synapse owner/provider/admin variables are dropped.
+- Added this bugfix record directory for live E2E failures.
+- Updated staging E2E to read its maintainer Agent Key from Secret Manager by default.
+
+## Verification Completed
+
+```bash
+npm test
+npm run verify:mcp
+npm run ci:quality
+npm run smoke:cli
+npm pack --dry-run
+SYNAPSE_E2E_AGENT_KEY_SOURCE=secret SYNAPSE_ENV=staging SYNAPSE_E2E_SERVICE_ID=svc_synapse_echo SYNAPSE_E2E_PAYLOAD_JSON='{"message":"mcp staging e2e","source":"synapse-mcp-server"}' SYNAPSE_E2E_COST_USDC='0.000000' npm run test:e2e:staging
+```
+
+## Required Follow-Up
+
+The staging Agent Key stored in Secret Manager secret `synapse-staging-e2e-agent-credential` was rotated by issuing a new dedicated credential named `mcp-staging-e2e-<timestamp>` and writing it as a new Secret Manager version.
+
+Final rerun:
+
+```bash
+SYNAPSE_E2E_AGENT_KEY_SOURCE=secret \
+SYNAPSE_ENV=staging \
+SYNAPSE_E2E_SERVICE_ID=svc_synapse_echo \
+SYNAPSE_E2E_PAYLOAD_JSON='{"message":"mcp staging e2e","source":"synapse-mcp-server"}' \
+SYNAPSE_E2E_COST_USDC='0.000000' \
+npm run test:e2e:staging
+```
+
+Result:
+
+```text
+Live MCP staging E2E credential source: secret-manager:synapse-staging-e2e-agent-credential
+Live MCP staging E2E selected service: svc_synapse_echo
+Live MCP staging E2E passed: svc_synapse_echo -> inv-<redacted> -> SUCCEEDED
+```
+
+The MCP staging live E2E release gate is unblocked.

docs/bugfix/README.md

@@ -0,0 +1,15 @@
+# Bugfix Records
+
+Use this directory for live MCP E2E failures that must be fixed before release.
+
+Each record should include:
+
+- Date and environment profile.
+- Command that failed, with secrets redacted.
+- Sanitized error output.
+- Root cause.
+- Fix summary and changed files.
+- Verification commands.
+- Final status.
+
+Never include Agent Keys, owner private keys, owner JWTs, provider secrets, admin credentials, wallet seeds, raw authorization headers, or private Gateway payloads.

docs/evidence/staging-mcp-e2e-2026-05-13.md

@@ -0,0 +1,40 @@
+# Staging MCP E2E Evidence
+
+- Date: 2026-05-13
+- Environment: staging
+- Credential source: Google Secret Manager `synapse-staging-e2e-agent-credential`
+- Secret version used: latest after rotation
+- Service: `svc_synapse_echo`
+- Invocation: `inv-<redacted>`
+- Receipt status: `SUCCEEDED`
+
+## Command
+
+```bash
+SYNAPSE_E2E_AGENT_KEY_SOURCE=secret \
+SYNAPSE_ENV=staging \
+SYNAPSE_E2E_SERVICE_ID=svc_synapse_echo \
+SYNAPSE_E2E_PAYLOAD_JSON='{"message":"mcp staging e2e","source":"synapse-mcp-server"}' \
+SYNAPSE_E2E_COST_USDC='0.000000' \
+npm run test:e2e:staging
+```
+
+## Result
+
+```text
+Live MCP staging E2E credential source: secret-manager:synapse-staging-e2e-agent-credential
+Live MCP staging E2E selected service: svc_synapse_echo
+Live MCP staging E2E passed: svc_synapse_echo -> inv-<redacted> -> SUCCEEDED
+```
+
+## Checks Covered
+
+- MCP stdio server starts from built `dist/index.js`.
+- MCP tool list includes `discover_services`, `invoke_and_pay`, and `get_receipt`.
+- `discover_services` can find the staging smoke service.
+- `invoke_and_pay` can invoke the staging smoke service using the Secret Manager Agent Key.
+- `get_receipt` returns the same invocation id with a terminal success status.
+- Money fields remain strings when present.
+- E2E child-process environment is sanitized and does not inherit owner/provider/admin/private-key variables.
+
+No Agent Key, owner private key, owner JWT, provider secret, admin credential, wallet seed, raw authorization header, or private Gateway payload is included in this evidence.

llms.txt

@@ -150,7 +150,7 @@ Do not add public documentation that requires developers to run a Synapse Gatewa
 - Production Agent workflows should use `SYNAPSE_ENV=prod`; staging is only for preview/E2E validation, not production usage.
 - `npm run verify:mcp` runs typecheck, unit tests, build, and mock MCP E2E.
 - `npm run ci:quality` runs source-shape and quality-budget gates.
-- `npm run test:e2e:staging` requires a staging `SYNAPSE_AGENT_KEY=agt_xxx` and verifies the real staging Gateway over MCP stdio for preview smoke testing.
+- `npm run test:e2e:staging` verifies the real staging Gateway over MCP stdio for preview smoke testing. Maintainer staging E2E reads `SYNAPSE_AGENT_KEY` from Google Secret Manager secret `synapse-staging-e2e-agent-credential` by default; external developers can set `SYNAPSE_E2E_AGENT_KEY_SOURCE=env` and provide their own `SYNAPSE_AGENT_KEY=agt_xxx`.
 - Staging E2E performs broad discovery by default, sorts by lowest price, then selects a free fixed-price service by inspecting price fields. `SYNAPSE_E2E_QUERY` is optional service intent text, not a price filter. Legacy price-only values such as `free` are treated as broad discovery.
 - `npm run test:e2e:prod` is explicit-only production validation and must not be part of default CI.
 

scripts/e2e/agent-key-resolver.mjs

@@ -0,0 +1,48 @@
+import { execFileSync } from "node:child_process";
+
+export const DEFAULT_STAGING_AGENT_KEY_SECRET = "synapse-staging-e2e-agent-credential";
+
+export function resolveLiveAgentKey(profile, env = process.env, secretReader = readGcloudSecret) {
+  if (profile !== "staging" || env.SYNAPSE_E2E_AGENT_KEY_SOURCE === "env") {
+    return agentKeyFromEnv(env);
+  }
+
+  const secretName = env.SYNAPSE_E2E_AGENT_KEY_SECRET?.trim() || DEFAULT_STAGING_AGENT_KEY_SECRET;
+  const projectId = env.SYNAPSE_E2E_SECRET_PROJECT?.trim() || env.GCP_PROJECT_ID?.trim() || env.GOOGLE_CLOUD_PROJECT?.trim();
+  try {
+    const agentKey = secretReader(secretName, projectId);
+    validateAgentKey(agentKey);
+    return { agentKey, source: `secret-manager:${secretName}` };
+  } catch (error) {
+    if (env.SYNAPSE_E2E_AGENT_KEY_SOURCE === "secret") {
+      throw new Error(`Unable to read staging Agent Key from Secret Manager secret '${secretName}': ${errorMessage(error)}`);
+    }
+    return agentKeyFromEnv(env);
+  }
+}
+
+function agentKeyFromEnv(env) {
+  const agentKey = env.SYNAPSE_AGENT_KEY?.trim();
+  if (!agentKey) throw new Error("SYNAPSE_AGENT_KEY=agt_xxx is required for live MCP E2E.");
+  validateAgentKey(agentKey);
+  return { agentKey, source: "env:SYNAPSE_AGENT_KEY" };
+}
+
+function validateAgentKey(agentKey) {
+  if (!agentKey.startsWith("agt_")) {
+    throw new Error("SYNAPSE_AGENT_KEY must start with agt_.");
+  }
+}
+
+function readGcloudSecret(secretName, projectId) {
+  const args = ["secrets", "versions", "access", "latest", `--secret=${secretName}`];
+  if (projectId) args.push(`--project=${projectId}`);
+  return execFileSync("gcloud", args, {
+    encoding: "utf8",
+    stdio: ["ignore", "pipe", "pipe"]
+  }).trim();
+}
+
+function errorMessage(error) {
+  return error instanceof Error ? error.message : String(error);
+}

scripts/e2e/live.mjs

@@ -1,14 +1,18 @@
 #!/usr/bin/env node
+import { resolveLiveAgentKey } from "./agent-key-resolver.mjs";
 import { assert, assertToolList, asRecord, callToolData, withMcpClient } from "./mcp-client-helpers.mjs";
 
 const profile = process.argv[2] ?? "staging";
 if (!["local", "staging", "prod"].includes(profile)) {
   fail(`Unsupported E2E profile '${profile}'. Expected local, staging, or prod.`);
 }
 
-const agentKey = process.env.SYNAPSE_AGENT_KEY?.trim();
-if (!agentKey) fail("SYNAPSE_AGENT_KEY=agt_xxx is required for live MCP E2E.");
-if (!agentKey.startsWith("agt_")) fail("SYNAPSE_AGENT_KEY must start with agt_.");
+let credential;
+try {
+  credential = resolveLiveAgentKey(profile);
+} catch (error) {
+  fail(error.message);
+}
 
 if (profile === "prod" && process.env.SYNAPSE_ENV !== "prod") {
   fail("Prod E2E must be explicit: set SYNAPSE_ENV=prod and run npm run test:e2e:prod.");
@@ -23,7 +27,7 @@ const explicitMaxCostUsdc = process.env.SYNAPSE_E2E_MAX_COST_USDC?.trim();
 const payload = parsePayload(process.env.SYNAPSE_E2E_PAYLOAD_JSON);
 const idempotencyKey = process.env.SYNAPSE_E2E_IDEMPOTENCY_KEY?.trim() || `mcp-${profile}-e2e-${Date.now()}`;
 const env = {
-  SYNAPSE_AGENT_KEY: agentKey,
+  SYNAPSE_AGENT_KEY: credential.agentKey,
   SYNAPSE_ENV: profile,
   SYNAPSE_TIMEOUT_MS: process.env.SYNAPSE_TIMEOUT_MS || "60000"
 };
@@ -64,6 +68,7 @@ await withMcpClient(env, async (client) => {
     invokeArgs.costUsdc = costUsdc;
   }
 
+  console.log(`Live MCP ${profile} E2E credential source: ${credential.source}`);
   console.log(`Live MCP ${profile} E2E selected service: ${serviceId}`);
   const invokeData = asRecord(await callToolData(client, "invoke_and_pay", invokeArgs), "invoke_and_pay data");
   const gatewayResult = asRecord(invokeData.gateway ?? invokeData, "invoke_and_pay gateway result");

scripts/e2e/mcp-client-helpers.mjs

@@ -3,11 +3,33 @@ import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js"
 
 export const EXPECTED_TOOLS = ["discover_services", "get_receipt", "invoke_and_pay"];
 
+const BASE_ENV_ALLOWLIST = new Set([
+  "PATH",
+  "HOME",
+  "USER",
+  "TMPDIR",
+  "TMP",
+  "TEMP",
+  "SHELL",
+  "SystemRoot",
+  "ComSpec",
+  "PATHEXT",
+  "SSL_CERT_FILE",
+  "NODE_EXTRA_CA_CERTS"
+]);
+
+const SYNAPSE_CHILD_ENV_ALLOWLIST = new Set([
+  "SYNAPSE_AGENT_KEY",
+  "SYNAPSE_ENV",
+  "SYNAPSE_GATEWAY_URL",
+  "SYNAPSE_TIMEOUT_MS"
+]);
+
 export async function withMcpClient(env, fn) {
   const transport = new StdioClientTransport({
     command: process.execPath,
     args: ["dist/index.js"],
-    env: { ...process.env, ...env },
+    env: buildMcpChildEnv(env),
     stderr: "pipe"
   });
   const client = new Client({ name: "synapse-e2e-client", version: "0.1.0" });
@@ -68,3 +90,32 @@ export function asRecord(value, label) {
   }
   return value;
 }
+
+export function buildMcpChildEnv(overrides = {}, baseEnv = process.env) {
+  const childEnv = {};
+  for (const [key, value] of Object.entries(baseEnv)) {
+    if (isAllowedBaseEnv(key) || key.startsWith("SYNAPSE_E2E_")) {
+      assignStringEnv(childEnv, key, value);
+    }
+  }
+  for (const [key, value] of Object.entries(overrides)) {
+    if (isAllowedOverrideEnv(key)) {
+      assignStringEnv(childEnv, key, value);
+    }
+  }
+  return childEnv;
+}
+
+function isAllowedBaseEnv(key) {
+  return BASE_ENV_ALLOWLIST.has(key);
+}
+
+function isAllowedOverrideEnv(key) {
+  return SYNAPSE_CHILD_ENV_ALLOWLIST.has(key) || key.startsWith("SYNAPSE_E2E_") || BASE_ENV_ALLOWLIST.has(key);
+}
+
+function assignStringEnv(target, key, value) {
+  if (typeof value === "string" && value.length > 0) {
+    target[key] = value;
+  }
+}