diff --git a/README.md b/README.md index 9bc0830..63a3165 100644 --- a/README.md +++ b/README.md @@ -208,6 +208,17 @@ discover_services -> invoke_and_pay -> get_receipt Use staging only for preview validation and live E2E smoke tests. Do not use staging for production Agent workflows. ```bash +export SYNAPSE_ENV=staging + +npm run test:e2e:staging +``` + +For SynapseNetwork maintainers, staging E2E reads `SYNAPSE_AGENT_KEY` from Google Secret Manager by default using the `synapse-staging-e2e-agent-credential` secret. Set `SYNAPSE_E2E_SECRET_PROJECT` when the active `gcloud` project is not the staging secrets project. + +Developers who are not using SynapseNetwork's GCP project can force a local Agent Key: + +```bash +export SYNAPSE_E2E_AGENT_KEY_SOURCE=env export SYNAPSE_AGENT_KEY=agt_xxx export SYNAPSE_ENV=staging @@ -219,7 +230,6 @@ By default, staging E2E performs broad discovery with `sort=lowest_price`, then Specified staging service: ```bash -export SYNAPSE_AGENT_KEY=agt_xxx export SYNAPSE_ENV=staging export SYNAPSE_E2E_SERVICE_ID=svc_quotes_famous_top3 export SYNAPSE_E2E_PAYLOAD_JSON='{"topic":"agent payments"}' @@ -231,7 +241,6 @@ npm run test:e2e:staging Token-metered staging service: ```bash -export SYNAPSE_AGENT_KEY=agt_xxx export SYNAPSE_ENV=staging export SYNAPSE_E2E_SERVICE_ID=svc_deepseek_chat export SYNAPSE_E2E_PAYLOAD_JSON='{"messages":[{"role":"user","content":"hello"}],"max_tokens":32}' diff --git a/docs/bugfix/2026-05-12-staging-e2e-credential-invalid.md b/docs/bugfix/2026-05-12-staging-e2e-credential-invalid.md new file mode 100644 index 0000000..2626b7e --- /dev/null +++ b/docs/bugfix/2026-05-12-staging-e2e-credential-invalid.md @@ -0,0 +1,83 @@ +# Staging MCP E2E Credential Blocked + +- Date: 2026-05-12 +- Environment profile: staging +- Command: `SYNAPSE_ENV=staging SYNAPSE_E2E_SERVICE_ID=svc_synapse_echo SYNAPSE_E2E_PAYLOAD_JSON='{"message":"mcp staging e2e","source":"synapse-mcp-server"}' SYNAPSE_E2E_COST_USDC='0.000000' npm run test:e2e:staging` +- Status: fixed on 2026-05-13 + +## Sanitized Failure + +The MCP server started over stdio, listed tools, discovered `svc_synapse_echo`, and selected it for invocation. + +`invoke_and_pay` failed with: + +```json +{ + "status": 401, + "code": "CREDENTIAL_INVALID", + "message": "Credential is invalid" +} +``` + +A second run mapped the locally configured legacy `SYNAPSE_API_KEY` value into `SYNAPSE_AGENT_KEY`; it failed with the same sanitized Gateway error. + +After updating staging E2E to read `SYNAPSE_AGENT_KEY` from Google Secret Manager secret `synapse-staging-e2e-agent-credential`, the MCP server again started over stdio, listed tools, discovered `svc_synapse_echo`, and selected it for invocation. + +`invoke_and_pay` then failed with: + +```json +{ + "status": 401, + "code": "CREDENTIAL_INACTIVE", + "message": "Credential is inactive" +} +``` + +## Root Cause + +The current local Agent Key candidates are not accepted by the staging Gateway for paid invocation, and the Secret Manager-backed staging E2E credential is inactive in Gateway. The MCP adapter is still forwarding the credential as `X-Credential`; local unit tests and mock E2E cover this request construction. + +Secret Manager checks confirmed that `synapse-staging-e2e-agent-credential` exists and its latest version is enabled. The remaining issue is Gateway credential lifecycle state, not Secret Manager access. + +## Fix Summary + +- Hardened live MCP E2E stdio child-process env handling so owner/provider/admin/private-key environment variables are not inherited by the MCP server process. +- Added a regression test proving sensitive Synapse owner/provider/admin variables are dropped. +- Added this bugfix record directory for live E2E failures. +- Updated staging E2E to read its maintainer Agent Key from Secret Manager by default. + +## Verification Completed + +```bash +npm test +npm run verify:mcp +npm run ci:quality +npm run smoke:cli +npm pack --dry-run +SYNAPSE_E2E_AGENT_KEY_SOURCE=secret SYNAPSE_ENV=staging SYNAPSE_E2E_SERVICE_ID=svc_synapse_echo SYNAPSE_E2E_PAYLOAD_JSON='{"message":"mcp staging e2e","source":"synapse-mcp-server"}' SYNAPSE_E2E_COST_USDC='0.000000' npm run test:e2e:staging +``` + +## Required Follow-Up + +The staging Agent Key stored in Secret Manager secret `synapse-staging-e2e-agent-credential` was rotated by issuing a new dedicated credential named `mcp-staging-e2e-` and writing it as a new Secret Manager version. + +Final rerun: + +```bash +SYNAPSE_E2E_AGENT_KEY_SOURCE=secret \ +SYNAPSE_ENV=staging \ +SYNAPSE_E2E_SERVICE_ID=svc_synapse_echo \ +SYNAPSE_E2E_PAYLOAD_JSON='{"message":"mcp staging e2e","source":"synapse-mcp-server"}' \ +SYNAPSE_E2E_COST_USDC='0.000000' \ +npm run test:e2e:staging +``` + +Result: + +```text +Live MCP staging E2E credential source: secret-manager:synapse-staging-e2e-agent-credential +Live MCP staging E2E selected service: svc_synapse_echo +Live MCP staging E2E passed: svc_synapse_echo -> inv- -> SUCCEEDED +``` + +The MCP staging live E2E release gate is unblocked. diff --git a/docs/bugfix/README.md b/docs/bugfix/README.md new file mode 100644 index 0000000..e59b820 --- /dev/null +++ b/docs/bugfix/README.md @@ -0,0 +1,15 @@ +# Bugfix Records + +Use this directory for live MCP E2E failures that must be fixed before release. + +Each record should include: + +- Date and environment profile. +- Command that failed, with secrets redacted. +- Sanitized error output. +- Root cause. +- Fix summary and changed files. +- Verification commands. +- Final status. + +Never include Agent Keys, owner private keys, owner JWTs, provider secrets, admin credentials, wallet seeds, raw authorization headers, or private Gateway payloads. diff --git a/docs/evidence/staging-mcp-e2e-2026-05-13.md b/docs/evidence/staging-mcp-e2e-2026-05-13.md new file mode 100644 index 0000000..2756264 --- /dev/null +++ b/docs/evidence/staging-mcp-e2e-2026-05-13.md @@ -0,0 +1,40 @@ +# Staging MCP E2E Evidence + +- Date: 2026-05-13 +- Environment: staging +- Credential source: Google Secret Manager `synapse-staging-e2e-agent-credential` +- Secret version used: latest after rotation +- Service: `svc_synapse_echo` +- Invocation: `inv-` +- Receipt status: `SUCCEEDED` + +## Command + +```bash +SYNAPSE_E2E_AGENT_KEY_SOURCE=secret \ +SYNAPSE_ENV=staging \ +SYNAPSE_E2E_SERVICE_ID=svc_synapse_echo \ +SYNAPSE_E2E_PAYLOAD_JSON='{"message":"mcp staging e2e","source":"synapse-mcp-server"}' \ +SYNAPSE_E2E_COST_USDC='0.000000' \ +npm run test:e2e:staging +``` + +## Result + +```text +Live MCP staging E2E credential source: secret-manager:synapse-staging-e2e-agent-credential +Live MCP staging E2E selected service: svc_synapse_echo +Live MCP staging E2E passed: svc_synapse_echo -> inv- -> SUCCEEDED +``` + +## Checks Covered + +- MCP stdio server starts from built `dist/index.js`. +- MCP tool list includes `discover_services`, `invoke_and_pay`, and `get_receipt`. +- `discover_services` can find the staging smoke service. +- `invoke_and_pay` can invoke the staging smoke service using the Secret Manager Agent Key. +- `get_receipt` returns the same invocation id with a terminal success status. +- Money fields remain strings when present. +- E2E child-process environment is sanitized and does not inherit owner/provider/admin/private-key variables. + +No Agent Key, owner private key, owner JWT, provider secret, admin credential, wallet seed, raw authorization header, or private Gateway payload is included in this evidence. diff --git a/llms.txt b/llms.txt index 86adad4..f42d077 100644 --- a/llms.txt +++ b/llms.txt @@ -150,7 +150,7 @@ Do not add public documentation that requires developers to run a Synapse Gatewa - Production Agent workflows should use `SYNAPSE_ENV=prod`; staging is only for preview/E2E validation, not production usage. - `npm run verify:mcp` runs typecheck, unit tests, build, and mock MCP E2E. - `npm run ci:quality` runs source-shape and quality-budget gates. -- `npm run test:e2e:staging` requires a staging `SYNAPSE_AGENT_KEY=agt_xxx` and verifies the real staging Gateway over MCP stdio for preview smoke testing. +- `npm run test:e2e:staging` verifies the real staging Gateway over MCP stdio for preview smoke testing. Maintainer staging E2E reads `SYNAPSE_AGENT_KEY` from Google Secret Manager secret `synapse-staging-e2e-agent-credential` by default; external developers can set `SYNAPSE_E2E_AGENT_KEY_SOURCE=env` and provide their own `SYNAPSE_AGENT_KEY=agt_xxx`. - Staging E2E performs broad discovery by default, sorts by lowest price, then selects a free fixed-price service by inspecting price fields. `SYNAPSE_E2E_QUERY` is optional service intent text, not a price filter. Legacy price-only values such as `free` are treated as broad discovery. - `npm run test:e2e:prod` is explicit-only production validation and must not be part of default CI. diff --git a/scripts/e2e/agent-key-resolver.mjs b/scripts/e2e/agent-key-resolver.mjs new file mode 100644 index 0000000..2cb25b8 --- /dev/null +++ b/scripts/e2e/agent-key-resolver.mjs @@ -0,0 +1,48 @@ +import { execFileSync } from "node:child_process"; + +export const DEFAULT_STAGING_AGENT_KEY_SECRET = "synapse-staging-e2e-agent-credential"; + +export function resolveLiveAgentKey(profile, env = process.env, secretReader = readGcloudSecret) { + if (profile !== "staging" || env.SYNAPSE_E2E_AGENT_KEY_SOURCE === "env") { + return agentKeyFromEnv(env); + } + + const secretName = env.SYNAPSE_E2E_AGENT_KEY_SECRET?.trim() || DEFAULT_STAGING_AGENT_KEY_SECRET; + const projectId = env.SYNAPSE_E2E_SECRET_PROJECT?.trim() || env.GCP_PROJECT_ID?.trim() || env.GOOGLE_CLOUD_PROJECT?.trim(); + try { + const agentKey = secretReader(secretName, projectId); + validateAgentKey(agentKey); + return { agentKey, source: `secret-manager:${secretName}` }; + } catch (error) { + if (env.SYNAPSE_E2E_AGENT_KEY_SOURCE === "secret") { + throw new Error(`Unable to read staging Agent Key from Secret Manager secret '${secretName}': ${errorMessage(error)}`); + } + return agentKeyFromEnv(env); + } +} + +function agentKeyFromEnv(env) { + const agentKey = env.SYNAPSE_AGENT_KEY?.trim(); + if (!agentKey) throw new Error("SYNAPSE_AGENT_KEY=agt_xxx is required for live MCP E2E."); + validateAgentKey(agentKey); + return { agentKey, source: "env:SYNAPSE_AGENT_KEY" }; +} + +function validateAgentKey(agentKey) { + if (!agentKey.startsWith("agt_")) { + throw new Error("SYNAPSE_AGENT_KEY must start with agt_."); + } +} + +function readGcloudSecret(secretName, projectId) { + const args = ["secrets", "versions", "access", "latest", `--secret=${secretName}`]; + if (projectId) args.push(`--project=${projectId}`); + return execFileSync("gcloud", args, { + encoding: "utf8", + stdio: ["ignore", "pipe", "pipe"] + }).trim(); +} + +function errorMessage(error) { + return error instanceof Error ? error.message : String(error); +} diff --git a/scripts/e2e/live.mjs b/scripts/e2e/live.mjs index 66ccbe4..70b3de8 100755 --- a/scripts/e2e/live.mjs +++ b/scripts/e2e/live.mjs @@ -1,4 +1,5 @@ #!/usr/bin/env node +import { resolveLiveAgentKey } from "./agent-key-resolver.mjs"; import { assert, assertToolList, asRecord, callToolData, withMcpClient } from "./mcp-client-helpers.mjs"; const profile = process.argv[2] ?? "staging"; @@ -6,9 +7,12 @@ if (!["local", "staging", "prod"].includes(profile)) { fail(`Unsupported E2E profile '${profile}'. Expected local, staging, or prod.`); } -const agentKey = process.env.SYNAPSE_AGENT_KEY?.trim(); -if (!agentKey) fail("SYNAPSE_AGENT_KEY=agt_xxx is required for live MCP E2E."); -if (!agentKey.startsWith("agt_")) fail("SYNAPSE_AGENT_KEY must start with agt_."); +let credential; +try { + credential = resolveLiveAgentKey(profile); +} catch (error) { + fail(error.message); +} if (profile === "prod" && process.env.SYNAPSE_ENV !== "prod") { fail("Prod E2E must be explicit: set SYNAPSE_ENV=prod and run npm run test:e2e:prod."); @@ -23,7 +27,7 @@ const explicitMaxCostUsdc = process.env.SYNAPSE_E2E_MAX_COST_USDC?.trim(); const payload = parsePayload(process.env.SYNAPSE_E2E_PAYLOAD_JSON); const idempotencyKey = process.env.SYNAPSE_E2E_IDEMPOTENCY_KEY?.trim() || `mcp-${profile}-e2e-${Date.now()}`; const env = { - SYNAPSE_AGENT_KEY: agentKey, + SYNAPSE_AGENT_KEY: credential.agentKey, SYNAPSE_ENV: profile, SYNAPSE_TIMEOUT_MS: process.env.SYNAPSE_TIMEOUT_MS || "60000" }; @@ -64,6 +68,7 @@ await withMcpClient(env, async (client) => { invokeArgs.costUsdc = costUsdc; } + console.log(`Live MCP ${profile} E2E credential source: ${credential.source}`); console.log(`Live MCP ${profile} E2E selected service: ${serviceId}`); const invokeData = asRecord(await callToolData(client, "invoke_and_pay", invokeArgs), "invoke_and_pay data"); const gatewayResult = asRecord(invokeData.gateway ?? invokeData, "invoke_and_pay gateway result"); diff --git a/scripts/e2e/mcp-client-helpers.mjs b/scripts/e2e/mcp-client-helpers.mjs index f260a78..51f4a0d 100644 --- a/scripts/e2e/mcp-client-helpers.mjs +++ b/scripts/e2e/mcp-client-helpers.mjs @@ -3,11 +3,33 @@ import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js" export const EXPECTED_TOOLS = ["discover_services", "get_receipt", "invoke_and_pay"]; +const BASE_ENV_ALLOWLIST = new Set([ + "PATH", + "HOME", + "USER", + "TMPDIR", + "TMP", + "TEMP", + "SHELL", + "SystemRoot", + "ComSpec", + "PATHEXT", + "SSL_CERT_FILE", + "NODE_EXTRA_CA_CERTS" +]); + +const SYNAPSE_CHILD_ENV_ALLOWLIST = new Set([ + "SYNAPSE_AGENT_KEY", + "SYNAPSE_ENV", + "SYNAPSE_GATEWAY_URL", + "SYNAPSE_TIMEOUT_MS" +]); + export async function withMcpClient(env, fn) { const transport = new StdioClientTransport({ command: process.execPath, args: ["dist/index.js"], - env: { ...process.env, ...env }, + env: buildMcpChildEnv(env), stderr: "pipe" }); const client = new Client({ name: "synapse-e2e-client", version: "0.1.0" }); @@ -68,3 +90,32 @@ export function asRecord(value, label) { } return value; } + +export function buildMcpChildEnv(overrides = {}, baseEnv = process.env) { + const childEnv = {}; + for (const [key, value] of Object.entries(baseEnv)) { + if (isAllowedBaseEnv(key) || key.startsWith("SYNAPSE_E2E_")) { + assignStringEnv(childEnv, key, value); + } + } + for (const [key, value] of Object.entries(overrides)) { + if (isAllowedOverrideEnv(key)) { + assignStringEnv(childEnv, key, value); + } + } + return childEnv; +} + +function isAllowedBaseEnv(key) { + return BASE_ENV_ALLOWLIST.has(key); +} + +function isAllowedOverrideEnv(key) { + return SYNAPSE_CHILD_ENV_ALLOWLIST.has(key) || key.startsWith("SYNAPSE_E2E_") || BASE_ENV_ALLOWLIST.has(key); +} + +function assignStringEnv(target, key, value) { + if (typeof value === "string" && value.length > 0) { + target[key] = value; + } +} diff --git a/test/agent-key-resolver.test.mjs b/test/agent-key-resolver.test.mjs new file mode 100644 index 0000000..4dcd927 --- /dev/null +++ b/test/agent-key-resolver.test.mjs @@ -0,0 +1,101 @@ +import { describe, expect, it } from "vitest"; + +import { DEFAULT_STAGING_AGENT_KEY_SECRET, resolveLiveAgentKey } from "../scripts/e2e/agent-key-resolver.mjs"; + +describe("resolveLiveAgentKey", () => { + it("reads staging Agent Key from Secret Manager by default", () => { + const calls = []; + const result = resolveLiveAgentKey( + "staging", + { + SYNAPSE_AGENT_KEY: "agt_env", + GCP_PROJECT_ID: "project-a" + }, + (secretName, projectId) => { + calls.push({ secretName, projectId }); + return "agt_secret"; + } + ); + + expect(result).toEqual({ + agentKey: "agt_secret", + source: `secret-manager:${DEFAULT_STAGING_AGENT_KEY_SECRET}` + }); + expect(calls).toEqual([{ secretName: DEFAULT_STAGING_AGENT_KEY_SECRET, projectId: "project-a" }]); + }); + + it("allows an explicit staging secret name and project", () => { + const result = resolveLiveAgentKey( + "staging", + { + SYNAPSE_E2E_AGENT_KEY_SECRET: "custom-agent-key", + SYNAPSE_E2E_SECRET_PROJECT: "project-b" + }, + (secretName, projectId) => { + expect(secretName).toBe("custom-agent-key"); + expect(projectId).toBe("project-b"); + return "agt_custom"; + } + ); + + expect(result).toEqual({ agentKey: "agt_custom", source: "secret-manager:custom-agent-key" }); + }); + + it("can force env Agent Key for staging developer overrides", () => { + const result = resolveLiveAgentKey( + "staging", + { + SYNAPSE_E2E_AGENT_KEY_SOURCE: "env", + SYNAPSE_AGENT_KEY: "agt_env" + }, + () => { + throw new Error("secret reader should not be called"); + } + ); + + expect(result).toEqual({ agentKey: "agt_env", source: "env:SYNAPSE_AGENT_KEY" }); + }); + + it("falls back to env when auto secret lookup fails", () => { + const result = resolveLiveAgentKey( + "staging", + { + SYNAPSE_AGENT_KEY: "agt_env" + }, + () => { + throw new Error("gcloud unavailable"); + } + ); + + expect(result).toEqual({ agentKey: "agt_env", source: "env:SYNAPSE_AGENT_KEY" }); + }); + + it("fails hard when secret source is required and Secret Manager lookup fails", () => { + expect(() => + resolveLiveAgentKey( + "staging", + { + SYNAPSE_AGENT_KEY: "agt_env", + SYNAPSE_E2E_AGENT_KEY_SOURCE: "secret" + }, + () => { + throw new Error("permission denied"); + } + ) + ).toThrow("Unable to read staging Agent Key from Secret Manager"); + }); + + it("uses env for non-staging profiles", () => { + const result = resolveLiveAgentKey( + "prod", + { + SYNAPSE_AGENT_KEY: "agt_prod" + }, + () => { + throw new Error("secret reader should not be called"); + } + ); + + expect(result).toEqual({ agentKey: "agt_prod", source: "env:SYNAPSE_AGENT_KEY" }); + }); +}); diff --git a/test/mcp-client-helpers.test.mjs b/test/mcp-client-helpers.test.mjs new file mode 100644 index 0000000..c18a3bc --- /dev/null +++ b/test/mcp-client-helpers.test.mjs @@ -0,0 +1,49 @@ +import { describe, expect, it } from "vitest"; + +import { buildMcpChildEnv } from "../scripts/e2e/mcp-client-helpers.mjs"; + +describe("buildMcpChildEnv", () => { + it("passes only MCP runtime env and drops owner/provider/admin secrets", () => { + const childEnv = buildMcpChildEnv( + { + SYNAPSE_AGENT_KEY: "agt_test", + SYNAPSE_ENV: "staging", + SYNAPSE_GATEWAY_URL: "https://api-staging.synapse-network.ai", + SYNAPSE_TIMEOUT_MS: "60000", + SYNAPSE_CONSUMER_OWNER_PRIVATE_KEY: "owner-secret-from-overrides", + SYNAPSE_PROVIDER_OWNER_PRIVATE_KEY: "provider-secret-from-overrides", + SYNAPSE_ADMIN_SECRET: "admin-secret-from-overrides" + }, + { + PATH: "/usr/bin", + HOME: "/tmp/test-home", + SYNAPSE_API_KEY: "legacy-agent-key", + SYNAPSE_CONSUMER_OWNER_PRIVATE_KEY: "owner-secret", + SYNAPSE_PROVIDER_OWNER_PRIVATE_KEY: "provider-secret", + SYNAPSE_OWNER_JWT: "owner-jwt", + SYNAPSE_PROVIDER_SECRET: "provider-secret", + SYNAPSE_ADMIN_CREDENTIAL: "admin-credential", + SYNAPSE_E2E_SERVICE_ID: "svc_synapse_echo", + RANDOM_ENV: "random" + } + ); + + expect(childEnv).toMatchObject({ + PATH: "/usr/bin", + HOME: "/tmp/test-home", + SYNAPSE_AGENT_KEY: "agt_test", + SYNAPSE_ENV: "staging", + SYNAPSE_GATEWAY_URL: "https://api-staging.synapse-network.ai", + SYNAPSE_TIMEOUT_MS: "60000", + SYNAPSE_E2E_SERVICE_ID: "svc_synapse_echo" + }); + expect(childEnv).not.toHaveProperty("SYNAPSE_API_KEY"); + expect(childEnv).not.toHaveProperty("SYNAPSE_CONSUMER_OWNER_PRIVATE_KEY"); + expect(childEnv).not.toHaveProperty("SYNAPSE_PROVIDER_OWNER_PRIVATE_KEY"); + expect(childEnv).not.toHaveProperty("SYNAPSE_OWNER_JWT"); + expect(childEnv).not.toHaveProperty("SYNAPSE_PROVIDER_SECRET"); + expect(childEnv).not.toHaveProperty("SYNAPSE_ADMIN_CREDENTIAL"); + expect(childEnv).not.toHaveProperty("SYNAPSE_ADMIN_SECRET"); + expect(childEnv).not.toHaveProperty("RANDOM_ENV"); + }); +}); diff --git a/vitest.config.ts b/vitest.config.ts index ed078a3..cfc7292 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -3,6 +3,6 @@ import { defineConfig } from "vitest/config"; export default defineConfig({ test: { environment: "node", - include: ["test/**/*.test.ts"] + include: ["test/**/*.test.ts", "test/**/*.test.mjs"] } });