Skip to content

feat: Agent output schema validation pipeline with ContractRegistry #384

@jafreck

Description

@jafreck

Summary

Add a ResultParser that validates agent output against AgentContract output schemas, closing the loop on the contract system that currently defines schemas but doesn't enforce them at parse time.

Motivation

The framework defines AgentContract<TInput, TOutput> with inputSchema and outputSchema (Zod types), and provides extractCadreJson() to pull structured JSON from agent output. But there's no integration between the two — extractCadreJson parses raw JSON without validating it against the contract's output schema.

This means:

  1. Agents can return structurally invalid output that passes extraction but fails downstream
  2. Consumers must manually wire extractCadreJson + Zod .parse() for each agent
  3. There's no centralized registry mapping agent names to their contracts for automatic validation

AAMF implements parseAamfOutput() which takes agent output + a schema registry, extracts the structured block, validates it against the per-agent schema, and returns a typed result or structured error. Every agent result goes through this pipeline, catching schema violations immediately.

Proposed API

Contract Registry

class ContractRegistry {
  /** Register an agent contract. */
  register<TInput, TOutput>(contract: AgentContract<TInput, TOutput>): void;

  /** Get the contract for a named agent. */
  get(agentName: string): AgentContract<unknown, unknown> | undefined;

  /** Check if a contract is registered for the given agent. */
  has(agentName: string): boolean;

  /** List all registered agent names. */
  list(): string[];
}

ResultParser

interface ParseResult<T> {
  success: boolean;
  /** Parsed and validated output, if successful. */
  data?: T;
  /** Raw extracted JSON before validation (for debugging). */
  raw?: unknown;
  /** Validation errors from the Zod schema. */
  validationErrors?: Array<{ path: string; message: string }>;
  /** Extraction error (no cadre-json block found, malformed JSON, etc.). */
  extractionError?: string;
}

class ResultParser {
  constructor(registry: ContractRegistry);

  /**
   * Parse agent output: extract cadre-json block, validate against the
   * agent's registered output schema, return typed result.
   *
   * @param agentName - The agent whose contract to validate against.
   * @param content - Raw agent output (stdout) containing a cadre-json block.
   * @returns Typed parse result with validation details.
   */
  parse<T>(agentName: string, content: string): ParseResult<T>;

  /**
   * Parse with an explicit schema (no registry lookup).
   * Useful for one-off parsing or agents not in the registry.
   */
  parseWithSchema<T>(content: string, schema: ZodType<T>): ParseResult<T>;
}

AgentResult Integration

interface AgentResult {
  // ... existing fields ...

  /** Whether the agent's output was successfully parsed and validated against its contract. */
  outputParsed: boolean;

  /** Parsed output data (when outputParsed is true). */
  parsedOutput?: unknown;

  /** Parse/validation errors (when outputParsed is false). */
  parseErrors?: Array<{ path: string; message: string }>;
}

Extraction Format Support

The existing extractCadreJson() handles cadre-json blocks. Add support for configurable block markers:

interface ExtractionOptions {
  /** Block marker name (default: 'cadre-json'). */
  blockMarker?: string;
  /** Whether to try JSON.parse on the entire content if no block is found. */
  fallbackToRawJson?: boolean;
}

function extractStructuredOutput<T>(
  content: string,
  schema: ZodType<T>,
  options?: ExtractionOptions,
): ParseResult<T>;

Implementation Notes

  • ResultParser is stateless beyond the registry reference — safe to share across concurrent invocations
  • The registry should be populated at initialization time from agent definitions, not lazily
  • parseWithSchema allows ad-hoc validation without a registry — useful for phase-level outputs that aren't tied to a specific agent
  • ExtractionOptions.blockMarker supports consumers who use different marker conventions (e.g., AAMF uses aamf-json)
  • Validation errors include Zod paths for precise diagnostics: "issues[0].severity: Expected 'critical' | 'major' | 'minor', received 'high'"
  • Consider emitting an event (agent-output-validation-failed) when parsing fails, for observability

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions