Skip to content

Feature: Validate custom GitHub prompts against parser schema #12

@mrjcleaver

Description

@mrjcleaver

Problem

When users configure custom prompts in their GitHub repository, there's no validation to ensure the prompt output matches the format expected by the response parser.

This leads to "Summary could not be extracted from response" errors when the custom prompt returns a different JSON structure.

Example

Custom prompt returned:

{
  "🎯 Overview": "...",
  "💡 Key Technical Points": [...],
  "🔧 Action Items": [...]
}

Parser expected:

{
  "summary": "...",
  "key_points": [...],
  "action_items": [...]
}

Proposed Solution

Add a validation step when loading custom prompts from GitHub:

  1. Schema Validator: Create a JSON schema for the expected output format
  2. Prompt Testing: When a custom prompt is loaded, include instructions about the required output format
  3. Validation Endpoint: Add a /config validate-prompt command that tests the prompt with sample data
  4. Documentation: Update GitHub prompt template with clear schema requirements

Alternative

Make the parser more flexible to handle multiple JSON formats (backward compatible approach).

Priority

Medium - affects users who want to customize prompts but creates poor UX when format doesn't match

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions