Skip to content

Eval: parseCLIOutput silently swallows non-JSON lines #56

@olaservo

Description

@olaservo

Problem

In `eval.ts`, the `parseCLIOutput()` function has an empty catch block (lines 189-192) that silently discards any non-JSON lines from CLI output:

```typescript
} catch {
// Not JSON, might be plain text or partial line
// Skip non-JSON lines in stream-json mode
}
```

If the CLI output format changes, includes debug messages, or produces malformed JSON, there's no diagnostic output to help debug the issue.

Suggestion

Add a `--verbose` or `--debug` flag that, when enabled, logs skipped lines:

```typescript
} catch {
if (verbose) {
console.error(`[debug] Skipped non-JSON line: ${line.substring(0, 100)}`);
}
}
```

Files

  • `evals/eval.ts` (parseCLIOutput, lines 147-200)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions