Feature Request: Support rich metadata in scorer return values

[alfakini]

Hey! I'm not sure if this is currently possible, so I'm opening this issue to ask.

I'd like my scorer to return rich metadata alongside the score. Currently, it seems like we can only return primitives (Int, Float) or a Hash that gets converted to a float score. What I'd love to be able to do is return a structured result like this:

return {
  score: 0.5,
  metadata: {
    failure_type: "missing_argument_keys",
    reason: "Correct tool \"#{expected_name}\" but missing #{missing_keys.length} expected key(s): #{missing_keys.join(", ")}",
    expected_name: expected_name,
    actual_name: actual_name,
    mismatched_arguments: ...
    }
  }
}

I would like this to be accessible on the Experiment interface.

This would be really useful for debugging and analysis. Being able to attach structured context (failure reasons, mismatched values, etc.) directly to a score makes it much easier to understand why a score is what it is, and also create filters in the Experiment to inspect a subset of failures.

Is there a way to achieve this today that I'm missing? If not, would you be open to supporting this kind of structured return value from Scorers?

Thanks!

[delner]

Did some investigation into this. Braintrust does support submitting metadata as a part of a score, however the Ruby SDK doesn't implement this right now. I'll work on a PR to add support for this. When implemented, you should be able to return a Hash with score and metadata keys (like you have above) and those should become tags on the scorer span.

I'll keep you posted.

[delner]

Okay, opened #120 to address this, but have to merge some other changes first. Feel free to review/try the PR.

[delner]

This was released in 0.3.0. Closing this issue!