Support Playground request parameters in Remote Evals (parity with Python SDK)

[gabe-kent]

Use case

When running an evaluator from the Braintrust Playground, the UI sends a POST to /eval with:

• Evaluator name
• A dataset (e.g. dataset_id or dataset_name) as the data source
• Optional parameters from the Playground form (e.g. internal_model_id, definition, temperature, or other config)

The task should be able to use those parameter values (e.g. to load an internal model record or adjust behavior) while iterating over dataset rows. The Playground only runs against a dataset, so parameters are only useful if the task can read them.

---

Issue

Request parameter values are never passed into the evaluator task.

• The Eval handler in lib/braintrust/server/handlers/eval.rb never reads body["parameters"] and never passes them into evaluator.run.
• Evaluator#run and Braintrust::Eval.run have no parameters argument.
• The Runner in lib/braintrust/eval/runner.rb only calls task.call(test_case.input) — a single argument — so the task has no way to receive request-scoped parameters.
• Parameter definitions (schema) are already supported: the List handler exposes them for the Playground UI. Only the values sent in the POST body are unused.

So tasks cannot use Playground-supplied parameters when running against a dataset.

---

Why SDK changes are required

• With dataset_id (or dataset_name), the SDK resolves the data source and builds cases; the application never sees the per-row data.
• The app therefore cannot "merge" body["parameters"] into each row's input before the SDK runs. The only way to get request-scoped parameters into the task is for the SDK to read them from the request and pass them into the task (e.g. via a second argument).
• The Python SDK already does this: it reads and validates body["parameters"], passes them into the run, and gives the task a hooks object with hooks.parameters. The Ruby SDK should do the same so that Playground-driven evals can use parameters when running against a dataset.

---

Proposed changes (high level)

1. Eval handler — Read body["parameters"], optionally validate/apply defaults using the evaluator's parameter schema, and pass the result into evaluator.run (e.g. run_opts[:parameters]).
2. Evaluator#run and Braintrust::Eval.run — Add a parameters: keyword and pass it through to the Runner.
3. Runner — Accept parameters: in initialize. When invoking the task, if the task accepts two arguments (e.g. task.arity == 2), pass a second argument: a small hooks-like object that exposes the request parameters (e.g. hooks.parameters). Otherwise keep calling task.call(input) for backward compatibility.
4. New type — Introduce a minimal hooks object (e.g. Braintrust::Eval::Hooks) with a #parameters method returning the request parameter Hash, and pass one instance per run as the second argument to the task when it accepts two args.

After these changes, a task could use parameters the same way as in Python:

task: ->(input, hooks) {
  config = hooks.parameters
  # use config["internal_model_id"], config["temperature"], etc.
}

[delner]

Thanks for opening this! I think we should absolutely support parameters here.

Keep in mind, we actually just made some important (possibly timely!) changes to the Eval API that are yet unreleased with #111. Notably, we're moving to keyword args over positional args. I think this really helps your case here, because instead of dealing with the arity compatibility issue as you identified, users can define the args they want in any order. So it'd look more like task: ->(input:, hooks:) { } from your example, but task: ->(hooks:) { } or task: ->(input:) { } would be equally valid.

I'm not familiar with the Python implementation and hooks so this is something I need to wrap my head around a bit more. What are hooks and why should parameters be defined on them? Is this just a proxy object so we don't need to extend the arity of the blocks? If so, I think we could just do task: ->(input:, parameters:) directly with our new keyword arg syntax. In such case I don't think we'd support arity == 2, we'd just require users to use keyword args to get the benefit.

[gabe-kent]

I'm not familiar with the Python implementation and hooks so this is something I need to wrap my head around a bit more. What are hooks and why should parameters be defined on them? Is this just a proxy object so we don't need to extend the arity of the blocks? If so, I think we could just do task: ->(input:, parameters:) directly with our new keyword arg syntax. In such case I don't think we'd support arity == 2, we'd just require users to use keyword args to get the benefit.

TL;DR
I believe this would be sufficient: task: ->(input:, parameters:) 👍

High-level

In the Python SDK, hooks is a single context object passed to the task. It holds several things, not just parameters:

• parameters — validated request parameter values (the thing we care about for this issue)
• metadata — mutable dict the task can attach metadata to
• expected — expected value for the current case
• trial_index — for multi-trial evals
• tags — tags the task can read/append
• report_progress — callback for streaming progress (e.g. token-by-token) back to the Playground

So in Python, "parameters on hooks" is really "parameters are one field on the one context object we pass to the task." That way they can add more context later without changing the task signature.

For Ruby with keyword args, we don't need to mirror that whole object. Passing parameters: as its own keyword is simpler and fits Ruby well: e.g. task: ->(input:, parameters:) { ... }, with parameters a Hash or nil. If you later add things like report_progress or task-level metadata, you could introduce a hooks: keyword then or add more keywords. So I'd be fine with no hooks object and just parameters: for now. Requiring keyword args and not supporting positional arity is fine with me.

---

Python SDK reference (braintrust-sdk-python)

1. Dev server: read request params, validate, and pass into the run
py/src/braintrust/devserver/server.py

• Lines 143–147 — Read parameters from the parsed body, validate against the evaluator's parameter schema, return 400 on error:

  request_parameters = eval_data.get("parameters", {})
  validated_parameters = validate_parameters(request_parameters, evaluator.parameters)

• Lines 187–191 — Override the evaluator's params with the validated request params and pass into EvalAsync:

  eval_kwargs = {k: v for (k, v) in evaluator.__dict__.items() if ...}
  if validated_parameters is not None:
      eval_kwargs["parameters"] = validated_parameters

• Lines 155–169 — Wrapper that calls the user task with (input, hooks); the actual hooks instance is created inside the framework (see below).

2. Framework: build hooks and pass to the task
py/src/braintrust/framework.py

• Lines 1446–1452 — Create DictEvalHooks with parameters=evaluator.parameters (at run time this is the request's parameter values):

  hooks = DictEvalHooks(
      metadata,
      expected=datum.expected,
      trial_index=trial_index,
      tags=tags,
      report_progress=report_progress,
      parameters=evaluator.parameters,
  )

• Lines 1455–1464 — Decide whether to pass hooks by checking that the task has two parameters, then call the task:

  task_args = [datum.input]
  if len(inspect.signature(evaluator.task).parameters) == 2:
      task_args.append(hooks)
  # ...
  output = await await_or_run(event_loop, evaluator.task, *task_args)

3. Hooks class and parameters
py/src/braintrust/framework.py

• Lines 1147–1170 — DictEvalHooks definition and __init__ (stores parameters in self._parameters).
• Lines 1213–1214 — parameters property that returns that dict.

4. Request body parsing
py/src/braintrust/devserver/schemas.py

• Lines 206–211 — parse_eval_body includes parameters from the request in the parsed body.

5. Parameter validation (optional)
py/src/braintrust/parameters.py

• Lines 37–119 — validate_parameters(request_parameters, parameter_schema).

[delner]

hooks is a single context object passed to the task. It holds several things, not just parameters

Cool, this is helpful and confirms my assumption. I actually entertained doing this design (having an args object) as a means of side-stepping the arity-expansion problem, but decided that args.input, args.output etc just makes things that much wordier, hence I opted for the keyword arg style.

So it sounds like then we just have to introduce the parameters: arg to the Task block which should be relatively straightforward. I'll add this to the TODO list!