Draft
Conversation
Replace explicit `--language` and `--runner` flags with automatic detection based on file extensions. Introduce separate `--runner-js` and `--runner-python` flags for explicit runner overrides. Hide the deprecated `--language` flag and maintain backward compatibility for the `BT_EVAL_RUNNER` environment variable via `--runner-js-legacy-env`. Update test fixtures to use `runners_js` and `runners_python` instead of `runtime` and `runners`.
|
Latest downloadable build artifacts for this PR commit
Available artifact names
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds support for running Python and JavaScript eval files together in a single
bt evalinvocation. Previously, mixing.pyand.ts/.jsfiles in one command would fail with a "mixed eval file types are not supported yet" error.What changed
Language inference & parallel execution (
src/eval.rs)--language/BT_EVAL_LANGUAGEflag (now hidden and emits a deprecation warning — the language is always inferred from file extensions).build_eval_plan(single plan) withpartition_files_by_language+build_eval_plans(one plan per language), which splits the input file list into JS and Python partitions.future::try_join_all; the combined exit status prefers the first failure.--runner/BT_EVAL_RUNNERflag into two language-scoped flags:--runner-js/BT_EVAL_JS_RUNNER(JS/TS runner, e.g.tsx,bun,deno)--runner-python/BT_EVAL_PYTHON_RUNNER(Python runner, e.g.python3,uv run python)BT_EVAL_RUNNERenv var is kept as a backward-compatible alias for--runner-js.std::env::var("BT_EVAL_PYTHON_RUNNER")lookup frombuild_python_command; the runner is now always passed through the CLI layer.--dev) path updated to carry separatejs_runner_override/python_runner_overridefields instead of a singlelanguage_override+runner_override.Test fixtures (
tests/eval_fixtures.rs,tests/evals/)fixture.jsonschema updated:runtime/runner/runnersfields replaced byrunners_jsandrunners_pythonarrays.js/andpy/) so the newmixed/category is picked up automatically.js_runner × py_runner), exercising every combination.tests/evals/mixed/mixed-py-js/fixture with a paired.eval.ts+eval_basic.pythat both write to the same project, verified againsttsx,bun, anddenoJS runners.partition_files_by_language,build_eval_plans, and theBT_EVAL_RUNNER→BT_EVAL_JS_RUNNERbackward-compat env alias.