feat(benchmarks): implement Kaggle client (push/run functionality)

[dolaameng]

Benchmarks CLI Reference (push & run)

The benchmarks CLI manages benchmark tasks — registering evaluation code, scheduling runs against models, monitoring progress, and downloading results.

Aliases: kaggle benchmarks or kaggle b

All task subcommands are under kaggle benchmarks tasks (alias: kaggle b t).

---

Commands

push — Register a task

Upload a Python source file as a benchmark task definition. The file is expected to be a .py file with percent delimiters (e.g., # %%). The CLI converts it to an .ipynb file before uploading. If the task already exists, it creates a new version.

kaggle b t push <task> -f <file>

Parameter	Flag	Required	Description
task	positional	Yes	Task name (e.g. math-eval)
file	-f, --file	Yes	Path to the Python source file defining the task

Behavior:

1. Validates the file exists and has a .py extension.
2. Reads the source file and parses it with Python's ast module to extract task names from @task decorators (supports both @task and @kbench.task styles, as well as @task(name="...") with explicit names).
   • When an explicit name= keyword is provided, that name is used.
   • When no explicit name is provided, the function name is title-cased with underscores replaced by spaces (e.g. my_test_task → "My Test Task").
3. Validates that the file contains at least one @task decorator. If none are found, raises ValueError and stops.
4. Validates that the given task name matches one of the task names extracted from the file.
5. Converts the .py file content to .ipynb format (Jupyter Notebook) using jupytext (assuming percent format), and adds a Python 3 kernelspec to the notebook metadata.
6. Checks the server for an existing task with the same slug:
   • If the task exists and its creation_state is QUEUED or RUNNING (i.e. a previous version is still being built), the push is rejected with ValueError.
   • If the task exists and is in COMPLETED or ERRORED state, the push proceeds (creates a new version).
   • If the task does not exist (404), the push proceeds (creates a new task).
7. Sends the notebook content (JSON string) to create_benchmark_task.
8. If the server returns an error message in the response, raises ValueError with the error details.
9. On success, prints the task slug and its URL.

Errors:

• ValueError: File <path> does not exist — file path is invalid.
• ValueError: File <path> must be a .py file — file is not a Python file.
• ValueError: No @task decorators found in file <path>. The file must define at least one task. — the file does not contain any @task-decorated functions.
• ValueError: Task '<name>' not found in file <path>. Found tasks: ... — the task name doesn't match any @task-decorated function in the file.
• ValueError: Task '<name>' is currently being created (pending). Cannot push now. — a previous version of this task is still being processed by the server.
• ValueError: Failed to push task: <error> — the server returned an error message in the response.
• HTTPError — server-side error (e.g. authentication failure, permission denied).

Example:

kaggle b t push math-eval -f tasks/math_eval.py

run — Schedule task runs

Schedule benchmark task execution against one or more models.

kaggle b t run <task> [-m <model> ...] [--wait]

Parameter	Flag	Required	Description
task	positional	Yes	Task name (e.g. math-eval)
model	-m, --model	No	Model slug(s) to run against. Accepts multiple space-separated values
wait	--wait	No	Wait for runs to complete. Can specify a timeout in seconds (0 or omit = indefinite)
poll_interval	--poll-interval	No	Seconds between status polls when using --wait (default: 10)

Behavior:

1. Task readiness check: Before scheduling, verifies that the task exists and its creation_state is COMPLETED. If the task is not ready:

   • For ERRORED tasks, the error message includes the task info for debugging.
   • For other non-completed states (e.g. QUEUED, RUNNING), raises ValueError indicating the task is not ready to run.

2. Model selection: If no -m is provided, fetches the list of available benchmark models via list_benchmark_models and prompts the user interactively:

   No model specified. 5 model(s) available:
     1. gemini-pro (Gemini Pro)
     2. gemma-2b (Gemma 2B)
   Enter model numbers (comma-separated), 'all':
   

   • Enter comma-separated numbers (e.g. 1,3) to select specific models.
   • Enter all to run against every available model.
   • When there are more than 20 models, the list is paginated. Use n for next page and p for previous page.
   • Invalid input (non-numeric, out-of-range index) raises ValueError.
   • If no benchmark models exist on the server, raises ValueError: No benchmark models available. Cannot schedule runs.

3. Scheduling: Calls batch_schedule_benchmark_task_runs with the task slug and selected model slugs. Output:

   Submitted run(s) for task 'math-eval'.
     gemini-pro: Scheduled
     gemma-2b: Scheduled
     gemini-flash: Skipped (<reason>)
   

4. Waiting (--wait): After scheduling, if --wait is specified, polls list_benchmark_task_runs at a fixed interval (default 10 seconds, configurable via --poll-interval) until all runs reach a terminal state (COMPLETED or ERRORED) or the timeout is reached. Output while waiting:

   Waiting for run(s) to complete...
     2 run(s) still in progress...
     1 run(s) still in progress...
   All runs completed:
     gemini-pro: COMPLETED
     gemma-2b: ERRORED
   

   • If a timeout (in seconds) is specified and reached, it stops waiting and prints: Timed out waiting for runs after <timeout> seconds.
   • If 0 or no value is specified for --wait, it waits indefinitely.

Errors:

• ValueError: Task '<name>' is not ready to run (status: <state>). Only completed tasks can be run. — the task has not finished building (or errored during build).
• ValueError: No benchmark models available. Cannot schedule runs. — no models exist on the server and none were specified via -m.
• ValueError: Invalid selection: <input> — the user entered non-numeric or out-of-range input during interactive model selection.
• HTTPError — server-side error (task not found, authentication failure, etc.).

Examples:

# Run against specific models
kaggle b t run math-eval -m gemini-pro gemma-2b

# Run and wait for completion
kaggle b t run math-eval -m gemini-pro --wait

# Wait with a custom poll interval (30 seconds)
kaggle b t run math-eval -m gemini-pro --wait --poll-interval 30

# Wait with a timeout (60 seconds)
kaggle b t run math-eval -m gemini-pro --wait 60

# Interactive model selection (prompts user)
kaggle b t run math-eval

---

End to end test

https://paste.googleplex.com/6483737513689088

[andrewmwang]

Nice!

[rosbo]

Great work. Just a few small comments.

[rosbo]

It supports creating or updating an existing task. Not sure if register is the best verb here... You could use "create or update a task ...".

[dolaameng]

Good point. Done.

[rosbo]

Why are you passing "task" as a parameter to the CLI method instead of reading it from the Python file?

[dolaameng]

Yes it's because currently the users can define multiple tasks in a single file. We need user to decide which one to create.

[rosbo]

Does this get imported automatically by pytest?

[dolaameng]

Yes. I think pytest auto-imports it for the test in the same dir. This is actually to avoid the import error from _introspect_token. The old username and password works just fine.