OpenAI File Search + Braintrust Remote Eval

Client-ready demo for running OpenAI file_search through a Braintrust remote eval and testing it in Playground.

Repo contents

• Remote eval (Python): src/eval_file_search.py
• Optional dataset seeder (TypeScript): src/createDataset.ts

Prerequisites

• Python 3.10+
• Node.js 18+ (only for the dataset seeder script)
• Braintrust API key
• OpenAI API key
• OpenAI vector store ID (vs_...)

Setup

cp .env.example .env
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip braintrust openai python-dotenv pydantic
npm install

Fill in .env:

• BRAINTRUST_API_KEY
• BRAINTRUST_PROJECT
• BRAINTRUST_DATASET
• OPENAI_API_KEY
• OPENAI_VECTOR_STORE_ID

Seed dataset

npm run create-dataset

Run remote eval server (for Playground)

npm run eval

This starts Braintrust in --dev mode using src/eval_file_search.py.

Push eval

npm run eval:push

Parameters available in Playground

• model (default: gpt-4o)
• system_prompt
• max_results (default: 15)
• vector_store_id (falls back to OPENAI_VECTOR_STORE_ID)
• debug (returns structured diagnostic payload)

Implementation note

The eval uses client.responses.create(...) with tools=[{"type": "file_search", ...}], which is the supported path for built-in file search tool usage.