Easier way to handle multiple Braintrust projects in the same Rails process

[alfakini]

Hello team! Sharing here an idea, based on the problem we face in our current setup.

Got inspired to send this issue after #116, as I think we can comeup with a better API that helps us handle tracing and evals for multiple projects and OTel isolation.

First, let me explain our setup

We use the Braintrust Ruby SDK in production with multiple Braintrust projects within a single Rails process, and we keep Braintrust's OTel separate from the rest of the app's OTel. Trace context is stored in the metadata for each Session (traceparent, tracestate).

Per-process isolation: we use set_global: false and a dedicated TracerProvider per project. That way each agent's traces go to the right Braintrust project and we don't pollute the app's global OTel.

tracer_provider = OpenTelemetry::SDK::Trace::TracerProvider.new

braintrust = Braintrust.init(
  api_key: config[:api_key],
  default_project: config[:project_name],
  set_global: false, 
  auto_instrument: false,
  tracer_provider: tracer_provider
)

Then we only instrument the OpenAI clients that belong to that feature by passing the tracer_provider to instrumment!:

Braintrust.instrument!(:ruby_openai, target: openai_client, tracer_provider: tracer_provider)

Multi-turn conversations run in different Sidekiq jobs (one per message), but we want one trace for the whole conversation. So we use TraceContext propagators and persist the context on a DB record (Conversation Session). First message creates the root span and saves traceparent/tracestate, later jobs read that back and continue the same trace.

First message create root span and persist:

carrier = {}
propagator = OpenTelemetry::Trace::Propagation::TraceContext.text_map_propagator
propagator.inject(carrier, context: OpenTelemetry::Trace.context_with_span(span))

traceable.update!(
  traceparent: carrier["traceparent"],
  tracestate: carrier["tracestate"]
)

Later messages, restore and run inside that context:

carrier = {
  "traceparent" => traceable.traceparent,
  "tracestate" => traceable.tracestate
}.compact
parent_context = propagator.extract(carrier)

OpenTelemetry::Context.with_current(parent_context) do
  # All spans created here attach to the original trace
end

One gotcha: when we create a new root span we wrap it in OpenTelemetry::Context::ROOT so any ambient OTel from Rails middleware doesn't accidentally become the parent. Keeps the Braintrust trace clean.

OpenTelemetry::Context.with_current(OpenTelemetry::Context::ROOT) do
  tracer.in_span(name, attributes: attributes) do |span|
    # Inject the trace context (traceparent, tracestate)
    yield(span)
  end
end

What I would love to get

It would be great to have a first-class “multi-project with OTel isolation” API.

Idea: register projects by a stable key, and have the gem own the isolated TracerProvider per key.

Setup (e.g. once in an initializer): register each project with a key. The gem would create an isolated TracerProvider per key (no global, no cross-talk).

Braintrust.register(project: :braintrust_project_one)
Braintrust.register(project: braintrust_project_two)

Instrumenting a client: pass the project key instead of a tracer_provider.

Braintrust.instrument!(:ruby_openai, target: openai_client, project: :braintrust_project_one)

For custom spans and trace-context propagation we need the tracer (and optionally the tracer_provider) for a given project. Something like:

tracer = Braintrust.tracer(:braintrust_project_one)
tracer.in_span("my_operation", attributes: { ... }) { |span| ... }

# If we need the TracerProvider (e.g. for inject/extract with TraceContext):
tracer_provider = Braintrust.tracer_provider(:braintrust_project_one)

That would make multi-project + OTel isolation the default and easy to get right. Not sure if we need a "register" abstraction.

Would also be cool to have some "batteries included" with helpers to common spans. E.g.: root_span, task_span, tool_span, etc.

[delner]

As always, thanks for the thorough explanation here Alan R. Fachini (@alfakini) !

Your description of the setup is both very familiar and correct for OTel. I don't love the verbosity, but the protocol does require that kind of propagation if you're doing the equivalent of distributed tracing (going out of a single process) and "breaking" from the Rails root is a real concern... without that you can end up with a bunch of conjoined traces because of a long-running worker thread that never closes its own span (a real challenge with thread-context based tracing.)

Some thoughts/questions about your proposed design

1. When you say Braintrust.register that looks to me like a global variable registry... is that right? Works OK under simple contexts, but may run into trouble when processes fork, threads compete for the same resource, or loading/registration happens multiple times.
2. Could the project attributes reasonably be defined in something like a YAML file? That may help us side-step some of the in-memory problems potentially. (Not sure if this is the right idea, just exploring it.)
3. I like the idea of passing something like project instead of tracer_provider. Things like state, tracer_provider to me seem like implementation details that should not be a part of the public API... instead the API should speak in plain terms that represent the user's intent... I think project fits that well.

Ruminations on trace propagation

Not sure how I feel about the trace context propagation. Intuitively, I think Braintrust.tracer(:braintrust_project_one) might be a bit oversimplified unless we're equating one project to one active trace context. But is that strictly true? I don't think it is... a project is more of a repository... there's nothing saying you can't have multiple concurrent traces to the same project so how do you differentiate?

I also get the sense that this will have problems with trace_provider: I think the code is likely to depend on specific instances of tracer provider to function correctly and constructing a clone in another context may cause some kind of disconnect in traces. I think this is why there's Baggage/TraceContext which is to say "here's the essential stuff you need to continue a distributed trace" which should in theory be as simple as the trace ID and parent span ID... seems like the appropriate mechanism to continue distributed traces with.

This is not to suggest that current UX is the best we can hope for; I think whatever mechanism we provide likely has to rely on the OTel propagation but could wrap it more cleanly. I think an ideal interface is one where you'd express the minimal amount of information in user-domain terms. A design I did for the Datadog tracer before was like:

# Get trace continuation from active trace
trace_digest = Tracing.active_trace.to_digest

# Continue a trace: implicit continuation
# Digest will be "consumed" by the next `trace` operation
Tracing.continue_trace!(trace_digest)

...where the digest was a simple struct of the essentials to reconstruct trace state. Its serializable, immutable, thread-safe.

Some food for thought.

Next steps?

I think this should be rolled into the next major milestone for the Ruby SDK, which would be to overhaul state management for smoother UX and scaling. Supporting multiple projects seems like a clear design goal and success criterion for that milestone.

Next point of action would be to develop some architectural pattern candidates, so we can illustrate and weigh the pros/cons of them. I think we should incorporate what you've suggested here into one of those candidates.