Skip to content

PLTF-2905: Expand Laminar observability guidance for OHE #579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
juanmichelini wants to merge 1 commit into
base: main
Choose a base branch
from
+111 −66
Open

PLTF-2905: Expand Laminar observability guidance for OHE #579

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,12 @@ When documenting LLM setup or examples, ensure all three options are mentioned w
- `sdk/shared-snippets/how-to-run-example.mdx` - Shared snippet for running examples
- `sdk/guides/llm-subscriptions.mdx` - Dedicated guide for subscription login


## Documentation ownership patterns

- Keep `sdk/guides/observability.mdx` as the canonical tracing and OTEL reference for OpenHands SDK users, including Laminar links and generic backend configuration.
- Keep `enterprise/analytics.mdx` focused on OpenHands Enterprise deployment and admin-console setup, and link back to the SDK observability guide instead of duplicating OTEL reference material.

## Validation

### LLM pricing table validation
Expand Down
76 changes: 41 additions & 35 deletions enterprise/analytics.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,35 +1,30 @@
---
title: Analytics
description: Get started with LLM observability and tracing in OpenHands Enterprise.
description: Deploy Laminar for trace analysis in OpenHands Enterprise.
icon: rocket
---

This guide walks you through setting up and using Laminar for Analytics in OpenHands Enterprise.
You'll opt into Analytics and configure conversations to automatically send traces to Laminar.
This guide walks you through enabling Laminar in OpenHands Enterprise (OHE) so conversations automatically send traces for observability and analysis.

## Who This Is For

This guide is for users who want to explore analytics on their OpenHands Enterprise conversations.

### Why Laminar?
For SDK-level tracing concepts, OTEL environment variables, and non-Laminar backends, see [Observability & Tracing](/sdk/guides/observability).

[Laminar](https://laminar.sh/) is an open source observability platform for AI agents like OpenHands.
## Who This Is For

Use Laminar to view your conversation traces including prompts, tool calls, and answers. A trace is a record of what your agent did.
This guide is for users who want to deploy Laminar alongside OpenHands Enterprise and inspect traces from Enterprise conversations.

Laminar can help you see where the agent went wrong. From traces, you can create signals. A signal is a natural language instruction to extract structured data from traces. Use signals to analyze recurring behavior across traces. You can then create better situations for prompting and measure them in Laminar. You can also analyze and improve your skills.
## Why Laminar in OHE?

For example, you can view all conversation traces related to a specific skill.
Laminar helps you understand what your OpenHands deployment is doing in production:

Laminar can help you answer the following questions:
- On a trace, did the agent do a good job using the skill?
- On another trace, did the agent do a bad job?
- Inspect prompts, tool calls, answers, and nested agent behavior in Laminar's [trace views](https://laminar.sh/docs/platform/viewing-traces).
- Use [session replay for browser agents](https://laminar.sh/docs/tracing/browser-agent-observability) when conversations drive browser automation.
- Define [signals](https://laminar.sh/docs/signals/introduction) to classify failures, measure outcomes, and monitor recurring patterns across many traces.

For more information on evaluating skills, see [Evaluating Agent Skills](https://www.openhands.dev/blog/evaluating-agent-skills).

### Prerequisites
## Prerequisites

Before you begin, make sure you completed the [Quick Start guide](/enterprise/quick-start).
Before you begin, complete the [Quick Start guide](/enterprise/quick-start).

## Enable Analytics

Expand All @@ -41,63 +36,65 @@ Check the **Enable Analytics** box to have the installer set up and configure La

## Deploy

OpenHands will begin deploying. You can expect the deployment status to transition from
**Missing** to **Unavailable** to **Ready**. This typically takes 10-15 minutes.
OpenHands will begin deploying. You can expect the deployment status to transition from **Missing** to **Unavailable** to **Ready**. This typically takes 10-15 minutes.

![Deployment in progress](./images/laminar-deploy-in-progress.png)

Click **Details** next to the deployment status to monitor individual resources. Resources
shown in orange are still deploying -- wait until all resources are ready.
Click **Details** next to the deployment status to monitor individual resources. Resources shown in orange are still deploying, so wait until all resources are ready.

![Deployment status details](./images/laminar-deployment-status-details.png)

## Access Laminar UI
## Access the Laminar UI

Once the deployment status shows **Ready**, navigate to `https://analytics.app.<your-base-domain>`.

Click the **Continue with Keycloak** button:

![Laminar Keycloak Auth](./images/laminar-keycloak-auth.png)

## Create a Laminar project
If you want more background on Laminar Cloud versus self-hosting outside OHE, see Laminar's official [hosting options](https://laminar.sh/docs/hosting-options).

## Create a Laminar Project

Create a project in the Laminar UI:

![Laminar Create Project](./images/laminar-create-project.png)

Once a project has been created, Laminar is ready to listen for traces.

![Laminar Listen Traces](./images/laminar-listen-traces.png)

## Create an ingest only API Key
## Create an Ingest-Only API Key

Important: Always use ingest API keys when deploying.
Always use ingest-only API keys when deploying OHE.

Create a key with the right permissions. Ingest only keys are recommended as they only have write access to write traces. They cannot be used to read data.
Ingest-only keys are recommended because OHE only needs permission to write traces. They cannot be used to read trace data.

![Configure Laminar Ingest Only Key](./images/laminar-ingest-only-key.png)

## Set Laminar Project API Key to enable automatic conversation traces
## Set the Laminar Project API Key

Set the ingest only key as the Laminar Project API Key in the Admin Console configuration:
Set the ingest-only key as the **Laminar Project API Key** in the Admin Console configuration. This is the same `LMNR_PROJECT_API_KEY` described in the [SDK observability guide](/sdk/guides/observability).

![Configure Laminar Project API Key](./images/laminar-configure-key.png)

Click **Save config**.

## Deploy Updated Configuration

Deploy the config change after setting the Laminar Project API Key in the Admin Console.
Deploy the configuration change after setting the Laminar Project API Key in the Admin Console.

![Laminar Deploy Again](./images/laminar-deploy-again.png)

Wait for the deployment to complete.

## Start a conversation
## Start a Conversation

Navigate to the OpenHands UI at `https://app.<your-base-domain>`. Start a new conversation and try a prompt.

![Start a Conversation](./images/laminar-openhands-conversation.png)

Your conversations will now automatically send a trace to Laminar.
Your conversations will now automatically send traces to Laminar.

![Laminar Trace](./images/laminar-trace.png)

Expand All @@ -107,16 +104,25 @@ Your conversations will now automatically send a trace to Laminar.
src="https://github.com/user-attachments/assets/0cdf1625-3246-4388-a989-765f00d33ffb"
></video>

## What to Do Next in Laminar

Once traces are flowing, use Laminar's official docs to go deeper:

- [Viewing Traces](https://laminar.sh/docs/platform/viewing-traces) to inspect a single conversation in transcript, tree, or timeline views.
- [Signals](https://laminar.sh/docs/signals/introduction) to extract structured outcomes or failure modes across many traces.
- [Session replay for browser agents](https://laminar.sh/docs/tracing/browser-agent-observability) to debug browser-based automations.
- [Observability for OpenHands Software Agent SDK](https://laminar.sh/docs/tracing/integrations/openhands-sdk) for the OpenHands-specific tracing model.

## Next Steps

<CardGroup cols={2}>
<Card title="Observability & Tracing" icon="activity" href="/sdk/guides/observability">
Learn the full OpenHands tracing model, OTEL configuration options, and non-Laminar backends.
</Card>
<Card title="Prompting Best Practices" icon="lightbulb" href="/openhands/usage/tips/prompting-best-practices">
Get the most out of your AI coding agents with effective prompting techniques.
Get more reliable traces by improving the prompts you give your agents.
</Card>
<Card title="Contact Support" icon="headset" href="https://openhands.dev/contact">
Reach out to the OpenHands team for deployment assistance or questions.
</Card>
<Card title="OpenHands Documentation" icon="book" href="/overview/introduction">
Explore the full OpenHands documentation for usage guides and features.
</Card>
</CardGroup>
95 changes: 64 additions & 31 deletions sdk/guides/observability.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@ title: Observability & Tracing
description: Enable OpenTelemetry tracing to monitor and debug your agent's execution with tools like Laminar, MLflow, Honeycomb, or any OTLP-compatible backend.
---

> A full setup example is available [here](#example:-full-setup)!
> A full setup example is available [below](#example-full-setup).

## Overview

The OpenHands SDK provides built-in OpenTelemetry (OTEL) tracing support, allowing you to monitor and debug your agent's execution in real-time. You can send traces to any OTLP-compatible observability platform including:
The OpenHands SDK provides built-in OpenTelemetry (OTEL) tracing support, allowing you to monitor and debug your agent's execution in real time. You can send traces to any OTLP-compatible observability platform including:

- **[Laminar](https://laminar.sh/)** - AI-focused observability with browser session replay support
- **[Laminar](https://laminar.sh/)** - AI-focused observability with trace inspection, signals, and browser session replay
- **[MLflow](https://mlflow.org/)** - Open-source AI platform with tracing, evaluation, and LLM governance
- **[Honeycomb](https://www.honeycomb.io/)** - High-performance distributed tracing
- **Any OTLP-compatible backend** - Including Jaeger, Datadog, New Relic, and more
Expand All @@ -27,14 +27,18 @@ Tracing is automatically enabled when you set the appropriate environment variab

### Using Laminar

[Laminar](https://laminar.sh/) provides specialized AI observability features including browser session replays when using browser-use tools:
[Laminar](https://laminar.sh/) provides specialized AI observability features for OpenHands, including full conversation traces, browser session replay, and higher-level analysis features like signals.

```bash icon="terminal" wrap
# Set your Laminar project API key
export LMNR_PROJECT_API_KEY="your-laminar-api-key"
```

That's it! Run your agent code normally and traces will be sent to Laminar automatically.
That's it. Run your agent code normally and traces will be sent to Laminar automatically.

<Note>
For Laminar-specific walkthroughs, see the official docs for [OpenHands SDK tracing](https://laminar.sh/docs/tracing/integrations/openhands-sdk), [session replay for browser agents](https://laminar.sh/docs/tracing/browser-agent-observability), [viewing traces](https://laminar.sh/docs/platform/viewing-traces), and [signals](https://laminar.sh/docs/signals/introduction).
</Note>

For **self-hosted Laminar** deployments, you can also configure custom ports:

Expand All @@ -44,6 +48,17 @@ export LMNR_HTTP_PORT=8000
export LMNR_GRPC_PORT=8001
```

If you need help deciding between Laminar Cloud and self-hosted Laminar, see Laminar's official [hosting options](https://laminar.sh/docs/hosting-options).

### Why use Laminar with OpenHands?

Laminar is especially useful when you want to understand how an agent behaved across one run or across many runs:

- Inspect a single run in transcript, tree, or timeline views to see prompts, tool calls, outputs, and nested agent activity. See Laminar's guide to [viewing traces](https://laminar.sh/docs/platform/viewing-traces).
- Watch browser automation alongside trace spans with [session replay for browser agents](https://laminar.sh/docs/tracing/browser-agent-observability).
- Define [signals](https://laminar.sh/docs/signals/introduction) to classify failures, user friction, or success patterns across many traces.
- Keep each OpenHands conversation grouped under a single session ID so multi-turn debugging is easier.

### Using OpenTelemetry (OTLP) Backends

For OpenTelemetry (OTLP) compatible backends, set the following environment variables:
Expand Down Expand Up @@ -83,23 +98,25 @@ export OTEL_EXPORTER="otlp_http" # or "otlp_grpc"

## How It Works

The OpenHands SDK uses the [Laminar SDK](https://docs.lmnr.ai/) as its OpenTelemetry instrumentation layer. When you set the environment variables, the SDK:
The OpenHands SDK uses Laminar as its OpenTelemetry instrumentation layer for built-in tracing support. When you set the environment variables, the SDK:

1. **Detects Configuration**: Checks for OTEL environment variables on startup
2. **Initializes Tracing**: Configures OpenTelemetry with the appropriate exporter
3. **Instruments Code**: Automatically wraps key functions with tracing decorators
4. **Captures Context**: Associates traces with conversation IDs for session grouping
5. **Exports Spans**: Sends trace data to your configured backend
1. **Detects configuration**: Checks for OTEL environment variables on startup
2. **Initializes tracing**: Configures OpenTelemetry with the appropriate exporter
3. **Instruments code**: Automatically wraps key functions with tracing decorators
4. **Captures context**: Associates traces with conversation IDs for session grouping
5. **Exports spans**: Sends trace data to your configured backend

For Laminar-specific behavior and examples, see the official [OpenHands SDK integration guide](https://laminar.sh/docs/tracing/integrations/openhands-sdk).

### What Gets Traced

The SDK automatically instruments these components:

- **`agent.step`** - Each iteration of the agent's execution loop
- **Tool Executions** - Individual tool calls with input/output capture
- **LLM Calls** - API requests to language models via LiteLLM
- **Conversation Lifecycle** - Message sending, conversation runs, and title generation
- **Browser Sessions** - When using browser-use, captures session replays (Laminar only)
- **Tool executions** - Individual tool calls with input/output capture
- **LLM calls** - API requests to language models via LiteLLM
- **Conversation lifecycle** - Message sending, conversation runs, and title generation
- **Browser sessions** - When using browser-use, captures session replays (Laminar only)

### Trace Hierarchy

Expand All @@ -119,10 +136,9 @@ Traces are organized hierarchically:
</Tree.Folder>
</Tree>

Each conversation gets its own session ID (the conversation UUID), allowing you to group all traces from a single
conversation together in your observability platform.
Each conversation gets its own session ID (the conversation UUID), allowing you to group all traces from a single conversation together in your observability platform.

Note that in `tool.execute` the tool calls are traced, e.g., `bash`, `file_editor`.
In `tool.execute`, the tool calls are traced individually, such as `bash`, `file_editor`, or `task_tracker`.

## Configuration Reference

Expand Down Expand Up @@ -174,15 +190,29 @@ The SDK supports both HTTP and gRPC protocols:
export LMNR_PROJECT_API_KEY="your-laminar-api-key"
```

**Self-Hosted Laminar**: If you are running a self-hosted Laminar instance, you can configure the HTTP and gRPC ports via environment variables:
**Self-hosted Laminar**: If you are running a self-hosted Laminar instance, you can configure the HTTP and gRPC ports via environment variables:

```bash icon="terminal" wrap
export LMNR_PROJECT_API_KEY="your-laminar-api-key"
export LMNR_HTTP_PORT=8000
export LMNR_GRPC_PORT=8001
```

**Browser Session Replay**: When using Laminar with browser-use tools, session replays are automatically captured, allowing you to see exactly what the browser automation did.
**Browser session replay**: When using Laminar with browser-use tools, session replays are automatically captured, allowing you to see exactly what the browser automation did.

### OpenHands Enterprise Setup

If you are running OpenHands Enterprise (OHE), you can use the same Laminar integration without changing application code:

1. Complete the [OpenHands Enterprise quick start](/enterprise/quick-start).
2. Enable analytics in the Admin Console.
3. Deploy OHE and wait for the analytics service to become ready.
4. Open the Laminar UI at `https://analytics.app.<your-base-domain>`.
5. Create a Laminar project and an ingest-only API key.
6. Save that key as the **Laminar Project API Key** in the Admin Console.
7. Redeploy, then start a conversation in OpenHands.

For the full OHE flow with screenshots, see [Analytics in OpenHands Enterprise](/enterprise/analytics).

### MLflow Setup

Expand All @@ -206,7 +236,7 @@ export OTEL_EXPORTER_OTLP_HEADERS="x-mlflow-experiment-id=123" # Replace "123"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
```

Navigate to the MLflow UI (e.g., `http://localhost:5000`), select the experiment, and open the **Traces** tab to view the recorded traces.
Navigate to the MLflow UI (for example, `http://localhost:5000`), select the experiment, and open the **Traces** tab to view the recorded traces.

### Honeycomb Setup

Expand Down Expand Up @@ -236,7 +266,7 @@ export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="grpc"
```

Access the Jaeger UI at http://localhost:16686
Access the Jaeger UI at `http://localhost:16686`.

### Generic OTLP Collector

Expand Down Expand Up @@ -274,9 +304,9 @@ The SDK automatically adds these attributes to spans:

### Debugging Tracing Issues

If traces aren't appearing in your observability platform:
If traces are not appearing in your observability platform:

1. **Verify Environment Variables**:
1. **Verify environment variables**:
```python icon="python" wrap
import os

Expand All @@ -287,35 +317,37 @@ If traces aren't appearing in your observability platform:
print(f"OTEL Headers: {otel_headers}")
```

2. **Check SDK Logs**: The SDK logs observability initialization at debug level:
2. **Check SDK logs**: The SDK logs observability initialization at debug level:
```python icon="python" wrap
import logging

logging.basicConfig(level=logging.DEBUG)
```

3. **Test Connectivity**: Ensure your application can reach the OTLP endpoint:
3. **Test connectivity**: Ensure your application can reach the OTLP endpoint:
```bash icon="terminal" wrap
curl -v https://api.honeycomb.io:443/v1/traces
```

4. **Validate Headers**: Check that authentication headers are properly URL-encoded
4. **Validate headers**: Check that authentication headers are properly URL-encoded.

For Laminar-specific troubleshooting, see Laminar's official [tracing troubleshooting guide](https://laminar.sh/docs/tracing/troubleshooting).

## Troubleshooting

### Traces Not Appearing

**Problem**: No traces showing up in observability platform
**Problem**: No traces showing up in your observability platform.

**Solutions**:
- Verify environment variables are set correctly
- Check network connectivity to OTLP endpoint
- Check network connectivity to the OTLP endpoint
- Ensure authentication headers are valid
- Look for SDK initialization logs at debug level

### High Trace Volume

**Problem**: Too many spans being generated
**Problem**: Too many spans being generated.

**Solutions**:
- Configure sampling at the collector level
Expand All @@ -324,7 +356,7 @@ If traces aren't appearing in your observability platform:

### Performance Impact

**Problem**: Concerned about tracing overhead
**Problem**: Concerned about tracing overhead.

**Solutions**:
- Tracing has minimal overhead when properly configured
Expand Down Expand Up @@ -391,6 +423,7 @@ uv run python examples/01_standalone_sdk/27_observability_laminar.py

## Next Steps

- **[Analytics in OpenHands Enterprise](/enterprise/analytics)** - Deploy Laminar inside OHE and send conversation traces automatically
- **[Metrics Tracking](/sdk/guides/metrics)** - Monitor token usage and costs alongside traces
- **[LLM Registry](/sdk/guides/llm-registry)** - Track multiple LLMs used in your application
- **[Security](/sdk/guides/security)** - Add security validation to your traced agent executions
Loading