Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Open Generative UI

An open-source showcase for building rich, interactive AI-generated UI with [CopilotKit](https://copilotkit.ai) and [LangGraph](https://langchain-ai.github.io/langgraph/). Ask the agent to visualize algorithms, create 3D animations, render charts, or generate interactive diagrams — all rendered as live HTML/SVG inside a sandboxed iframe.
An open-source showcase for building rich, interactive AI-generated UI with [CopilotKit](https://copilotkit.ai) and [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview). Ask the agent to visualize algorithms, create 3D animations, render charts, or generate interactive diagrams — all rendered as live HTML/SVG inside a sandboxed iframe.

https://github.com/user-attachments/assets/ed28c734-e54e-4412-873f-4801da544a7f

Expand Down Expand Up @@ -104,14 +104,27 @@ Turborepo monorepo with three packages:
```
apps/
├── app/ Next.js 16 frontend (CopilotKit v2, React 19, Tailwind 4)
├── agent/ LangGraph Python agent (GPT-5.4, CopilotKit middleware)
├── agent/ Deep Agent (deepagents + CopilotKit middleware, skills-based)
└── mcp/ Standalone MCP server (design system + skills + document assembler)
```

### Deep Agent + Skills

The agent backend uses [LangChain Deep Agents](https://docs.langchain.com/oss/python/deepagents/overview) (`create_deep_agent`) with a skills-based architecture. Instead of injecting all visualization instructions into the system prompt, skills are defined as `SKILL.md` files in `apps/agent/skills/` and loaded on-demand via progressive disclosure:

```
apps/agent/skills/
├── advanced-visualization/SKILL.md # UI mockups, dashboards, Chart.js, generative art
├── master-playbook/SKILL.md # Response philosophy, decision trees, narration patterns
└── svg-diagrams/SKILL.md # SVG generation rules, component patterns, diagram types
```

Deep agents also provide built-in planning (`write_todos`), filesystem tools, and sub-agent support.

### How It Works

1. **User sends a prompt** via the CopilotKit chat UI
2. **Agent decides** whether to respond with text, call a tool, or render a visual component
2. **Deep agent decides** whether to respond with text, call a tool, or render a visual component — consulting relevant skills as needed
3. **`widgetRenderer`** — a frontend `useComponent` hook — receives the agent's HTML and renders it in a sandboxed iframe
4. **Skeleton loading** shows while the iframe loads, then content fades in smoothly
5. **ResizeObserver** inside the iframe reports content height back to the parent for seamless auto-sizing
Expand Down Expand Up @@ -154,7 +167,7 @@ apps/

## Tech Stack

Next.js 16, React 19, Tailwind CSS 4, LangGraph, CopilotKit v2, Turborepo, Recharts
Next.js 16, React 19, Tailwind CSS 4, LangChain Deep Agents, LangGraph, CopilotKit v2, Turborepo, Recharts

## License

Expand Down
18 changes: 6 additions & 12 deletions apps/agent/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,26 +4,24 @@
"""

import os
from pathlib import Path

from copilotkit import CopilotKitMiddleware
from langchain.agents import create_agent
from deepagents import create_deep_agent
from langchain_openai import ChatOpenAI

from src.query import query_data
from src.todos import AgentState, todo_tools
from src.form import generate_form
from src.templates import template_tools
from skills import load_all_skills

# Load all visualization skills
_skills_text = load_all_skills()

agent = create_agent(
agent = create_deep_agent(
model=ChatOpenAI(model=os.environ.get("LLM_MODEL", "gpt-5.4-2026-03-05")),
tools=[query_data, *todo_tools, generate_form, *template_tools],
middleware=[CopilotKitMiddleware()],
state_schema=AgentState,
system_prompt=f"""
context_schema=AgentState,
skills=[str(Path(__file__).parent / "skills")],
system_prompt="""
You are a helpful assistant that helps users understand CopilotKit and LangGraph used together.

Be brief in your explanations of CopilotKit and LangGraph, 1 to 2 sentences.
Expand All @@ -47,10 +45,6 @@
- Pre-styled form elements (buttons, inputs, sliders look native automatically)
- Pre-built SVG CSS classes for color ramps (.c-purple, .c-teal, .c-blue, etc.)

Follow the skills below for how to produce high-quality visuals:

{_skills_text}

## UI Templates

Users can save generated UIs as reusable templates and apply them later.
Expand Down
1 change: 1 addition & 0 deletions apps/agent/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,5 @@ dependencies = [
"copilotkit>=0.1.77",
"langgraph-api>=0.7.16",
"langchain-mcp-adapters>=0.2.1",
"deepagents>=0.1.0",
]
26 changes: 0 additions & 26 deletions apps/agent/skills/__init__.py

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
---
name: "Advanced Visualization Techniques"
description: "UI mockups, dashboards, advanced interactivity, generative art, simulations, math visualizations, and design system rules for producing rich HTML widget output."
allowed-tools: []
---

# Agent Visualization Skills — Volume 2: Advanced Techniques

Prerequisite: Volume 1 (SVG diagrams, basic interactive widgets, Chart.js, Mermaid).
Expand Down Expand Up @@ -427,7 +433,7 @@ hardcoded hex values.
```
- Height goes on the wrapper div ONLY, never on canvas.
- Always set `responsive: true, maintainAspectRatio: false`.
- For horizontal bar charts: height = (bars × 40) + 80 pixels.
- For horizontal bar charts: height = (bars x 40) + 80 pixels.

### Custom Legend (Always Use This)
Disable Chart.js default legend and build HTML:
Expand All @@ -450,7 +456,7 @@ plugins: { legend: { display: false } }
```

### Dashboard Layout
Metric cards on top chart below sendPrompt for drill-down:
Metric cards on top -> chart below -> sendPrompt for drill-down:
```html
<!-- Metric cards grid -->
<div style="display: grid;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
---
name: "Master Agent Playbook"
description: "Philosophy, decision-making framework, and technical skills for delivering visual, interactive, and educational AI responses."
allowed-tools: []
---

# Master Agent Playbook: Making AI Responses Extraordinary

This playbook teaches an AI coding agent how to go beyond plain text and deliver
Expand All @@ -22,20 +28,20 @@ The principle: **Show, don't just tell.** Before writing any response, ask:

```
User asks a question
├─ Is it a quick factual answer? Answer in 1-2 sentences.
├─ Is it conceptual / "how does X work"?
├─ Is it spatial or visual? SVG illustrative diagram
├─ Is it a process/flow? SVG flowchart or HTML stepper
├─ Is it data-driven? Interactive chart (Chart.js / Recharts)
└─ Is it abstract but explorable? Interactive HTML widget with controls
├─ Is it "build me X"? Working code artifact, fully functional
├─ Is it a comparison? Side-by-side table or comparative visual
└─ Is it emotional/personal? Warm text response. No visuals needed.
|
+- Is it a quick factual answer? -> Answer in 1-2 sentences.
|
+- Is it conceptual / "how does X work"?
| +- Is it spatial or visual? -> SVG illustrative diagram
| +- Is it a process/flow? -> SVG flowchart or HTML stepper
| +- Is it data-driven? -> Interactive chart (Chart.js / Recharts)
| +- Is it abstract but explorable? -> Interactive HTML widget with controls
|
+- Is it "build me X"? -> Working code artifact, fully functional
|
+- Is it a comparison? -> Side-by-side table or comparative visual
|
+- Is it emotional/personal? -> Warm text response. No visuals needed.
```

### The 3-Layer Response Pattern
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
---
name: "SVG Diagram Generation"
description: "Generating rich inline SVG diagrams to visually explain systems, processes, architectures, and abstract concepts."
allowed-tools: []
---

# SVG Diagram Generation Skill

You can generate rich, inline SVG diagrams to visually explain concepts. Use this skill whenever a visual would help the user understand a system, process, architecture, or mechanism better than text alone.
Expand Down Expand Up @@ -46,9 +52,9 @@ Always use this template:
- **Sentence case always**. Never Title Case or ALL CAPS.

### Text Width Estimation
At 14px, each character 8px wide. At 12px, each character 7px wide.
- "Load Balancer" (13 chars) at 14px 104px needs rect 140px wide (with padding).
- Always compute: `rect_width = max(title_chars × 8, subtitle_chars × 7) + 48px padding`.
At 14px, each character ~ 8px wide. At 12px, each character ~ 7px wide.
- "Load Balancer" (13 chars) at 14px ~ 104px -> needs rect ~ 140px wide (with padding).
- Always compute: `rect_width = max(title_chars x 8, subtitle_chars x 7) + 48px padding`.

### Colors (Light/Dark Mode Safe)
Use these semantic color sets that work in both modes:
Expand Down Expand Up @@ -169,7 +175,7 @@ If you're rendering inside a system that supports CSS variables, prefer:

1. **ViewBox height**: Find your lowest element (max y + height). Set H = that + 40px.
2. **No content past x=640 or below y=(H-40)**.
3. **Text fits in boxes**: `(char_count × 8) + 48 < rect_width` for 14px text.
3. **Text fits in boxes**: `(char_count x 8) + 48 < rect_width` for 14px text.
4. **No arrows through boxes**: Trace every line's path — if it crosses a rect, reroute.
5. **All `<path>` connectors have `fill="none"`**.
6. **All text has appropriate fill color** — never rely on inheritance (SVG defaults to black).
Expand Down Expand Up @@ -207,7 +213,7 @@ For complex topics, use multiple smaller SVGs instead of one dense one:
<text x="340" y="56" text-anchor="middle" dominant-baseline="central"
font-size="12" fill="#0F6E56">HTTP POST /api/data</text>

<!-- Arrow 12 -->
<!-- Arrow 1->2 -->
<line x1="340" y1="76" x2="340" y2="100" stroke="#534AB7"
stroke-width="1.5" marker-end="url(#arrow)"/>

Expand All @@ -219,7 +225,7 @@ For complex topics, use multiple smaller SVGs instead of one dense one:
<text x="340" y="142" text-anchor="middle" dominant-baseline="central"
font-size="12" fill="#534AB7">Validate and transform</text>

<!-- Arrow 23 -->
<!-- Arrow 2->3 -->
<line x1="340" y1="162" x2="340" y2="186" stroke="#854F0B"
stroke-width="1.5" marker-end="url(#arrow)"/>

Expand Down
34 changes: 25 additions & 9 deletions apps/app/src/components/generative-ui/widget-renderer.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -369,7 +369,7 @@ window.addEventListener('message', function(e) {
var scriptCloses = (rawHtml.match(/<\\/script>/gi) || []).length;
var allScriptsClosed = scriptOpens <= scriptCloses;
tmp.querySelectorAll('script').forEach(function(s) {
incomingScripts.push({ src: s.src, text: s.textContent });
incomingScripts.push({ src: s.src, text: s.textContent, type: s.type || '' });
s.remove();
});

Expand Down Expand Up @@ -414,24 +414,36 @@ window.addEventListener('message', function(e) {
content.innerHTML = tmp.innerHTML;
}

// Execute only new scripts — skip entirely while a <script> tag is still streaming
// Execute only new scripts — skip entirely while a <script> tag is still streaming.
// External scripts (src) are loaded sequentially and we wait for each to finish
// before running subsequent scripts, so inline code can use libraries like Three.js.
if (allScriptsClosed) {
incomingScripts.forEach(function(scriptInfo) {
(function runScripts(scripts, idx) {
if (idx >= scripts.length) return;
var scriptInfo = scripts[idx];
var key = scriptInfo.src || scriptInfo.text;
if (!key || !key.trim()) return;
if (!key || !key.trim()) { runScripts(scripts, idx + 1); return; }
var hash = btoa(unescape(encodeURIComponent(key))).slice(0, 16).replace(/[^a-zA-Z0-9]/g, '');
if (!hash || content.getAttribute('data-exec-' + hash)) return;
if (!hash || content.getAttribute('data-exec-' + hash)) { runScripts(scripts, idx + 1); return; }
content.setAttribute('data-exec-' + hash, '1');
try {
var newScript = document.createElement('script');
if (scriptInfo.type) newScript.type = scriptInfo.type;
if (scriptInfo.src) {
newScript.src = scriptInfo.src;
newScript.onload = function() { runScripts(scripts, idx + 1); };
newScript.onerror = function() { runScripts(scripts, idx + 1); };
content.appendChild(newScript);
} else {
newScript.textContent = scriptInfo.text;
content.appendChild(newScript);
runScripts(scripts, idx + 1);
}
content.appendChild(newScript);
} catch(e) { console.warn('[widget] script exec failed:', e); }
});
} catch(e) {
console.warn('[widget] script exec failed:', e);
runScripts(scripts, idx + 1);
}
})(incomingScripts, 0);
}
reportHeight();
}
Expand Down Expand Up @@ -469,7 +481,11 @@ function assembleShell(initialHtml: string = ""): string {
style-src 'unsafe-inline';
img-src 'self' data: blob:;
font-src 'self' data:;
connect-src 'self';
connect-src 'self'
https://cdnjs.cloudflare.com
https://esm.sh
https://cdn.jsdelivr.net
https://unpkg.com;
">
<style>
${THEME_CSS}
Expand Down
Loading