CodeReplay is an engine-first tool for developer content creators to produce beautiful, animated code videos while keeping their preferred editor workflow. Users script keystroke sequences, apply visual effects (highlights, fades, zoom/focus), choose themes, and export polished videos (MP4, WebM, GIF). The core engine is a Rust library decoupled from any specific frontend, enabling integration into a reference app, VSCode/Zed extensions, Neovim/TUI integrations, or CLI tools.
┌─────────────────────────────────────────────────────────┐
│ Frontend Adapters (Tauri / Extensions / CLI) │
│ ┌──────────┐ ┌──────────┐ ┌────────┐ ┌─────────────┐ │
│ │ Editor │ │ Timeline │ │Preview │ │ Modal Input │ │
│ │ Panel │ │ Panel │ │ Panel │ │ Handler │ │
│ └────┬─────┘ └────┬─────┘ └───┬────┘ └──────┬──────┘ │
│ └─────────────┴───────────┴─────────────┘ │
│ │ Engine API │
├─────────────────────────┼───────────────────────────────┤
│ Core Engine │
│ ┌──────────┐ ┌────────────┐ ┌──────────┐ ┌─────────┐ │
│ │ Document │ │ Script │ │ State │ │ Plugin │ │
│ │ Model │ │ Model │ │ Evaluator│ │ Host │ │
│ └─────┬────┘ └─────┬──────┘ └────┬─────┘ └────┬────┘ │
│ │ │ │ │ │
│ ┌─────┴────┐ ┌─────┴──────┐ ┌────┴─────┐ ┌────┴────┐ │
│ │Tree-sitter│ │ Effects │ │ wgpu │ │ Theme │ │
│ │ Syntax │ │ Registry │ │ Renderer │ │ System │ │
│ └──────────┘ └────────────┘ └────┬─────┘ └─────────┘ │
│ │ │
│ ┌──────┴──────┐ │
│ │ Encoder │ │
│ │ (ffmpeg) │ │
│ └─────────────┘ │
└─────────────────────────────────────────────────────────┘
Core principle: The engine is a Rust library crate (codereplay-core). Frontends/adapters are separate crates that depend on it. The API boundary between them is the only contract. No host-editor UI logic leaks into the engine; no timeline/evaluation logic leaks into frontends.
codereplay/
├── Cargo.toml # Workspace root
├── crates/
│ ├── core/ # Top-level engine facade — re-exports sub-crates
│ │ └── src/lib.rs # pub use codereplay_*
│ ├── document/ # Code buffer, tree-sitter integration, token model
│ ├── script/ # Action sequence, timeline, serialization
│ ├── evaluator/ # Flattens script → visual state at time T
│ ├── renderer/ # wgpu rendering pipeline, frame output
│ ├── encoder/ # ffmpeg bindings, MP4/WebM/GIF export
│ ├── effects/ # Built-in effects + effect trait/registry
│ ├── theme/ # Theme definitions, style resolution
│ ├── plugin/ # Plugin loading (dynamic libraries), plugin API
│ ├── lsp/ # Language Server for script authoring/editor integration
│ ├── platform-desktop/ # Native implementations (ffmpeg, system fonts, fs)
│ ├── platform-web/ # WASM implementations (WebCodecs, bundled fonts, browser fs)
│ └── app/ # Reference Tauri frontend (modal authoring UI)
├── plugins/ # Example/built-in plugins
├── themes/ # Bundled theme files
└── SPEC.md
core ─┬─► document ──► tree-sitter
├─► script
├─► evaluator ──► script, document, effects, theme
├─► renderer ──► evaluator, theme (wgpu, glyphon)
├─► encoder ───► renderer (ffmpeg)
├─► effects
├─► theme
└─► plugin ───► effects, theme (libloading)
lsp ──► core, script
The code buffer with tree-sitter parsing.
The document can be initialized from any host/editor content and then driven by script actions. Host integrations can mirror external edits into this model so playback and export remain deterministic.
pub struct Document {
pub language: Language,
pub content: Rope, // ropey for efficient text manipulation
tree: Tree, // tree-sitter parse tree
}
pub struct Token {
pub id: TokenId, // stable identifier
pub kind: TokenKind, // keyword, identifier, literal, etc.
pub range: Range<usize>, // byte range in document
pub text: String,
}
pub enum TokenKind {
Keyword,
Identifier,
FunctionName,
TypeName,
StringLiteral,
NumericLiteral,
Comment,
Operator,
Punctuation,
Whitespace,
}The document re-parses incrementally on every edit via tree-sitter's edit() + parse(). Tokens are derived from the parse tree and assigned stable IDs that survive edits.
An ordered sequence of actions that define what happens over time.
pub struct Script {
pub actions: Vec<Action>,
pub groups: Vec<SelectionGroup>,
pub tracks: Vec<Track>,
pub clips: Vec<Clip>,
}
pub struct Action {
pub id: ActionId,
pub kind: ActionKind,
pub duration: Duration,
pub easing: Easing,
pub track: TrackId,
pub start_at: Option<Duration>,
}
pub enum ActionKind {
Type { content: String, wpm: u32, cursor_style: CursorStyle },
Delete { count: usize, direction: Direction },
Pause,
MoveCursor { target: CursorTarget },
ApplyEffect {
target: Selector,
effect: EffectId,
params: EffectParams,
timing: EffectTiming,
stagger_by: StaggerBy,
composition: EffectComposition,
lifetime: EffectLifetime,
},
ApplyStyle {
target: Selector,
style: StyleOverride,
composition: StyleComposition,
lifetime: StyleLifetime,
},
ClearEffects { target: Selector, effect: Option<EffectId> },
ClearStyles { target: Selector },
Transition { kind: TransitionKind },
Camera { keyframe: CameraKeyframe },
PlayClip { clip: ClipId },
}
pub enum Selector {
Token(TokenId),
TokenRange(TokenId, TokenId),
Group(GroupId),
Pair { open: TokenId, close: TokenId },
Query(String),
Capture { query: String, name: String },
Anchor(AnchorRef),
Union(Box<[Selector]>),
Intersect(Box<[Selector]>),
Subtract { base: Box<Selector>, remove: Box<Selector> },
Not(Box<Selector>),
Lines(Range<usize>),
All,
}
pub struct EffectTiming {
pub default_rate: f32,
pub default_start_offset: f32,
pub default_stagger: f32,
pub selector_rates: Box<[SelectorRate]>,
}Time model: Actions are scheduled per track. Actions on the same track are sequential by default; start_at enables explicit overlap across tracks (and within a track when needed). Total duration is the maximum action end time across all scheduled actions. This keeps playback deterministic while allowing layered choreography.
The evaluator computes the full visual state at any point in time by replaying the script up to that time.
pub struct VisualState {
pub visible_content: String,
pub cursor: Option<CursorState>,
pub tokens: Vec<RenderedToken>,
pub background: BackgroundState,
pub viewport: ViewportState,
}
pub struct RenderedToken {
pub token: Token,
pub style: ResolvedStyle, // base theme + any overrides/effects
pub position: Vec2, // computed layout position
pub opacity: f32,
pub scale: f32,
pub transform: Transform2D,
}
pub struct ResolvedStyle {
pub fg_color: Color,
pub bg_color: Option<Color>,
pub font_weight: FontWeight,
pub font_style: FontStyle, // italic, normal
pub decoration: Option<TextDecoration>, // underline, strikethrough
pub glow: Option<GlowParams>,
}Evaluation strategy:
- Start with the current document baseline state (which may be empty)
- Build scheduled actions from track/default ordering + explicit
start_at - Walk all actions active at time
Tand apply them with eased local progress - Resolve selectors (groups, pairs, queries, captures, anchors, algebra)
- Resolve styles/transforms with timing (
rate,start_offset,stagger) and composition/lifetime - Return the resulting
VisualState
For live preview, cache the last evaluated state and incrementally update when the user scrubs the timeline or during playback.
Serialization policy:
- Keep
.codereplayproject files and external interchange outputs JSON/XML-first for interoperability, readability, and stable diffs. - Add an optional binary snapshot/cache format for internal engine acceleration only (candidate: Cap'n Proto; alternatives: postcard/bincode).
- Treat binary snapshots as ephemeral and versioned; never make them the canonical project interchange format.
{
"version": "0.1.0",
"meta": {
"name": "Hello World Layered Demo",
"resolution": [1920, 1080],
"fps": 60,
"background": { "color": "#1e1e2e" }
},
"theme": "catppuccin-mocha",
"font": {
"family": "JetBrains Mono",
"size": 18,
"line_height": 1.6
},
"document": {
"language": "rust",
"initial_content": ""
},
"script": {
"tracks": [
{ "id": 1, "name": "code", "kind": "Code" },
{ "id": 2, "name": "fx", "kind": "Effects" },
{ "id": 3, "name": "camera", "kind": "Camera" }
],
"groups": [
{
"id": 0,
"name": "function-name",
"selector": {
"Capture": {
"query": "(function_item name: (identifier) @name)",
"name": "name"
}
}
}
],
"clips": [
{
"id": 0,
"name": "intro-pulse",
"actions": [
{
"kind": {
"ApplyEffect": {
"target": { "Group": 0 },
"effect": "bold",
"params": { "scale": 1.12 },
"timing": {
"default_rate": 1.0,
"default_start_offset": 0.0,
"default_stagger": 0.04,
"selector_rates": []
},
"stagger_by": "TokenOrder",
"composition": "Replace",
"lifetime": "DuringAction"
}
},
"duration_ms": 500,
"easing": "ease_in_out"
}
]
}
],
"actions": [
{
"id": 0,
"track": 1,
"kind": {
"Type": {
"content": "fn main() {\n let value = 42;\n}",
"wpm": 70,
"cursor_style": "Line"
}
},
"duration_ms": 1600,
"easing": "linear"
},
{
"id": 1,
"track": 2,
"start_at_ms": 300,
"kind": {
"ApplyEffect": {
"target": { "Group": 0 },
"effect": "highlight",
"params": { "color": "#f9e2af", "intensity": 0.8 },
"timing": {
"default_rate": 1.0,
"default_start_offset": 0.08,
"default_stagger": 0.02,
"selector_rates": [
{
"selector": { "Capture": { "query": "(identifier) @id", "name": "id" } },
"rate": 1.2,
"start_offset": 0.0,
"stagger": 0.015
}
]
},
"stagger_by": "TokenOrder",
"composition": "Replace",
"lifetime": "Persistent"
}
},
"duration_ms": 900,
"easing": "ease_in_out"
},
{
"id": 2,
"track": 3,
"start_at_ms": 500,
"kind": { "Camera": { "keyframe": { "offset_x": 80.0, "offset_y": 10.0, "zoom": 1.15 } } },
"duration_ms": 700,
"easing": "ease_out"
},
{
"id": 3,
"track": 2,
"start_at_ms": 2000,
"kind": { "ClearEffects": { "target": { "Group": 0 }, "effect": "highlight" } },
"duration_ms": 150,
"easing": "linear"
}
]
}
}Minimal quick-start example:
{
"version": "0.1.0",
"meta": {
"name": "Quick Start",
"resolution": [1280, 720],
"fps": 60
},
"theme": "catppuccin-mocha",
"document": {
"language": "rust",
"initial_content": ""
},
"script": {
"actions": [
{
"id": 0,
"track": 0,
"kind": {
"Type": {
"content": "fn main() {}",
"wpm": 70,
"cursor_style": "Line"
}
},
"duration_ms": 1200,
"easing": "linear"
}
]
}
}Notes:
actionsare scheduled ontracklanes;track: 0is the default lane.- Omitted
start_at_msmeans "start after the previous action on the same track." - Omitted advanced fields on
ApplyEffect/ApplyStyle(timing, composition, lifetime) use engine defaults. - Omitted
tracks/groups/clipsare treated as empty collections.
VisualState → Layout → Glyph Atlas → GPU Draw Calls → Frame (pixels)
│
┌───────────┤
▼ ▼
Screen Encoder
(preview) (export)
| Concern | Choice | Rationale |
|---|---|---|
| GPU API | wgpu |
Cross-platform, Rust-native, mature |
| Text layout | cosmic-text |
Handles shaping, bidi, fallback |
| Text rendering | glyphon |
Built on cosmic-text, wgpu-native glyph rasterization |
| Font technique | SDF (via glyphon) | Crisp at any scale, smooth animations |
| Frame encoding | ffmpeg (via CLI) |
Battle-tested, all formats, avoids binding complexity |
pub trait Renderer {
/// Render a single frame for the given visual state
fn render_frame(&mut self, state: &VisualState) -> Result<()>;
/// Read back pixels from the last rendered frame (for export)
fn read_pixels(&self) -> Result<Vec<u8>>;
/// Resize the render surface
fn resize(&mut self, width: u32, height: u32);
}Preview runs on a dedicated render thread. The main thread sends VisualState updates; the render thread draws at 60fps. When no state changes occur, the renderer skips redundant draws.
pub trait Encoder {
fn begin(&mut self, config: &ExportConfig) -> Result<()>;
fn write_frame(&mut self, pixels: &[u8], timestamp: Duration) -> Result<()>;
fn finish(&mut self) -> Result<PathBuf>;
}
pub struct ExportConfig {
pub format: OutputFormat,
pub resolution: (u32, u32),
pub fps: u32,
pub quality: Quality,
}
pub enum OutputFormat {
Mp4,
WebM,
Gif,
}Export process:
- Compute total duration from script
- Step through time at
1/fpsintervals - Evaluate
VisualStateat each time step - Render frame via wgpu (headless surface)
- Read back pixels
- Pipe raw frames to ffmpeg stdin (
-f rawvideo -pix_fmt rgba) - ffmpeg encodes to target format
Users can script terminal sessions alongside code — showing command execution, output, and switching between code and terminal views. Terminal content is fully scripted (not live PTY capture), giving complete control over timing and presentation.
pub struct TerminalState {
pub lines: Vec<TerminalLine>,
pub cursor: TerminalCursor,
pub prompt: String, // e.g., "$ " or "❯ "
pub shell_style: ShellStyle, // bash, zsh, fish, powershell
}
pub struct TerminalLine {
pub segments: Vec<TerminalSegment>,
pub is_input: bool, // true = user-typed command, false = output
}
pub struct TerminalSegment {
pub text: String,
pub style: AnsiStyle, // fg/bg color, bold, italic, underline
}
pub struct AnsiStyle {
pub fg: Option<AnsiColor>,
pub bg: Option<AnsiColor>,
pub bold: bool,
pub italic: bool,
pub underline: bool,
}
pub enum AnsiColor {
Named(NamedColor), // Red, Green, Blue, etc. (16 standard)
Rgb(u8, u8, u8), // 24-bit color
}New ActionKind variants for terminal:
pub enum ActionKind {
// ... existing variants ...
/// Type a command into the terminal prompt
TerminalInput {
command: String,
wpm: u32,
},
/// Display command output (appears instantly or line-by-line)
TerminalOutput {
lines: Vec<TerminalLine>,
mode: OutputMode, // Instant, LineByLine, Streaming
},
/// Clear the terminal
TerminalClear,
/// Switch between code editor and terminal views
ViewSwitch {
target: ViewTarget,
transition: ViewTransition,
},
}
pub enum ViewTarget {
Code,
Terminal,
Split { ratio: f32 }, // 0.5 = equal split
}
pub enum ViewTransition {
Cut,
Fade { duration: Duration },
SlideDown { duration: Duration },
SlideRight { duration: Duration },
}
pub enum OutputMode {
Instant, // All lines appear at once
LineByLine { delay_ms: u32 }, // Each line appears with a delay
Streaming { chars_per_sec: u32 }, // Characters stream in (like real output)
}The evaluator's VisualState is extended with a layout mode:
pub struct VisualState {
// ... existing fields ...
pub layout: LayoutState,
pub terminal: Option<TerminalState>,
}
pub enum LayoutState {
CodeOnly,
TerminalOnly,
Split {
orientation: SplitOrientation,
ratio: f32, // 0.0-1.0, portion allocated to code
},
}
pub enum SplitOrientation {
Horizontal, // code top, terminal bottom
Vertical, // code left, terminal right
}Since terminal content is scripted (not live PTY), it works identically across all frontends:
- Standalone app: terminal panel rendered by the engine
- VSCode/Zed extension: terminal panel is part of the engine's rendered preview, not the host editor's terminal
- Web: same engine rendering to WebGPU canvas
The terminal is a visual element within the video, not a real terminal. This means the exact same script produces the exact same video on any platform.
pub trait Effect: Send + Sync {
/// Unique identifier
fn id(&self) -> &str;
/// Human-readable name
fn name(&self) -> &str;
/// Parameter schema (for UI and validation)
fn params_schema(&self) -> ParamsSchema;
/// Apply this effect to a token's style at interpolation factor t (0.0–1.0)
fn apply(
&self,
base_style: &ResolvedStyle,
params: &EffectParams,
t: f32,
) -> ResolvedStyle;
/// Optional: modify token position/transform
fn transform(
&self,
base_transform: &Transform2D,
params: &EffectParams,
t: f32,
) -> Transform2D {
*base_transform
}
}| Effect | Description |
|---|---|
highlight |
Applies a background color or glow behind the token |
bold |
Increases font weight, optionally scales up slightly |
pop |
Scale up + slight bounce easing, draws attention |
fade_in |
Opacity 0 → 1 |
fade_out |
Opacity 1 → 0 |
zen |
Dims all tokens except the target (focus mode) |
crossfade |
Interpolates between two full visual states |
slide_in |
Token/line slides in from a direction |
typewriter |
Character-by-character reveal with cursor |
Transitions apply to the entire scene rather than individual tokens:
pub enum TransitionKind {
CrossFade { duration: Duration },
SlideLeft { duration: Duration },
SlideRight { duration: Duration },
ZoomIn { focal_point: Vec2, duration: Duration },
Cut, // instant switch
}Themes are TOML files:
[meta]
name = "Catppuccin Mocha"
author = "catppuccin"
version = "1.0.0"
[colors]
background = "#1e1e2e"
foreground = "#cdd6f4"
cursor = "#f5e0dc"
selection = "#45475a"
line_number = "#6c7086"
[syntax]
keyword = { fg = "#cba6f7", bold = true }
identifier = { fg = "#cdd6f4" }
function_name = { fg = "#89b4fa" }
type_name = { fg = "#f9e2af" }
string_literal = { fg = "#a6e3a1" }
numeric_literal = { fg = "#fab387" }
comment = { fg = "#6c7086", italic = true }
operator = { fg = "#89dceb" }
punctuation = { fg = "#6c7086" }
[editor]
padding = { x = 40, y = 30 }
border_radius = 12
shadow = { color = "#00000066", blur = 20, offset = [0, 4] }
show_line_numbers = true- Base theme → provides defaults for all token kinds
- Effect styles/transforms → apply by timeline order with effect timing (rate/offset/stagger)
- Action style overrides → apply on top using style composition mode
- Lifetime rules (
DuringAction,Persistent,ForDuration) decide whether a layer is active - Clear actions (
ClearEffects,ClearStyles) remove matching active layers - Later actions win over earlier actions for equal precedence/composition paths
Modal authoring is a core interaction model across frontends. A reference frontend can implement it directly, while editor integrations can map host keybindings to these same modal operations.
pub enum Mode {
/// Navigate timeline, select actions, playback controls
Normal,
/// Edit code content in the document
Insert,
/// Apply effects to selected tokens
Effect,
/// Apply style overrides to selected tokens
Style,
}Normal mode:
| Key | Action |
|---|---|
i |
Enter Insert mode |
e |
Enter Effect mode |
s |
Enter Style mode |
v |
Start token selection (visual mode) |
Space |
Play/pause preview |
j/k |
Navigate actions in script |
d |
Delete selected action |
u |
Undo |
Ctrl+r |
Redo |
: |
Command palette |
Insert mode:
| Key | Action |
|---|---|
Esc |
Return to Normal mode |
| Any text | Types into document, auto-generates Type action in script |
Enter |
Newline |
Backspace |
Delete, generates Delete action |
Effect mode:
| Key | Action |
|---|---|
Esc |
Return to Normal mode |
h |
Apply highlight to selection |
b |
Apply bold to selection |
p |
Apply pop to selection |
z |
Apply zen mode to selection |
f |
Apply fade-in to selection |
1-9 |
Set effect duration (×100ms) |
Style mode:
| Key | Action |
|---|---|
Esc |
Return to Normal mode |
c |
Change foreground color (opens picker) |
b |
Change background color |
f |
Change font weight |
i |
Toggle italic |
In Normal/Effect/Style modes, v enters visual selection. Selection is token-granular:
vthen move withw(next token),b(prev token),e(end of token group)Vselects entire lines- Selection is highlighted in the preview in real-time
Selections can be persisted as named groups and combined with selector algebra:
- Basic targets:
Token,TokenRange,Lines,All - Structural targets:
Group,Pair,Capture,Anchor - Algebra:
Union,Intersect,Subtract,Not
Selected tokens commonly become Selector::TokenRange, but users can promote them to reusable Group targets for later actions.
| Plugin type | What it provides | API |
|---|---|---|
| Theme | A .toml theme file |
Dropped into ~/.codereplay/themes/ |
| Effect | Custom Effect trait impl |
Dynamic library (.so/.dll/.dylib) |
| Transition | Custom TransitionKind |
Dynamic library |
| Exporter | Custom Encoder impl |
Dynamic library |
| Language | Tree-sitter grammar + queries | Grammar .so + highlight queries |
| Keybindings | Custom key mappings | TOML config |
/// Plugins implement this and export via `#[no_mangle]`
pub trait Plugin: Send + Sync {
fn name(&self) -> &str;
fn version(&self) -> &str;
fn on_load(&mut self, host: &dyn PluginHost);
fn on_unload(&mut self);
}
/// The host provides services to plugins
pub trait PluginHost {
fn register_effect(&self, effect: Box<dyn Effect>);
fn register_encoder(&self, encoder: Box<dyn Encoder>);
fn register_transition(&self, transition: Box<dyn TransitionFactory>);
fn log(&self, level: LogLevel, message: &str);
}Plugins are loaded via libloading. The plugin directory is ~/.codereplay/plugins/. Each plugin is a directory containing a manifest (plugin.toml) and the shared library.
These are config-only plugins — just files dropped into the right directory. No code loading required.
~/.codereplay/
├── themes/
│ ├── my-custom-theme.toml
│ └── nord.toml
├── plugins/
│ ├── blur-effect/
│ │ ├── plugin.toml
│ │ └── libblur_effect.so
│ └── svg-exporter/
│ ├── plugin.toml
│ └── libsvg_exporter.so
└── keybindings.toml
This is the public API that any frontend or host-editor adapter consumes. It is the only coupling point.
/// Top-level engine handle
pub struct Engine {
document: Document,
script: Script,
evaluator: Evaluator,
renderer: Renderer,
encoder: Option<Box<dyn Encoder>>,
plugin_host: PluginHost,
theme_registry: ThemeRegistry,
effect_registry: EffectRegistry,
}
impl Engine {
// --- Project lifecycle ---
pub fn new() -> Self;
pub fn load_project(path: &Path) -> Result<Self>;
pub fn save_project(&self, path: &Path) -> Result<()>;
// --- Document ---
pub fn document(&self) -> &Document;
pub fn edit_document(&mut self, edit: DocumentEdit) -> Result<()>;
// --- Script ---
pub fn script(&self) -> &Script;
pub fn push_action(&mut self, action: Action) -> ActionId;
pub fn push_action_on_track(&mut self, track: TrackId, action: Action) -> ActionId;
pub fn push_action_at(&mut self, track: TrackId, start_at: Duration, action: Action) -> ActionId;
pub fn insert_action(&mut self, index: usize, action: Action) -> ActionId;
pub fn remove_action(&mut self, id: ActionId) -> Result<()>;
pub fn move_action(&mut self, id: ActionId, new_index: usize) -> Result<()>;
pub fn define_group(&mut self, name: &str, selector: Selector) -> GroupId;
pub fn define_track(&mut self, name: &str, kind: TrackKind) -> TrackId;
pub fn define_clip(&mut self, name: &str, actions: Vec<ClipAction>) -> ClipId;
pub fn validate_script(&self) -> Vec<ScriptDiagnostic>;
// --- Evaluation ---
pub fn evaluate_at(&self, time: Duration) -> VisualState;
pub fn actions_at(&self, time: Duration) -> Vec<(ActionId, f32)>;
pub fn total_duration(&self) -> Duration;
// --- Rendering ---
pub fn render_frame(&mut self, state: &VisualState) -> Result<()>;
pub fn render_to_surface(&mut self, surface: &wgpu::Surface, state: &VisualState) -> Result<()>;
// --- Export ---
pub fn export(&mut self, config: ExportConfig, progress: impl Fn(f32)) -> Result<PathBuf>;
// --- Theme ---
pub fn set_theme(&mut self, name: &str) -> Result<()>;
pub fn available_themes(&self) -> Vec<ThemeInfo>;
// --- Effects ---
pub fn available_effects(&self) -> Vec<EffectInfo>;
// --- Undo/Redo ---
pub fn undo(&mut self) -> Result<()>;
pub fn redo(&mut self) -> Result<()>;
}- Provide a reference Tauri app for full workflow authoring (timeline, preview, modal editing)
- Provide host-editor adapters/extensions so creators can keep using their preferred editor
- Provide an LSP bridge for cross-editor authoring features (diagnostics, completions, navigation, code actions)
- Keep modal authoring semantics consistent across frontends via shared engine actions
- Keep rendering/export behavior identical across frontends by relying on the same engine API
- Web UI layer for rich editor/timeline/preview components
- Rust backend (Tauri commands) calls directly into
codereplay-core - No network hop — Tauri commands are FFI calls
- Cross-platform (macOS, Windows, Linux)
- The web UI can later be adapted for a web-only version
- If GPUI becomes viable later, only the
appcrate changes — core is untouched
┌──────────────────────────────────────────────────────┐
│ Toolbar: [Mode indicator] [Theme] [Export] [Settings]│
├──────────────────────────┬───────────────────────────┤
│ │ │
│ Code Editor │ Live Preview │
│ (editable) │ (wgpu canvas) │
│ │ │
│ │ │
│ │ │
├──────────────────────────┴───────────────────────────┤
│ Timeline: [action 1] [action 2] [action 3] ... │
│ ◄──────────── scrubber ─────────────────────────► │
├──────────────────────────────────────────────────────┤
│ Status bar: [NORMAL] [duration: 4.2s] [tokens: 23] │
└──────────────────────────────────────────────────────┘
The frontend handles ONLY:
- Rendering host-appropriate UI (reference code editor panel, or host-editor integration views)
- Capturing keyboard input and routing through the modal system
- Displaying the live preview (embedding wgpu surface or receiving rendered frames)
- Timeline UI (drag, reorder, scrub)
- Settings/preferences dialogs
- File open/save dialogs
All timeline, evaluation, rendering, and encoding logic lives in the engine.
codereplay-lsp provides editor-agnostic authoring intelligence for CodeReplay scripts:
- Diagnostics from script validation (
missing marker/group, invalid timing/rates, unsupported selectors) - Completions/hover for action variants, effect IDs, selector forms, and timing/composition/lifetime enums
- Go-to definition/references for groups, tracks, clips, and markers
- Code actions (create group from selection, add marker, wrap selector with algebra operators)
- Optional execute commands to trigger preview/evaluate-at-time through an engine sidecar
Design boundary:
- LSP owns authoring UX and static analysis
- Core engine owns evaluation, rendering, and export
- Existing language servers (rust-analyzer, pyright, etc.) remain responsible for code language semantics
Goal: Render a static code snippet with syntax highlighting to a window.
- Workspace setup with backend engine crates
-
codereplay-document: Rope buffer + tree-sitter integration for Rust -
codereplay-theme: Theme model + style resolution for tokens -
codereplay-renderer: wgpu/glyphon rendering foundation - Minimal binary that opens a window and displays syntax-highlighted code
Goal: Play back a scripted typing sequence with effects.
-
codereplay-script: Define action types, parse from JSON -
codereplay-evaluator: Evaluate script at time T, produce VisualState -
codereplay-effects: Implement core built-ins (highlight,fade_in,bold) + typing action flow - Playback loop in the renderer (advance time, evaluate, render)
- Easing functions
Goal: Export a scripted sequence to interchange and video targets.
-
codereplay-encoder: interchange backends (codereplay-package, OTIO, FCPXML) -
codereplay-encoder: ffmpeg integration (pipe raw frames to ffmpeg CLI) - Headless wgpu rendering (no window needed)
- Export command in the CLI binary (
export-interchange) - MP4/GIF/WebM video encoding path
- Editor import-spec validation pipeline for AE/Premiere/Resolve/FCP
Goal: Modal authoring workflow available in both the reference app and host-editor integrations.
- Tauri reference app shell with panel layout
- Modal input handler (Normal/Insert/Effect/Style)
- Live preview panel (wgpu surface embedded via Tauri)
- Timeline panel with scrubber
- Project save/load (
.codereplayJSON files) -
codereplay-lspMVP (diagnostics + completion + definitions for script symbols) - At least one editor adapter prototype (e.g., VSCode or Zed)
Goal: Extensibility and polish.
- Plugin loading (
libloading) - Theme file loading from
~/.codereplay/themes/ - More effects:
pop,zen,crossfade,slide_in - More transitions
- Multi-language support (load tree-sitter grammars dynamically)
- Undo/redo system
- Keybinding customization
- Advanced LSP code actions + refactors for selector/group/track workflows
The core engine is pure Rust compiled to different targets. Only the encoder and surface creation differ per platform.
| Target | Build | Use case |
|---|---|---|
| Native (x86_64/aarch64) | cargo build |
Tauri app, editor extensions, CLI tool |
WASM (wasm32-unknown-unknown) |
cargo build --target wasm32-unknown-unknown |
Web app, browser-hosted editor integrations |
| Component | Desktop (Native) | Web (WASM) | VSCode/Zed Extension |
|---|---|---|---|
| Renderer | wgpu → Vulkan/Metal/DX12 | wgpu → WebGPU | wgpu → WebGPU (webview) |
| Surface | Native window handle (winit) | <canvas> element |
Extension webview canvas |
| Encoder | FfmpegEncoder — pipe to ffmpeg CLI |
WebCodecsEncoder — browser WebCodecs API |
WebCodecsEncoder or delegate to host |
| Text layout | cosmic-text (native) | cosmic-text (WASM) | cosmic-text (WASM) |
| Text rendering | glyphon → wgpu | glyphon → wgpu/WebGPU | glyphon → wgpu/WebGPU |
| Tree-sitter | Compiled-in grammars (native) | .wasm grammars (web-tree-sitter) |
.wasm grammars |
| Plugin loading | libloading (dynamic .so/.dll) |
Not supported (WASM sandbox) | Not supported |
| File I/O | std::fs |
Browser File System Access API / IndexedDB | Extension filesystem API |
| Font loading | System fonts via cosmic-text |
Bundled fonts (WOFF2) or user-uploaded | Bundled fonts |
The engine uses trait objects for platform-varying behavior. Each platform provides its own implementations:
/// Platform-specific encoder
pub trait Encoder: Send {
fn begin(&mut self, config: &ExportConfig) -> Result<()>;
fn write_frame(&mut self, pixels: &[u8], timestamp: Duration) -> Result<()>;
fn finish(&mut self) -> Result<ExportResult>;
}
/// Platform-specific file system
pub trait FileSystem: Send + Sync {
fn read(&self, path: &str) -> Result<Vec<u8>>;
fn write(&self, path: &str, data: &[u8]) -> Result<()>;
fn list_dir(&self, path: &str) -> Result<Vec<String>>;
}
/// Platform-specific font provider
pub trait FontProvider: Send + Sync {
fn load_font(&self, family: &str, weight: FontWeight) -> Result<FontData>;
fn available_families(&self) -> Vec<String>;
}Desktop implementations:
pub struct FfmpegEncoder { /* pipes raw frames to ffmpeg CLI */ }
pub struct NativeFileSystem { /* std::fs wrapper */ }
pub struct SystemFontProvider { /* cosmic-text system font discovery */ }Web/WASM implementations:
pub struct WebCodecsEncoder { /* browser WebCodecs API via wasm-bindgen */ }
pub struct BrowserFileSystem { /* File System Access API + IndexedDB fallback */ }
pub struct BundledFontProvider { /* fonts bundled in the WASM binary or fetched */ }- No
Send + Syncon WASM: The engine runs single-threaded in WASM. Trait bounds useSendfor native but are relaxed for WASM via conditional compilation (#[cfg(target_arch = "wasm32")]). - No dynamic plugins on WASM: The plugin system is disabled on web targets. Only built-in effects and themes (bundled or loaded as config files) are available.
- Async encoding:
WebCodecsEncoderis async (browser APIs are promise-based). The encoder trait supports both sync (native) and async (WASM) paths viaasync_traitor a poll-based API. - Bundle size: Tree-sitter grammars add ~200-500KB per language as
.wasmfiles. Ship only the most common languages; load others on demand. - WebGPU availability: As of 2025, WebGPU is supported in Chrome, Edge, and Firefox. Safari has experimental support. For browsers without WebGPU, the web build will show a "WebGPU required" message — no canvas2d fallback (not worth the complexity).
# Cargo.toml workspace features
[workspace.features]
desktop = ["ffmpeg-encoder", "native-fs", "system-fonts", "plugin-loading"]
web = ["webcodecs-encoder", "browser-fs", "bundled-fonts"]The codereplay-core crate is platform-agnostic. Platform-specific implementations live in:
codereplay-platform-desktop— native encoder, filesystem, fontscodereplay-platform-web— WASM encoder, filesystem, fonts
The frontend crate (app for Tauri, or a future web-app) pulls in the right platform crate.
| Risk | Mitigation |
|---|---|
| wgpu text rendering quality | Prototype glyphon early in Phase 1. If quality is insufficient, fall back to swash + manual atlas. |
| ffmpeg dependency management | Shell out to system ffmpeg. Provide clear install instructions. Consider bundling a static ffmpeg binary for distribution. |
| Tree-sitter grammar loading | Start with compiled-in Rust grammar. Dynamic loading is a Phase 5 concern. |
| Plugin ABI stability | Pin a plugin API version. Plugins compiled against v0.1 must work with engine v0.1.x. Semver the plugin API. |
| Tauri + wgpu integration | Tauri supports custom protocol handlers and raw window handles. Prototype the wgpu surface embedding early. Fallback: render frames to images and display in an <img> tag (lower FPS but works). |
- Collaborative editing
- Cloud rendering
- Audio/voiceover sync
- Video import/overlay
- Mobile platforms
- Real-time streaming/broadcasting