NeuroMiner OBS Plugin

NeuroMiner OBS Plugin is an OS-level input capture plugin for OBS Studio, designed for world-model game data collection. It records global keyboard/mouse input and aligns it with OBS video frames, producing synchronized multimodal training data.

What It Records

For each OBS recording session, the plugin creates a UUID folder and writes:

• recording.<ext>: the original recorded video moved into the session folder
• actions.jsonl: per-frame input snapshots synchronized to OBS frame timing
• events.jsonl: raw sub-frame input events with microsecond session offsets
• metadata.json: session metadata (game name, platform, fps, duration, bias)

The plugin starts/stops data collection automatically with OBS recording start/stop events.

Core Use Case

This project targets game interaction dataset generation for world-model and behavior model training. The data format is built to support:

• frame-level action conditioning (actions.jsonl)
• sub-frame event reconstruction (events.jsonl)
• latency compensation and resampling in post-processing scripts

Platform Support

Platform	Status
macOS	Supported
Windows x64	Supported
Linux	Future

Install

macOS

Download the macos-universal artifact, extract it, then run the .pkg installer.

Windows

Download the windows-x64 artifact, extract it, then run the generated *-windows-x64-setup.exe installer. The installer auto-detects OBS installation paths and pre-fills the OBS plugin directory (...\obs-studio\obs-plugins).

Usage

1. Open OBS and find the NeuroMiner Input Recorder dock.
2. Set a game name (used in metadata.json).
3. Keep Enable Input Capture enabled.
4. Start OBS recording.
5. Interact with the game normally.
6. Stop OBS recording.
7. Locate the generated UUID session folder next to your recording output.

Build From Source

macOS

cmake --preset macos
cmake --build --preset macos

Windows (on Windows host)

cmake --preset windows-x64
cmake --build --preset windows-x64 --config RelWithDebInfo --parallel

Data Processing Scripts

Quick processing commands:

cd scripts
uv run python estimate_latency.py /path/to/<UUID>/
uv run python resample_events.py /path/to/<UUID>/

For full options and detailed script behavior, see scripts/README.md.

Data Format

Each recording session is written to one UUID directory.

recording.<ext>

• OBS output video file moved into the session directory.
• Extension depends on your OBS recording format (for example, .mp4).

metadata.json

• stream_name (string): session UUID.
• game_name (string): value set in the plugin dock.
• recorder_version (string): plugin version.
• platform ("mac" | "windows" | "linux"): runtime platform tag.
• video_meta (object):
  • width (int)
  • height (int)
  • fps (number)
  • total_frames (int)
  • duration_ms (int)
• session_start_timestamp_ms (int): Unix timestamp in milliseconds.
• input_latency_bias_ms (number): bias used for post-processing alignment.

actions.jsonl

One JSON object per video frame. Each line contains:

• frame (int): frame index.
• timestamp_ms (int): wall-clock timestamp when frame snapshot was recorded.
• frame_pts_ms (number): deterministic frame PTS (frame * 1000 / fps).
• capture_ns (int): OBS compositor timestamp in nanoseconds.
• key (string[]): currently pressed keys at this frame.
• mouse (object):
  • dx, dy (int): accumulated mouse movement since previous frame.
  • x, y (int): latest absolute mouse position.
  • scroll_dy (int): accumulated scroll delta since previous frame.
  • button (string[]): currently pressed mouse buttons.

events.jsonl

Raw sub-frame input event stream. Each line includes:

• type (string): one of:
  • key_down, key_up
  • mouse_move
  • mouse_button_down, mouse_button_up
  • scroll
  • (flags_changed on macOS)
• timestamp_ms (int): wall-clock event timestamp.
• session_offset_us (int): microseconds from session start.
• Optional fields by event type:
  • key (string) for keyboard events
  • button (string) for mouse button events
  • dx, dy, x, y for mouse movement
  • scroll_dy for scroll events

actions_resampled.jsonl (generated by scripts)

• Produced by scripts/resample_events.py.
• Reconstructed high-precision per-frame actions derived from events.jsonl and input_latency_bias_ms.
• Recommended output for training ingestion after latency compensation.

Privacy and Safety Notes

• Input capture is global while a recording session is active.
• Do not record sensitive input (passwords, private chats, credentials).
• Use test accounts and controlled environments for dataset collection.

License

Licensed under GPL-2.0-or-later. See LICENSE.