spark-tui

A terminal UI for Apache Spark performance analysis via the Databricks driver proxy.

---

Features

• Live job dashboard — ranked by duration, auto-refreshing via background poller
• 16 suspect detectors — flags slow stages, data skew, disk spill, partition count issues, broadcast join opportunities, Python UDF usage, cache opportunities, and more
• Bottleneck classification — identifies Large Scan, Wide Shuffle, and Data Explosion patterns
• SQL correlation — links jobs and stages back to SQL executions with plan hints
• Critical path analysis — annotates the longest-running stage per job with a "CP" marker
• Estimated savings — each suspect includes an estimated time savings to help prioritize fixes
• Drill-down navigation — job list → job detail (stages + bar chart) → SQL plan view
• Help overlay — press h for context-sensitive keybinding reference and PySpark recommendations
• Smooth scrolling — tui-scrollview integration for fluid scrolling in detail views
• Color-coded severity — warning (yellow) and critical (red) indicators at a glance
• PySpark recommendations — each suspect includes concrete PySpark tuning suggestions
• Cross-platform releases — CI/CD builds for Linux, macOS (x86 + ARM), and Windows
• Zero setup — reads credentials from CLI flags, env vars, or ~/.databrickscfg

Quick Start

Prerequisites

• Rust toolchain (1.85+ for edition 2024)
• A running Databricks cluster with an active Spark application

Install

git clone https://github.com/tadeasf/spark-tui.git
cd spark-tui
cargo install --path .

Configure

Provide credentials using any of these methods (highest priority first):

Method	Host	Token	Cluster ID
CLI flags	--host	--token	--cluster-id
Environment	DATABRICKS_HOST	DATABRICKS_TOKEN	DATABRICKS_CLUSTER_ID
~/.databrickscfg	host	token	cluster_id

Run

# With CLI flags
spark-tui --host adb-123.azuredatabricks.net --token dapi... --cluster-id 0123-456789-abcdef

# With environment variables
export DATABRICKS_HOST=adb-123.azuredatabricks.net
export DATABRICKS_TOKEN=dapi...
export DATABRICKS_CLUSTER_ID=0123-456789-abcdef
spark-tui

# With a specific databrickscfg profile
spark-tui --profile my-workspace

Configuration

Flag	Env Var	Default	Description
--host	DATABRICKS_HOST	—	Workspace hostname (e.g. adb-123.azuredatabricks.net)
--token	DATABRICKS_TOKEN	—	Personal access token
--cluster-id	DATABRICKS_CLUSTER_ID	—	Cluster ID
--profile, -p	DATABRICKS_CONFIG_PROFILE	auto-detect	Profile name from ~/.databrickscfg
--poll-interval	SPARK_TUI_POLL_INTERVAL	10	Refresh interval in seconds
--event-log-path	SPARK_TUI_EVENT_LOG_PATH	—	DBFS path to Spark event log file (historical mode)
--sparkui-cookie	SPARK_TUI_SPARKUI_COOKIE	—	DATAPLANE_DOMAIN_DBAUTH cookie for Spark UI auth

Historical Mode

When a cluster is terminated, spark-tui automatically falls back to historical data using a 4-strategy chain:

1. Spark UI REST API — probes the Databricks Historical Spark UI endpoint (requires --sparkui-cookie for authenticated environments). If the UI is still warming up (downloading event logs), retries with backoff for ~53 seconds.
2. Spark History Server — probes known Databricks history server proxy URLs.
3. DBFS event logs — reads event logs from the cluster's configured log delivery path or --event-log-path.
4. Default DBFS paths — scans well-known DBFS directories for event log files.

The status line shows a HISTORICAL badge when viewing data from a terminated cluster.

Getting the Spark UI cookie

For the Spark UI strategy on authenticated Databricks workspaces:

1. Open the Spark UI in your browser (Cluster > Spark UI tab)
2. Open DevTools (F12) > Application > Cookies
3. Copy the value of DATAPLANE_DOMAIN_DBAUTH from the adb-dp-* domain
4. Pass it via --sparkui-cookie <value> or SPARK_TUI_SPARKUI_COOKIE=<value>

Keybindings

Key	Action
Tab / Shift+Tab	Switch between Jobs and Suspects tabs
j / ↓	Move selection down
k / ↑	Move selection up
g / Home	Jump to top
G / End	Jump to bottom
Enter	Drill into job detail / stage detail
s	Open SQL plan view (from job detail)
h	Toggle help overlay (keybinding reference / SQL recommendations)
Esc	Go back one level
q	Quit

Screenshots

Documentation

Full documentation is available at the GitHub Pages site.

Build docs locally:

cargo install mdbook
mdbook serve docs
# Open http://localhost:3000

License

MIT

Contributing

See Contributing Guide for development setup, testing, and code conventions.